Apparatus and method for adaptive label smoothing for confidence calibration

By employing learnable margin parameters and adaptive label smoothing weights, the method corrects overconfidence in deep learning models, ensuring accurate alignment of predicted probabilities with actual accuracy and improving discriminative ability.

KR103047790B1Active Publication Date: 2026-11-11IND ACADEMIC COOP FOUND YONSEI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
KR1020230014420
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-02-02
Publication Date
2026-11-11
Estimated Expiration
2043-02-02

AI Technical Summary

Technical Problem

Recent deep learning models suffer from overconfidence issues, where predicted probabilities exceed actual accuracy, and existing confidence correction methods fail to account for unique image characteristics, affecting discriminative ability.

Method used

A method that uses learnable margin parameters and adaptive label smoothing weights based on image classification difficulty to adjust predicted probability values, incorporating a learnable margin matrix and cross-entropy loss function to correct reliability.

Benefits of technology

Maintains predicted probability values in alignment with actual accuracy by reflecting image characteristics, enhancing the reliability and discriminative ability of deep learning networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112023012686428-PAT00040_ABST
    Figure 112023012686428-PAT00040_ABST
Patent Text Reader

Abstract

An embodiment of the present invention discloses a reliability correction method comprising the steps of: inputting image data to be classified into an image classification model; deriving a logit output value and a predicted probability value of the image data from the image classification model; and deriving a label smoothing loss function from the image classification model using at least one of a label smoothing weight based on image classification difficulty and a learnable margin parameter.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to confidence calibration that enables a deep learning network to have a reliable final prediction probability value (confidence). Background Technology

[0002] Outputting accurate confidence values ​​in image classification models is important in deep learning network training.

[0003] For example, if an image is input and the predicted probability for a certain class is output as 0.8, it should mean that the image is of that class with an 80% probability. Through this, it is possible to determine whether or not to follow the model's prediction value when making decisions.

[0004] In other words, a reliable image classification model must have matching prediction accuracy and confidence values ​​(prediction probability) for the samples.

[0005] However, recent deep learning models have suffered from the problem of overconfidence, where the predicted probability is greater than the actual accuracy. A technique to align the model's accuracy with the predicted probability is called confidence correction technology. To address this, conventional confidence correction technologies used a new loss function to mitigate the overconfidence problem along with the existing cross-entropy loss function during model training.

[0006] In addition, label smoothing is a technique proposed to solve the overfitting problem of deep learning models and improve generalization performance; it is a confidence-aware learning method that is itself effective in solving the over-confidence problem of models. The over-confidence problem is a situation where the predicted probability output for a specific label becomes larger than the actual probability, causing the distribution of output values ​​to take the form of a delta function.

[0007] Label smoothing mitigates the overconfidence problem by reducing the maximum output value of prediction probabilities and regularizing the output values ​​for different labels. However, the problem persisted that making the output values ​​of prediction probabilities similar for all classes had a negative effect on the discriminative ability that image recognition models ultimately require.

[0008] To address this, the subsequent method was improved by proposing a loss function that does not apply label smoothing when the output value of the prediction probability approaches a certain level.

[0009] That is, when deriving the loss function, label smoothing is not applied if the logit distance corresponding to the difference in output values ​​is less than or equal to a defined margin (see Equation 1). Here, the difference in output values ​​can be calculated as the difference between the logit value of the class corresponding to the maximum output value and the logit values ​​for the remaining classes.

[0010]

[0011] L MbLS is the loss function of the prior art, L CE is cross-entropy error (loss),

[0012] is logit distance, m: margin

[0013] The above method applies the same margin (m) to all images to ensure that each image has a similar difference in output values. However, since this fails to reflect the unique characteristics of each image, it did not fully achieve the objective of adjusting reliability (maximum prediction probability value). The problem to be solved

[0014] The objective of the present invention is to correct the reliability of a deep learning network using learnable margin parameters set considering the characteristics of the class.

[0015] The objective of the present invention is to correct the reliability of a deep learning network by reflecting the characteristic that the distribution of predicted probability values ​​changes depending on the classification difficulty of the image during label smoothing. means of solving the problem

[0016] According to an embodiment of the present invention, a method of operation of a computing device providing a label smoothing-based reliability correction function may include: (a) inputting image data to be classified into an image classification model; (b) the image classification model deriving a logit output value and a predicted probability value of the image data; and (c) the image classification model deriving a label smoothing loss function using at least one of a label smoothing weight based on image classification difficulty and a learnable margin parameter.

[0017] Additionally, among the above steps (c), the step of the image classification model deriving a label smoothing loss function using label smoothing weights based on image classification difficulty may include the step of deriving the label smoothing loss function by adaptively setting the label smoothing weights according to the image classification difficulty.

[0018] Additionally, the step of adaptively setting the label smoothing weight according to the image classification difficulty may include setting the label smoothing weight higher as the image classification difficulty is lower, and setting the label smoothing weight lower as the image classification difficulty is higher.

[0019] In addition, the data with high image classification difficulty may include at least one image among an image in which an object is partially visible in the image data, an image in which a viewpoint indicates a part of the object, and an image that is difficult for an image classification model to predict.

[0020] Additionally, data with low image classification difficulty may include at least one image among an image in which an object is clearly visible, an image in which an object is located in the center, and an image that is easy for an image classification model to predict.

[0021] Among the above steps (c), the step in which the image classification model derives a label smoothing loss function using label smoothing weights based on image classification difficulty is:

[0022] The method may include the step of determining the label smoothing weight as the maximum value among the prediction probability values ​​output from the softmax layer, and deriving the label smoothing loss function using the determined label smoothing weight.

[0023] Additionally, the step of deriving a label smoothing loss function using the margin parameter that the image classification model can learn in step (c) above may include the step of deriving the label smoothing loss function using the sum of the 'difference value between the logit output value and the learnable margin parameter' and the cross-entropy loss function.

[0024] Additionally, the method may include a step in which the learnable margin parameter is composed of a margin matrix with width and height of 'number of classes * number of classes', and the image classification model updates the learnable margin parameter whenever the confidence correction is performed using a label smoothing loss function. Effects of the invention

[0025] The objective of the present invention is to maintain the predicted probability value for the correct class and the actual accuracy (correctness likelihood) at the same level by using learnable margin parameters set considering the characteristics of the class.

[0026] The present invention can maintain the predicted probability value for the correct class and the actual accuracy (correctness likelihood) at the same level by reflecting the characteristic that the distribution of predicted probability values ​​changes according to the classification difficulty of the image during label smoothing. Brief explanation of the drawing

[0027] FIG. 1 shows a computing device (100) according to one embodiment of the present invention. FIG. 2 shows a flowchart of a label smoothing-based reliability correction operation according to one embodiment of the present invention. FIG. 3 shows an artificial intelligence model according to one embodiment of the present invention. Figure 4 shows an example of weighting according to one embodiment of the present invention. Specific details for implementing the invention

[0028] The technology described below is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the technology described below.

[0029] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other components in between.

[0030] Terms such as first, second, A, B, etc., may be used to describe various components, but such components are not limited by the said terms and are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of rights of the technology described below, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related described items or any of the multiple related described items.

[0031] In terms used in this specification, singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as “includes” should be understood to mean that the described features, number, steps, actions, components, parts, or combinations thereof exist, and not to exclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] Before providing a detailed description of the drawings, it is to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Furthermore, each component described below may additionally perform some or all of the functions of other components in addition to its own primary function, and it goes without saying that some of the primary functions of each component may be exclusively performed by other components.

[0033] Furthermore, in performing the method or operation method, each process constituting the method may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0034] The present invention discloses a reliability correction computing device utilizing adaptive label smoothing technology. Specifically, during network training, a conventional cross-entropy loss function and a new adaptive margin-based label smoothing loss function are used.

[0035] This technology allows for more effective reliability correction by flexibly applying loss functions and margins according to the class of an image when performing label smoothing-based reliability correction in a deep learning network, by reflecting actual probability values ​​in the label smoothing process while considering the characteristics of each image.

[0036] Specifically, the label smoothing loss function increases the entropy of the prediction probability distribution to prevent the network from overfitting and making overconfident predictions. In particular, confidence can be corrected more effectively by flexibly applying the loss function depending on the input instance and class.

[0037] The following explains that a computing device for reliability correction corrects the reliability of a deep learning network. A computing device for reliability correction is a device that consistently processes input data and performs calculations necessary for the design of a building according to a specific model or algorithm. For example, a computing device for reliability correction can be implemented in the form of a PC, a server on a network, a smart device, or a chipset with a design program embedded therein.

[0038] First, the configuration of a computing device (100) that provides a label smoothing function according to an embodiment of the present invention will be described.

[0039] FIG. 1 shows a computing device (100) that provides a label smoothing-based reliability correction function according to one embodiment of the present invention.

[0040] The components of the computing device (100) providing a label smoothing-based reliability correction function illustrated in FIG. 1 are exemplary. Only some of the components illustrated in FIG. 1 may constitute the computing device (100) providing a reliability correction function, and additional component(s) other than the components illustrated in FIG. 1 may be included in the computing device (100) providing a reliability correction function.

[0041] As illustrated in FIG. 1, a computing device (100) providing a reliability correction function may include a processor (110), memory (120), a communication unit (130), and an image classification unit (140).

[0042] The communication unit (130) can transmit and receive data with external devices, such as other electronic devices or servers, using wired or wireless communication technology. For example, the communication unit (130) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0043] The memory (120) can store data that supports various functions of the computing device (100).

[0044] The processor (110) can determine at least one executable operation of the computing device (100). And, the processor (110) can control the components of the computing device (100) to perform the determined operation.

[0045] To this end, the processor (110) can request, search, receive, or utilize data in memory (120), and can control the components of the computing device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0046] The image classification unit (140) may include a data classification unit (141), a learning unit (142), and a reliability correction unit (143). The image classification unit (140) will be described later.

[0047] At this time, if the processor (110) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0048] The processor (110) can control at least some of the components of the computing device (100) or a combination of the components in order to run an application stored in memory (120).

[0049] A computing device (100) according to one embodiment of the present invention can transmit and receive data through an interconnection via wireless and / or wired connections. The computing device of the present disclosure may include any type of computing device capable of performing computations on data in electronic form.

[0050] For example, it can be implemented in stationary devices or mobile devices such as TVs, projectors, mobile phones, smartphones, desktop computers, laptops, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, tablet PCs, wearable devices, set-top boxes (STBs), DMB receivers, radios, washing machines, refrigerators, desktop computers, digital signage, robots, vehicles, etc.

[0051] FIG. 2 shows a flowchart of a label smoothing-based reliability correction operation according to one embodiment of the present invention.

[0052] First, referring to FIG. 1, the image classification unit (140) may include a data classification unit (141), a learning unit (142), and a reliability correction unit (143).

[0053] The image classification unit includes an artificial intelligence model which is an image classification model, and the image classification model may include at least one layer. Additionally, the image classification model may be a classification model that performs classification on the image data when image data is input.

[0054] The data classification unit (141) may refer to a module that classifies input image data into a specific class. The training unit (142) can train an image recognition model using a predetermined dataset or training data so that when image data is input into the image classification model (140), a class corresponding to the image data is output.

[0055] The reliability correction unit (143) performs a conventional reliability correction function, and when performing reliability correction based on label smoothing, it can perform reliability correction by flexibly applying a loss function and margin according to the class of the image by reflecting the actual probability value in the label smoothing process while considering the characteristics of each image.

[0056] An image classification model may be able to operate based on the above-mentioned data classification unit, learning unit, and reliability correction unit.

[0057] Meanwhile, the above operations may be controlled through the processor (110).

[0058] The following explains in detail.

[0059] A computing device (100) for reliability correction according to an embodiment of the present invention can input image data into an image classification model (S201).

[0060] The data classification unit (141) of the image recognition model according to an embodiment of the present invention can generate a logit output value and a predicted probability value for the input image data (S203).

[0061] Specifically, the logit output value refers to the feature vector of the input image data, and there may be as many logit feature values ​​as there are classes to be classified in the input image data.

[0062] The predicted probability value may be a probability value corresponding to a classification class obtained by inputting the above logit output value into a softmax layer. The predicted probability values ​​are all normalized to values ​​between 0 and 1 by the softmax layer, and the sum of the output values ​​can always be 1. The maximum value among the above predicted probability values ​​is named the confidence value.

[0063] According to an embodiment of the present invention, the reliability correction unit (143) can perform a reliability correction operation of an image classification model by deriving a logit output value and a predicted probability value for the generated image data, and applying label smoothing to a weight and a learnable margin parameter based on the image classification difficulty (S205).

[0064] According to an embodiment, it is also possible for the image classification model to derive a label smoothing loss function using at least one of a label smoothing weight based on image classification difficulty and a learnable margin parameter.

[0065] Afterwards, the image classification model can be trained by updating the parameters of the deep learning model to minimize the error (S207).

[0066] At this time, the above-mentioned learnable margin parameters can be learned through network loss learning (S207).

[0067] Afterwards, the image classification model is trained using the learning unit (142) in a direction that minimizes the error, and when the number of training attempts exceeds the number of completion attempts (S209 - YES), the training can be completed.

[0068] A computing device according to an embodiment of the present invention can evaluate the performance of a reliability correction function through verification of verification data and performance evaluation of test data (S211).

[0069] Below, the operations of S205 and S207 will be explained in detail through FIG. 3 and Equation 2.

[0070] FIG. 3 shows an artificial intelligence model according to one embodiment of the present invention.

[0071] The artificial intelligence model (300) shown in FIG. 3 may include an image classification model (300) as previously described. The image classification model may be a model that outputs an image classification result corresponding to the image data when image data is input.

[0072] An image classification model (300) according to an embodiment of the present invention may include a data classification unit (320) for classifying image data and a reliability correction unit (330) for reliability correction. Specifically, the reliability correction unit (330) may perform reliability correction using a logit output value including a feature vector of image data input to the image classification model, a softmax layer (332), a probability value (333) output from the softmax layer, a learnable margin parameter (335), and a weighted loss function (334) based on the characteristics of the input image data.

[0073] Referring to FIG. 3, the image recognition model (300) has a logit output value (Logit Value, output from a neural network The predicted probability value ( ) that passes through the last Softmax layer It occurs based on ). In this case, the Logit output value and the predicted probability value are the number of dataset classes ( Can be generated as much as ). The image classification model uses the largest probability value among the predicted probability values ​​( The final image is predicted as a class having ). At this time, the maximum probability value can be set as the confidence value.

[0074] 'Mathematical Formula 2' is a formula for calculating the loss function of an image classification model according to an embodiment of the present invention.

[0075]

[0076] : Cross-entropy loss value

[0077] : Label smoothing weight (weight of logit distance loss)

[0078] : Maximum value among Logit outputs, : All values ​​excluding the maximum among the Logit outputs (k=0 ~ number of classes-1 , ), Learnable margin parameter

[0079] Referring to 'Mathematical Formula 2' above : The maximum value among the Logit outputs and : All Logit values ​​excluding the maximum (k=0 ~ number of classes-1 , After calculating the difference, the logit distance can be derived by subtracting the learnable margin parameter.

[0080] Label smoothing weights are assigned to the above logit distance, and the reliability correction unit (330) can perform reliability correction of the image classification model by applying learnable margin parameters to the loss function (LOSS).

[0081] First, the reliability correction unit (330) can perform a reliability correction operation of the image classification model by adjusting the label smoothing weight based on the image classification difficulty based on the logit output value and the predicted probability value for the generated image data (S205-1).

[0082] In this case, the image classification difficulty can be set according to different criteria for each object, taking into account the characteristics of the input image data.

[0083] FIG. 4 is a diagram showing a configuration for adjusting label smoothing weights based on image classification difficulty according to an embodiment of the present invention.

[0084] Referring to Figure 4, data with low image classification difficulty (Easy Instance, 410) corresponds to images that are easy for an image classification model to predict, such as when an object is clearly visible in the image data or when an object is located in the center, and may represent a case where the entropy of the softmax output is small (Delta Distribution).

[0085] In addition, data with high image classification difficulty (Hard Instance, 420) corresponds to images that are difficult for an image classification model to predict, such as when an object appears partially in the image data or when a viewpoint represents a part of the object, and may refer to cases where the entropy of the softmax output is large (Uniform Distribution).

[0086] The image classification model according to an embodiment of the present invention can adaptively set the label smoothing weight according to the image classification difficulty.

[0087] Specifically, the lower the difficulty of the image classification, the higher the label smoothing weight can be set.

[0088] The higher the difficulty of the image classification, the lower the label smoothing weight can be set compared to a predetermined value.

[0089] For example, the label smoothing weights of the above 'Equation 2' The maximum value among the predicted probability values ​​output from the softmax layer It can be set to.

[0090] This is because, as the image classification difficulty decreases, the maximum probability value (p) for the image classification model's prediction increases; therefore, high weights must be assigned to effectively mitigate the overconfidence problem.

[0091] The reliability correction unit (330) according to an embodiment of the present invention can perform a reliability correction operation of an image classification model by applying a learnable margin parameter to a loss function (S205-2).

[0092] Referring to FIG. 3, the learnable parameter may be composed of a margin matrix (335) with a width and height of 'number of classes * number of classes', and the margin parameter may be stored in each row and column of the margin matrix, corresponding to each class to be compared.

[0093] As in 'Mathematical Formula 2' above : The maximum value among the Logit values ​​and : All Logit values ​​excluding the maximum value (k=0 ~ number of classes-1, After calculating the difference, the learnable margin parameter is subtracted, and the confidence correction unit (330) can perform the confidence correction operation of the image classification model by applying the learnable margin parameter to the loss function (LOSS). In addition, the learnable margin parameter can be automatically updated as the loss function of the image classification model (300) is learned.

[0094] As in the above example, by setting learnable parameters for each class, the network learns the margin portion and the different margins between classes on its own, so it will be possible to solve the problem caused by hyperparameters with fixed values.

[0095] Specifically, through the above method, if class I, whose correct answer class is i, is more similar to class j than to class k and has very little similarity to l, these relational characteristics can be reflected in the over-confidence judgment. This is different from the conventional method of judging overconfidence between classes i and j, and classes i and k, using the same margin (predefined hyperparameter) to determine different images. class Compared to techniques that apply loss with the same criteria and margin for that as well, when judging Overconfidence class By using a margin that takes into account the difference, it is possible to perform a more accurate reliability correction that reflects image characteristics.

[0096] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all of these.

[0097] The present invention described above can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Claims

Claim 1 A method of operation of a computing device providing a reliability correction function based on label smoothing comprises: (a) inputting image data to be classified into an image classification model; (b) the image classification model deriving a logit output value and a predicted probability value of the image data; and (c) the image classification model deriving a label smoothing loss function using a label smoothing weight based on image classification difficulty and a plurality of learnable margin parameters, wherein the image classification model adaptively sets the label smoothing weight according to the image classification difficulty, and determines the maximum value among the predicted probability values ​​output from the softmax layer of the image classification model as the label smoothing weight, and the label smoothing loss function includes a cross-entropy loss value and a logit distance loss multiplied by the label smoothing weight, and the logit distance loss is, in the difference between the maximum value among the logit output values ​​and each of the remaining logit output values ​​excluding the maximum value, the class corresponding to the maximum value among the margin matrix including the plurality of learnable margin parameters and the remaining logit output values A method calculated based on multiple terms, subtracting elements located in rows and columns corresponding to each corresponding class. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 A method according to claim 1, wherein the plurality of learnable margin parameters correspond to each element of a margin matrix in which the number of rows and columns are each the number of classes, and further comprising the step of updating the plurality of learnable margin parameters whenever the reliability correction is performed based on the label smoothing loss function. Claim 9 delete Claim 10 A computing device providing a label smoothing-based reliability correction function, comprising: memory; an image classification unit; and includes a processor that controls the operation of the image classification unit, wherein the processor (a) inputs image data to be classified into an image classification model, (b) the image classification model derives a logit output value and a predicted probability value of the image data, and (c) the image classification model derives a label smoothing loss function using a label smoothing weight based on image classification difficulty and a plurality of learnable margin parameters, wherein the image classification model adaptively sets the label smoothing weight according to the image classification difficulty, and determines the maximum value among the predicted probability values ​​output from the softmax layer of the image classification model as the label smoothing weight, and the label smoothing loss function includes a cross-entropy loss value and a logit distance loss multiplied by the label smoothing weight, and the logit distance loss is, in the difference between the maximum value among the logit output values ​​and each of the remaining logit output values ​​excluding the maximum value, the row corresponding to the class corresponding to the maximum value and the class corresponding to each of the remaining logit output values ​​among the margin matrix including the plurality of learnable margin parameters A computing device calculated based on multiple terms obtained by subtracting elements located in columns. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 A computing device according to claim 10, wherein the plurality of learnable margin parameters correspond to each element of a margin matrix in which the number of rows and columns are the number of classes, and the processor updates the plurality of learnable margin parameters whenever the reliability correction is performed based on the label smoothing loss function. Claim 18 delete