Label confidence evaluation method and apparatus

Abstract

This invention provides an objective and reliable label confidence evaluation method and apparatus for assessing the predictive accuracy of a model. [Solution] The method includes the steps of: acquiring an image set consisting of multiple images, each containing a label corresponding to at least one class; dividing the image set into K unit sets; selecting one of the unit sets as a validation set and then repeatedly performing the process of training a predetermined network function using the remaining K-1 unit sets as training data K times to generate K evaluation models; inputting the validation sets into the corresponding evaluation models and outputting predicted values ​​corresponding to each of the multiple images; and evaluating the reliability of the labels by comparing the predicted values ​​with the labels of the corresponding images.

Description

Technical Field

[0001] This application relates to a method and apparatus for evaluating label reliability.

Background Art

[0002] The performance of a deep learning model highly depends on the quality of data, especially the accuracy of labels. Most deep learning models are trained through large-scale datasets, and the model can make accurate predictions only when this dataset contains correct labels for each image or sample. However, in practice, a lot of labeling work is required to build a large-scale dataset, and label errors frequently occur in this process.

[0003] Labeling work is mainly performed by crowd workers, but these often lack a deep understanding of the data itself and the deep learning model. For this reason, the labeling work can be simplified or done in a wrong way. For example, when an incorrect label is assigned to an image or sample that does not belong to the correct class, or when different labels are given to objects of the same class, or when important objects are not detected in the object detection or segmentation work and are missing in the labeling process, various types of label errors may occur.

[0004] Such errors directly affect the learning process of the model. Since a deep learning model constructs a prediction model based on the given data, if it learns data containing incorrect labels, the performance of the model will decline and the reliability of the prediction results will also decrease. In particular, since a deep learning model learns patterns from large-scale data, even a small number of label errors can have a serious impact on the overall model performance.

[0005] Therefore, in order to solve the problems caused by such label errors, new technologies for identifying low-reliability labels are required. [Overview of the project] [Problems that the invention aims to solve]

[0006] This application aims to provide a method and apparatus for evaluating label reliability. [Means for solving the problem]

[0007] An embodiment of this application provides a method for evaluating label confidence. The method may include the steps of: acquiring an image set consisting of a plurality of images, each containing a label corresponding to at least one class; dividing the image set into K unit sets; selecting one of the unit sets as a validation set, and then repeatedly performing the process of training a predetermined network function using the remaining K-1 unit sets as training data K times to generate K evaluation models; inputting the validation sets into the corresponding evaluation models to output predicted values ​​corresponding to each of the plurality of images; and evaluating the confidence of the labels by comparing the predicted values ​​with the labels of the corresponding images.

[0008] Furthermore, the labels are for the purpose of image classification, the predicted values ​​include information on the probability values ​​corresponding to each of the classes, and the step of evaluating the reliability of the labels may include: determining self-confidence based on the probability values ​​corresponding to the correct labels extracted from the predicted values ​​corresponding to each image; and determining the reliability of the labels for each image using the difference between the self-confidence and the highest probability value among the other classes excluding the correct labels.

[0009] Furthermore, the step of evaluating the confidence level of the label may further include a step of normalizing the difference between the self-confidence level and the highest probability value among the other classes excluding the correct label to a predetermined range and calculating a normalized margin, and the step of determining the confidence level of the label may be performed based on the normalized margin.

[0010] Furthermore, the label is for object detection purposes, and the predicted value includes information on the coordinates of at least one bounding box, the class and probability value predicted to exist in each of the bounding boxes, and the step of evaluating the reliability of the label may be performed through a confidence score calculated based on at least one of the following: a bad location score based on the degree of mismatch between the bounding box of the predicted value and the bounding box included in the corresponding label for each image; an overlooked score when the bounding box of the label corresponding to the predicted value is missing; and a swapped score when the class of the predicted value is different from the class of the corresponding label.

[0011] Furthermore, the steps for evaluating the reliability of the labels may include: calculating a first confidence score per bounding box for each image based on at least one of the position error score, the oversight score, and the class replacement score for the bounding boxes included in the predicted value and at least one of the labels; calculating a second confidence score per image for each image through a predetermined calculation that enhances the influence of the bounding boxes with low first confidence scores; and determining the reliability of the labels for each image based on the second confidence scores.

[0012] Furthermore, the position error score can be calculated based on at least one of the following: the IoU (Intersection-over-Union) value between the bounding box of the predicted value and the bounding box included in the corresponding label, and the difference in center coordinates.

[0013] Furthermore, the labels are for the purpose of segmentation, the predicted values ​​include information on the probability values ​​corresponding to each of the classes for each pixel of the image, and the steps for evaluating the confidence of the labels may include: preprocessing the predicted values ​​by extracting and storing the probability values ​​of the correct labels from the predicted values ​​for each pixel of each image; calculating a first confidence score per pixel through the preprocessed predicted values; calculating a second confidence score per image for each image through a predetermined operation that enhances the influence of pixels with low first confidence scores; and determining the confidence of the labels for each image based on the second confidence scores.

[0014] The method may further include the steps of: obtaining a second image set comprising a plurality of second images, each containing a second label corresponding to at least one of the aforementioned classes; dividing the second image set into K second unit sets; inputting each of the second unit sets into the corresponding evaluation model to output a second predicted value corresponding to each of the plurality of aforementioned second images; and evaluating the confidence level of the second label by comparing the second predicted value with the second label of the corresponding second image.

[0015] The process may further include the steps of: obtaining a second image set comprising a plurality of second images, each containing a second label corresponding to at least one of the aforementioned classes; inputting the second image set into K evaluation models and ensembling the output values ​​to output a second predicted value corresponding to each of the plurality of aforementioned second images; and evaluating the confidence level of the second label by comparing the second predicted value with the second label of the corresponding second image.

[0016] An embodiment of this application provides a computer program for object detection. The computer program may be stored on a recording medium to execute the label reliability evaluation method according to an embodiment of this application.

[0017] An embodiment of the present application provides a label confidence evaluation device. The device includes at least one processor and a memory for storing a program executable by the processor, wherein the processor, by executing the program, acquires an image set consisting of a plurality of images, each containing a label corresponding to at least one class, divides the image set into K unit sets, selects one of the unit sets as a verification set, and then repeats the process of training a predetermined network function using the remaining K-1 unit sets as training data K times to generate K evaluation models, inputs the verification sets into the corresponding evaluation models, outputs predicted values ​​corresponding to each of the plurality of images, and evaluates the confidence of the labels by comparing the predicted values ​​with the labels of the corresponding images. [Effects of the Invention]

[0018] According to the embodiments of this application, the confidence level of labels can be scored to help identify erroneous labels, thereby ultimately improving the predictive accuracy of the model in an objective and reliable manner by identifying, reviewing, and correcting data samples that are likely to be erroneous.

[0019] The effects that can be obtained from the embodiments of this application are not limited to those mentioned above, and any other effects not mentioned will be clearly understood by a person with ordinary skill in the art to which this application pertains from the following description. [Brief explanation of the drawing]

[0020] [Figure 1] This is a block diagram of a label reliability evaluation device according to an embodiment of the present application. [Figure 2]It is a flowchart of a label reliability evaluation method according to an embodiment of the present application. [Figure 3] It is a drawing for explaining the generation process of an evaluation model and predicted values in the label reliability evaluation method according to an embodiment of the present application. [Figure 4] It is an embodiment of the S250 stage in FIG. 3. [Figure 5] It is an embodiment of the S250 stage in FIG. 3. [Figure 6a] It is a drawing for explaining the process of determining the reliability of a label for object detection in the label reliability evaluation method according to an embodiment of the present application. [Figure 6b] It is a drawing for explaining the process of determining the reliability of a label for object detection in the label reliability evaluation method according to an embodiment of the present application. [Figure 6c] It is a drawing for explaining the process of determining the reliability of a label for object detection in the label reliability evaluation method according to an embodiment of the present application. [Figure 7] It is an embodiment of the S250 stage in FIG. 3. [Figure 8] It is a flowchart of a label reliability evaluation method according to an embodiment of the present application.

Embodiments for Carrying Out the Invention

[0021] To more fully understand the drawings cited from the present application, a brief description of each drawing is provided. Since the technical idea of the present application can be subjected to various modifications and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail. However, this is not intended to limit the technical idea of the present application to specific embodiments, but includes all modifications, equivalents, and alternatives included in the scope of the technical idea of the present application.

[0022] When it is determined that a specific description of related known technologies may obscure the gist of the present application, the detailed description thereof is omitted in explaining the technical idea of the present application.

[0023] The terms used herein are for illustrative purposes only and are not intended to limit and / or restrict this application. Singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, numbers used herein (e.g., 1st, 2nd, etc.) are merely identifiers to distinguish one component from another.

[0024] In this specification, when a part is said to be connected to another part, this includes not only direct connections but also indirect connections through other components in between. Furthermore, when a part is said to contain some component, this does not mean excluding other components, but rather that it may further contain other components, unless otherwise stated.

[0025] Furthermore, in this application, the term “or” is intended to mean an implicational “or” rather than an exclusive “or.” That is, where not distinctly specified or not clearly defined in context, “X utilizes A or B” is intended to mean one of the natural implicational substitutions. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, “X utilizes A or B” can apply to any of the aforementioned cases. Also, the term “and / or” as used herein refers to and includes all possible combinations of one or more of the related configurations listed.

[0026] Furthermore, terms such as "~part," "~device," "~child," and "~module" described in this application refer to a unit that processes at least one function or operation, which can be embodied in hardware, software, or a combination of hardware and software, such as a processor, microprocessor, microcontroller, CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerate Processor Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

[0027] Furthermore, we aim to clarify that the classification of configurations in this application is merely a classification based on the main function that each configuration is responsible for. That is, two or more configurations described below may be integrated into a single configuration, or a single configuration may be divided into two or more configurations based on more subdivided functions. Moreover, each configuration described below may additionally perform some or all of the functions that other configurations are responsible for, in addition to the main function that it is responsible for, and of course, some of the main functions that each configuration is responsible for may be exclusively performed by other configurations.

[0028] In this specification, the term "artificial intelligence learning model" may be used in the same sense as "artificial intelligence model," "computational model," or "machine learning model." Artificial intelligence learning models can be trained using a variety of algorithms, such as decision trees, random forests, Gaussian naive Bayes, k-nearest neighbors, Ada Boost, support vector machines, voting, bagging, neural networks, and deep learning. However, they are not limited to these.

[0029] Artificial intelligence learning models can be trained using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Learning an artificial intelligence model can be a process in which the model applies knowledge necessary for it to perform a specific action.

[0030] When algorithms such as neural networks and deep learning are applied to an artificial intelligence learning model, the AI ​​learning model can be referred to as a network function. The term "network function" can be used interchangeably with "neural network." A neural network can consist of a set of interconnected computational units, which can generally be referred to as nodes. Such nodes may also be referred to as neurons. A neural network consists of at least one node, and nodes can be interconnected by one or more links.

[0031] Neural networks can include, but are not limited to, deep neural networks (DNNs). Deep neural networks can include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siam networks, and generative adversarial networks (GANs).

[0032] The embodiments of this application will be described in detail below.

[0033] Figure 1 is a block diagram of a label reliability evaluation device according to an embodiment of this application.

[0034] Referring to Figure 1, the label reliability evaluation device 100 may include a communication unit 110, an input unit 120, a memory 130, and a processor 140.

[0035] The communication unit 110 can receive or transmit data from internal and external sources. The communication unit 110 may include a wired / wireless communication unit. If the communication unit 110 includes a wired communication unit, it may include one or more components that enable communication through a local area network (LAN), wide area network (WAN), value-added network (VAN), mobile radio communication network, satellite communication network, and combinations thereof. If the communication unit 110 includes a wireless communication unit, it can send and receive data or signals wirelessly using cellular communication, wireless LAN (e.g., Wi-Fi), etc. In this embodiment, the communication unit 110 can send and receive data or signals with an external device or external server under the control of the processor 140.

[0036] The input unit 120 can receive various user commands through external operations. For this purpose, the input unit 120 can include or be connected to one or more input devices. For example, the input unit 120 can be connected to various input interfaces such as a keypad or mouse to receive user commands. For this purpose, the input unit 120 may include interfaces such as a Thunderbolt in addition to a USB port. Furthermore, the input unit 120 can include or be connected to various input devices such as a touchscreen or buttons to receive external user commands.

[0037] Memory 130 can store programs and / or program instructions for the operation of processor 140, and can temporarily or permanently store input / output data. Memory 130 may include at least one type of storage medium from among flash memory, hard disk, multimedia card micro, card type memory (e.g., SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory, magnetic disk, and optical disk.

[0038] Furthermore, the memory 130 can store a variety of network functions and algorithms, as well as a variety of data, programs (one or more of which are instructions), applications, software, instructions, and code for driving and controlling the device 100.

[0039] The processor 140 can control the overall operation of the device 100. The processor 140 can execute one or more programs or software stored in the memory 130. The processor 140 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor 140 on which the method according to the embodiment of this application is performed.

[0040] In this embodiment, the processor 140 acquires an image set consisting of multiple images, each containing a label corresponding to at least one class, divides the image set into K unit sets, selects one of the unit sets as a validation set, and then repeats the process of training a predetermined network function using the remaining K-1 unit sets as training data K times to generate K evaluation models, inputs the validation sets into the corresponding evaluation models to output predicted values ​​corresponding to each of the multiple images, and evaluates the reliability of the labels by comparing the predicted values ​​with the labels of the corresponding images.

[0041] In this embodiment, the labels are for the image classification task, the predicted values ​​contain information about the probability values ​​corresponding to each class, and the processor 140 determines self-confidence based on the probability values ​​corresponding to the correct labels extracted from the predicted values ​​for each image. The confidence level of the image-specific labels can then be determined by using the difference between the self-confidence and the highest probability value among the other classes excluding the correct labels.

[0042] In this embodiment, the processor 140 can calculate a normalized margin by normalizing the difference between the self-confidence level and the highest probability value among the other classes excluding the correct label, and then determine the confidence level of the label based on the normalized margin.

[0043] In the embodiment, the label is for object detection, and the predicted value includes information on the coordinates of at least one bounding box, the class and probability value that is predicted to be present in each bounding box, and the processor 140 can evaluate the reliability of the label through a confidence score calculated based on at least one of the following: a bad location score based on the degree of mismatch between the bounding box of the predicted value and the bounding box included in the corresponding label for each image; an overlooked score when the bounding box of the predicted value and the corresponding label are missing; and a swapped score when the class of the predicted value is different from the class of the corresponding label.

[0044] In the embodiment, the processor 140 calculates a first confidence score per bounding box for bounding boxes included in at least one of the predicted value and label based on at least one of the position error points, overlooked points and class replacement points for each image, calculates a second confidence score per image for each image through a predetermined operation that enhances the influence of bounding boxes with low first confidence scores, and determines the confidence level of the label for each image based on the second confidence score.

[0045] In the embodiment, the processor 140 can calculate the number of position error points based on at least one of the IoU (Intersection-over-Union) value and the difference in center coordinates between the bounding box of the predicted value and the bounding box contained in the corresponding label.

[0046] In this embodiment, the labels are for the object segmentation operation, the predicted values ​​include information on the probability values ​​corresponding to each class for each pixel of the image, the processor 140 preprocesses the predicted values ​​by extracting and storing the probability values ​​of the correct labels from the predicted values ​​for each pixel for each image, calculates a first confidence score per pixel through the preprocessed predicted values, calculates a second confidence score per image for each image through a predetermined operation that enhances the influence of pixels with low first confidence scores, and can determine the confidence level of the labels for each image based on the second confidence score.

[0047] In this embodiment, the processor 140 acquires a second image set consisting of multiple second images, each containing a second label corresponding to at least one class; divides the second image set into K second unit sets; inputs each second unit set into its corresponding evaluation model to output a second predicted value corresponding to each of the multiple second images; and compares the second predicted value with the second label of the corresponding second image to evaluate the confidence level of the second label.

[0048] In this embodiment, the processor 140 acquires a second image set consisting of multiple second images, each containing a second label corresponding to at least one class. By inputting each second image set into K evaluation models and ensembling the output values, the processor can output a second predicted value corresponding to each of the multiple second images. The confidence level of the second label can then be evaluated by comparing the second predicted value with the second label of the corresponding second image.

[0049] The apparatus shown in Figure 1 is illustrative, and various configurations can be applied according to the embodiments of this application.

[0050] Figure 2 is a flowchart of the label reliability evaluation method according to an embodiment of this application, and Figure 3 is a diagram illustrating the evaluation model and the process of generating predicted values ​​in the label reliability evaluation method according to an embodiment of this application.

[0051] At the S210 stage, an image set 310 can be obtained, which consists of multiple images, each containing a label corresponding to at least one class.

[0052] Here, at least one class may be defined to perform at least one task on an image. For example, the task could include image classification, object detection, or object segmentation.

[0053] Furthermore, a label is assigned as ground truth to multiple images for machine learning and / or deep learning purposes, and may be one of the classes defined for the aforementioned tasks.

[0054] At step S220, the image set 310 can be divided into K unit sets. That is, the images contained in the image set 310 can be divided into the same number of units to generate K unit sets. For example, if the image set 310 contains 1000 images, the image set 310 can be divided into four unit sets of 250 images each.

[0055] The divided unit sets can be designated as validation sets or training sets for cross-validation, as detailed below.

[0056] In step S230, after selecting one of the unit sets as the validation set, the process of training the network function with the remaining K-1 unit sets using training data is repeated K times to generate K evaluation models.

[0057] TIFF2026076140000002.tif45166

[0058] TIFF2026076140000003.tif23166

[0059] TIFF2026076140000004.tif20166

[0060] At the S250 level, the confidence level of each image label can be evaluated by comparing the predicted values ​​with the corresponding image labels.

[0061] In the embodiment, step S250 can be performed in different ways depending on the type of label, i.e., the type of work to be performed. The reliability evaluation method for work-specific labels will be described in detail below with reference to Figures 3, 5, and 7.

[0062] The method illustrated in Figure 2 is illustrative, and various configurations can be applied according to the embodiments of this application.

[0063] Figure 4 shows an embodiment of step S250 in Figure 3.

[0064] Referring to Figure 4, this could be for image classification work, and the predicted values ​​corresponding to multiple images may include information (e.g., a probability vector for the class) about the probability value that each image belongs to in each class.

[0065] In the examples, the confidence of the labels in the classification task can be evaluated based on at least one of self-confidence and normalized margin.

[0066] If the confidence level of a label is assessed through self-confidence and normalization margins, the S250 scale may include the S410-S430 scales.

[0067] At stage S410, self-confidence can be determined based on the probability values ​​corresponding to the correct labels extracted from the predicted values ​​for each image. In this case, self-confidence is a probability value and can have a value between 0 and 1.

[0068] For example, if the predicted probability vector is [0.1, 0.7, 0.2] and the second class is the correct label, the self-confidence score may be determined to be 0.7. A lower self-confidence score indicates a lower probability that the model will predict the image as the correct label, and therefore a higher probability that the label is incorrect.

[0069] At the S420 stage, the difference between the self-confidence level and the highest probability value among the other classes (excluding the correct label) can be calculated, and the normalized margin can be calculated based on this difference.

[0070] For example, if the probability vector of the predicted values ​​is [0.1, 0.7, 0.2] and the second class is the correct label, then the self-confidence is 0.7 as mentioned above, the probability of the highest class excluding this is 0.2, and the difference between these is 0.5.

[0071] By applying the following mathematical formula 1 to normalize such differences to a range between 0 and 1, the normalized margin can be calculated.

[0072] [Mathematical formula 1]

[0073]

number

[0074] Here, (m i This refers to the difference between the self-confidence level and the highest probability value among the other classes, excluding the correct label.

[0075] At the S430 stage, the confidence level of each image label can be determined based on the normalization margin.

[0076] A low normalization margin score indicates that the probability of an image belonging to a labeled class is similar to the probability of it belonging to another class. In other words, it means the model has low confidence in the label of the image in question, and can be evaluated as being likely to be an incorrect label.

[0077] The method illustrated in Figure 4 is illustrative, and various configurations can be applied in the embodiments of this application. For example, although Figure 4 details the calculation of a normalized margin based on self-confidence and then the evaluation of confidence, the embodiment may perform the evaluation of label confidence using only self-confidence.

[0078] Figure 5 shows an embodiment of step S250 in Figure 3, and Figures 6a to 6c are diagrams illustrating the process of determining the reliability of an object detection label using the label reliability evaluation method according to the embodiment of this application.

[0079] Referring to Figures 5 and 6a-6c, the labels are for object detection, and the predicted values ​​corresponding to multiple images may include information about the coordinates of at least one bounding box that detected at least one object in the image, the class of object predicted to be present in each bounding box, and the probability value.

[0080] In the embodiment, the confidence level of the label for object detection can be calculated based on at least one of the following: bad location score, overlooked score, and swapped class score.

[0081] Here, the position error score can refer to a score based on the degree of discrepancy between the bounding box of the predicted value and the bounding box of the corresponding label. In other words, the position error score is a score assigned to a label that reflects cases where the label is not accurately depicted in position on an object (e.g., a person), as shown in Figure 6a(a).

[0082] Furthermore, the oversight score may refer to a score that reflects the case where the bounding box for the predicted value and the corresponding label is missing. In other words, the oversight score is a score assigned to the predicted value that reflects the case where the bounding box included in the predicted value (the bounding box for "dog") is missing from the image label, as shown in Figure 6b.

[0083] Furthermore, class replacement points can refer to points that reflect cases where the predicted value's class differs from the corresponding label's class. In other words, class replacement points are points assigned to a label that reflect cases where the label's class changes to another class that is not correct (for example, when an airplane is incorrectly labeled as a chair), as shown in Figure 6c.

[0084] The S250 stage can include stages S510 through S530.

[0085] At stage S510, a first confidence score per bounding box can be calculated based on the bounding box position error score, oversight score, and / or class replacement score included in the label and / or predicted value for each image.

[0086] In the embodiment, the position error points can be calculated by the similarity between bounding boxes based on at least one of the following: the IoU (Intersection-over-Union) value between the bounding box of the predicted value and the bounding box included in the corresponding label, and the difference in center coordinates.

[0087] IoU can be calculated as the ratio of the size of the overlapping area between the bounding box of the predicted value and the bounding box of the label, which refer to the same class, to the size of the total bounding box area, as shown in Figure 6a(b).

[0088] Furthermore, the difference in center coordinates between the bounding boxes of predicted values ​​and labels that refer to the same class can be calculated, for example, by the Gaussian kernel distance. By additionally considering the difference in center coordinates, it is possible to prevent identical points where IoU is 0 (i.e., there is no overlapping region of the bounding boxes).

[0089] Furthermore, in the examples, if a prediction with a high probability exists in a position without a label, a low oversight score can be assigned, assuming that the prediction in question is an overlooked label.

[0090] Furthermore, in the embodiment, if a label overlaps with an existing label but has a high probability of being in a different class, it is assumed that the label's class was incorrectly specified. A lower class replacement score can be assigned if the probability is high, and a higher class replacement score if the probability is low.

[0091] The calculated position error score, oversight score, and class replacement score can be determined as the first confidence score for each bounding box.

[0092] At step S520, the second confidence score for each image can be calculated through a predetermined calculation that enhances the influence of the bounding box with the lowest first confidence score. In other words, if there are multiple bounding boxes, each assigned a first confidence score for an image, a predetermined calculation can be applied so that the score of the bounding box with the lowest first confidence score is more reflected in determining the second confidence score, which is the confidence score of the entire image.

[0093] In the embodiment, step S520 can be performed through softmin pooling. For example, the softmin vector of the first confidence score S of the bounding boxes present in the image can be calculated, and the second confidence score for each image can be calculated through the dot product of the softmin vector and the first confidence score S. Through this, bounding boxes with lower first confidence scores can be given a higher weight when calculating the score for each image.

[0094] In the embodiment, at step S520, a second confidence score per image can be calculated for each of the position error, oversight, and class replacement.

[0095] At the S530 stage, the confidence level of each image label can be determined based on the second confidence score.

[0096] For example, in the S530 stage, the confidence level of the final image-specific label can be determined by geometric mean of the image unit scores calculated for each of the three cases mentioned above (i.e., position error, oversight, and class swapping). Through geometric mean, the lowest second confidence score can be given more weight to the final image-specific confidence level.

[0097] The method illustrated in Figure 5 is illustrative, and various configurations can be applied according to the embodiments of this application.

[0098] Figure 7 shows an embodiment of step S250 in Figure 3.

[0099] Referring to Figure 7, the labels are for the segmentation process, and the predicted values ​​can contain information about the probability values ​​corresponding to each class for each pixel in the image.

[0100] The S250 stage can include stages S710 through S740.

[0101] In step S710, the predicted values ​​can be preprocessed by extracting and saving the probability value of the correct label from the predicted value for each pixel in each image. At this time, the extraction of the probability value of the correct label can be performed in the same way as the method for extracting the self-confidence level in step S410, which is detailed with reference to Figure 4.

[0102] In other words, when calculating the confidence score, all that is needed is the probability for the correct class; therefore, the probability values ​​for the remaining classes are unnecessary, and efficient memory management is possible by not saving them during preprocessing.

[0103] For example, if we assume an image size of 512x512 and we want to predict 10 classes, we would normally have to store all the probability values ​​for a size of 512x512x10. However, as with the S710 step system, if we only store the probability value for the corresponding class based on the correct label, only one value is stored per pixel, so we only need to store data for a size of 512x512, which reduces memory usage to one-tenth.

[0104] At the S720 stage, a first confidence score per pixel can be calculated using the pre-processed predicted values. For example, the self-confidence score extracted and stored for each pixel can be used as the first confidence score.

[0105] At step S730, a second confidence score per image can be calculated for each of the images through a predetermined calculation that enhances the influence of the pixels with low first confidence scores.

[0106] Step S730 can be performed through softmin pooling, similar to step S520, which is detailed with reference to Figure 5. For example, the softmin vector of the first confidence score S of the pixels present in the image can be calculated, and the second confidence score for each image can be calculated through the dot product of the softmin vector and the first confidence score S. Through this, pixels with lower first confidence scores can be given a higher weight when calculating the score for each image.

[0107] At the S740 level, the confidence level of the label for each image can be determined based on the second confidence score. For example, an image with a high second confidence score can be evaluated as having a high label confidence level.

[0108] The method illustrated in Figure 7 is illustrative, and various configurations can be applied according to the embodiments of this application.

[0109] Figure 8 is a flowchart of a label reliability evaluation method according to an embodiment of this application. Specifically, Figure 8 is a flowchart illustrating the process of evaluating the reliability of a label through a pre-generated evaluation model for a new image set including the label.

[0110] At step S810, a second image set can be obtained, consisting of multiple second images, each containing a second label corresponding to at least one class. At this time, the second labels may be used to perform the same tasks as the labels, as detailed in Figure 2. For example, these tasks may include image classification, object detection, and object segmentation.

[0111] At step S820, the second unit set can be input into a pre-generated evaluation model to output a second predicted value corresponding to each of the multiple second images.

[0112] In this case, the second prediction value can take different forms depending on the type of second label or the type of task to be performed. For example, if the second label is for image classification, the second prediction value may include information on the probability values ​​corresponding to each class. Also, for example, if the second label is for object detection, the second prediction value may include information on the coordinates of at least one bounding box, the class predicted to exist in each bounding box, and the probability values. Also, for example, if the second label is for object segmentation, the second prediction value may include information on the probability values ​​corresponding to each class for each pixel of the image.

[0113] In the embodiment, step S820 can be performed by dividing the second image set into K second unit sets. That is, the second image set is divided into the same number of second unit sets as the evaluation model generated through the method in Figure 2, and after inputting each into the corresponding evaluation model, multiple second predicted values ​​corresponding to the second image can be generated by integrating the predicted values ​​output from each evaluation model.

[0114] In the embodiment, step S820 can be performed through an ensemble of K evaluation models. For example, if the second image set is input to each of the K evaluation models without splitting, each evaluation model will output the same number of prediction values ​​as the number of second images contained in the second image set, and the final second prediction value can be obtained by ensembling these output values. Various methods known in the field, such as voting and bagging, can be applied to the ensemble.

[0115] At stage S830, the confidence level of the second label can be evaluated by comparing the second predicted value with the second label of the corresponding second image. The confidence level of the second label can be evaluated by applying the same method as described above, referring to Figures 4 to 7.

[0116] In other words, according to Method 800, assuming that the existing K evaluation models have been sufficiently well-trained and that the distribution of the new data is similar to that of the data on which the K evaluation models were trained, the confidence level for the new data can be calculated without additional training. Furthermore, highly reliable predictions are possible through an ensemble of the K evaluation models.

[0117] The method illustrated in Figure 8 is illustrative, and various configurations can be applied according to the embodiments of this application.

[0118] The methods according to embodiments of this application can be embodied in a program instruction form that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be specifically designed and configured for this application or may be publicly known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code produced by compilers, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0119] Furthermore, the methods relating to the disclosed embodiments may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0120] A computer program product may include a software program (S / W program) and a storage medium readable by the computer on which the S / W program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is distributed electronically through an electronic device manufacturer or an electronic marketplace (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the S / W program may be stored on or temporarily generated on a storage medium. In this case, the storage medium may be the manufacturer's server, the electronic marketplace's server, or a relay server temporarily storing the S / W program.

[0121] A computer program product may include the storage medium for the server or the client device in a system consisting of a server and client devices. Alternatively, if a third device (e.g., a smartphone) is connected to the server or client device via communication, the computer program product may include the storage medium for the third device. Alternatively, the computer program product may include the software program itself that is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0122] In this case, one of the server, client device, and third device can execute the computer program product to carry out the method according to the disclosed embodiment. Alternatively, two or more of the server, client device, and third device can execute the computer program product to carry out the method according to the disclosed embodiment in a distributed manner.

[0123] For example, a server (such as a cloud server or an artificial intelligence server) can execute a computer program product stored on the server and control a client device connected to the server in communication to perform the method according to the disclosed embodiment.

[0124] Although the embodiments have been described in detail above, the scope of the rights of this application is not limited thereto. Various modifications and improvements made by persons skilled in the art, utilizing the basic concepts of this application as defined in the following claims, also fall within the scope of the rights of this application.

Claims

1. A method for evaluating label confidence, The step of obtaining an image set consisting of multiple images, each containing a label corresponding to at least one class; The step of dividing the aforementioned image set into K unit sets; After selecting one of the aforementioned unit sets as the validation set, the process of training a predetermined network function using the remaining K-1 unit sets as training data is repeated K times to generate K evaluation models; A step of inputting the verification set into the corresponding evaluation model and outputting a predicted value corresponding to each of the multiple images; and, This step includes comparing the predicted values ​​with the corresponding labels of the images to evaluate the reliability of the labels. A method characterized by the following:

2. The aforementioned labels are for the purpose of image classification. The aforementioned predicted value includes information for the probability value corresponding to each of the aforementioned classes. The step of evaluating the reliability of the aforementioned label is: A step of determining self-confidence based on the probability value corresponding to the correct label extracted from the predicted values ​​corresponding to each image; and, This step includes determining the confidence level of the image-specific label by utilizing the difference between the self-confidence level and the highest probability value among the other classes excluding the correct label. The method according to claim 1.

3. The step of evaluating the reliability of the aforementioned label is: The process further includes the step of normalizing the difference between the self-confidence level and the highest probability value among the other classes excluding the correct label to a predetermined range and calculating a normalized margin. The step of determining the confidence level of the label is performed based on the normalization margin. The method according to claim 2.

4. The aforementioned label is for object detection purposes. The predicted value includes information on the coordinates of at least one bounding box, the class predicted to exist in each of the bounding boxes, and the probability value. The step of evaluating the reliability of the aforementioned label is: The process is carried out through a confidence score calculated based on at least one of the following for each image: a bad location score based on the degree of mismatch between the bounding box of the predicted value and the bounding box included in the corresponding label; an overlooked score when the bounding box of the label corresponding to the predicted value is missing; and a swapped score when the class of the predicted value differs from the class of the corresponding label. The method according to claim 1.

5. The step of evaluating the reliability of the aforementioned label is: A step of calculating a first confidence score per bounding box for the bounding box included in the predicted value and at least one of the labels, based on at least one of the position error score, the oversight score, and the class replacement score for each image; A step of calculating a second image-unit confidence score for each of the images through a predetermined calculation that enhances the influence of the bounding box, which has a low first confidence score; and, The step of determining the confidence level of the label for each of the images based on the second confidence score; The method according to claim 4.

6. The aforementioned position error points are The method according to claim 4, calculated based on at least one of the IoU (Intersection-over-Union) value and the difference in center coordinates between the bounding box of the predicted value and the bounding box included in the corresponding label.

7. The aforementioned labels are for the purpose of object segmentation. The predicted value includes information for the probability value corresponding to each of the classes for each pixel of the image, The step of evaluating the confidence level of the label involves preprocessing the predicted values ​​by extracting and storing the probability value of the correct label from the predicted values ​​for each pixel in each image; A step of calculating a first confidence score in pixels using the pre-processed predicted values; A step of calculating a second image-unit confidence score for each of the images through a predetermined operation that enhances the influence of the pixels with low first confidence scores; and, The step of determining the confidence level of the label for each of the images based on the second confidence score; The method according to claim 1.

8. A step of obtaining a second image set consisting of a plurality of second images, each containing a second label corresponding to at least one of the aforementioned classes; The step of dividing the aforementioned second image set into K second unit sets; A step of inputting the second unit set into the corresponding evaluation model and outputting a second predicted value corresponding to each of the multiple second images; and, The step further includes comparing the second predicted value with the second label of the corresponding second image to evaluate the reliability of the second label; The method according to claim 1.

9. A step of obtaining a second image set consisting of a plurality of second images, each containing a second label corresponding to at least one of the aforementioned classes; A step of inputting each of the second image sets into K of the evaluation models and ensembling the output values ​​to output a second predicted value corresponding to each of the multiple second images; and, The step further includes comparing the second predicted value with the second label of the corresponding second image to evaluate the reliability of the second label; The method according to claim 1.

10. A recording medium stored in order to perform the method according to any one of claims 1 to 9 A computer program characterized by the following features.

11. A label confidence evaluation device, At least one processor; and, Includes memory for storing programs that can be executed by the aforementioned processor, The processor, by executing the program, acquires an image set consisting of multiple images, each containing a label corresponding to at least one class; divides the image set into K unit sets; selects one of the unit sets as a validation set; and then repeats the process of training a predetermined network function using the remaining K-1 unit sets as training data K times to generate K evaluation models; inputs the validation sets into the corresponding evaluation models to output predicted values ​​corresponding to each of the multiple images; and evaluates the confidence level of the labels by comparing the predicted values ​​with the labels of the corresponding images. A device characterized by the following features.

Images