Method for removing or relabeling wrong labels in dataset

Abstract

Disclosed, according to some embodiments of the present disclosure, is a method for removing or relabeling wrong labels in a dataset, performed by a computing device comprising at least one processor. The method may include the steps of: extracting features from a first dataset by using a pre-trained first model; performing a labeling operation of determining labels for the features by using a pre-trained second model, so as to determine first probability values; determining first predicted labels on the basis of the first probability values; repeating the labeling operation by using the first predicted labels, so as to determine second predicted labels; and removing or relabeling wrong labels in the first dataset by using the first predicted labels, the second predicted labels, and current labels included in the first dataset.

Description

How to remove or relabel anomalies on a dataset

[0001] The present disclosure relates to a method for removing an overlabeling on a data set or performing a relabeling on a data set.

[0002] Deep neural networks (DNNs) come in a variety of architectures and are used in a wide range of fields, including computer vision and language processing. Recently, the creation of large-scale datasets has spurred the development of DNNs. However, these datasets often contain noisy labels due to errors in annotation, crowdsourcing, or web scraping. Because generating high-quality datasets requires a lengthy and complex process, the generation of noisy data is inevitable.

[0003] When a dataset contains noisy labels, the model first trains on clean labels, followed by the noisy labels, a phenomenon known as the memorization effect. This phenomenon causes models trained on datasets containing noisy labels to have low loss for normal data and high loss for mislabeled data.

[0004] Therefore, there is a need to develop a technique to remove mislabeling from a data set or to perform relabeling on the data set.

[0005] The present disclosure is conceived in response to the aforementioned background technology, and aims to provide a method for removing mislabeling of a data set or performing relabeling of a data set.

[0006] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0007] According to one embodiment of the present disclosure for solving the above-described problem, a method for performing removal or re-labeling of a data set performed by a computing device including at least one processor may include the steps of: extracting a feature from a first data set using a pre-learned first model; determining a first probability value (soft label) by performing a labeling operation for determining a label for the feature using a pre-learned second model; determining a first predicted label based on the first probability value; determining a second predicted label by repeatedly performing the labeling operation using the first predicted label; and performing removal or re-labeling of the mislabeling for the first data set using at least one of the first predicted label, the second predicted label, and a current label included in the first data set.

[0008] The step of performing removal or relabeling of mislabeling on the first data set using at least one of the first predicted label, the second predicted label, and the current label included in the first data set may include the steps of: generating a first transition matrix using the first predicted label and the current label, and generating a second transition matrix using the second predicted label and the current label; and performing removal or relabeling of mislabeling on the first data set based on the first transition matrix and the second transition matrix.

[0009] The step of performing removal or relabeling of mislabeling on the first data set using at least one of the first predicted label, the second predicted label, and the current label included in the first data set may include: determining a first score using the first predicted label and the current label, and determining a second score using the second predicted label and the current label; determining a first threshold for a noise ratio using the first transpose matrix and the second transpose matrix; and performing removal or relabeling of mislabeling on the first data set based on the first threshold, the first score, the second score, the first transpose matrix, the second transpose matrix, and the current label.

[0010] The above first threshold value is calculated by mathematical expressions 10 and 11, and mathematical expression 10 is And, the above mathematical expression 11 is, and above is the first threshold, and is the first transpose matrix, and is the second transpose matrix, c is a specific class, and is a threshold used when the noise type of a specific class is symmetrical, and the above is a threshold used when the noise type of a specific class is asymmetric, and the above is the proportion of normal data among the data included in the data set related to the second transpose matrix, and is the proportion of normal data among the data included in the data set related to the first transpose matrix, and is the transpose matrix, and the above is the ratio of normal data to symmetric noise, and can be the ratio of the second label to the asymmetric noise.

[0011] The step of performing removal or relabeling of mislabeling on the first data set using the first predicted label, the second predicted label, and the current label included in the first data set may include: determining a reevaluation label based on the first threshold, the first score, the second score, the first transpose matrix, the second transpose matrix, and the current label; determining whether the first predicted label and the reevaluation label are the same; determining at least one first label classified into a first class included in the first predicted label when the first predicted label and the reevaluation label are the same; and performing removal or relabeling of mislabeling on the first data set when it is determined that a plurality of labels adjacent to the at least one first label are classified into the first class.

[0012] The technical solutions obtainable in the present disclosure are not limited to the solutions mentioned above, and other solutions not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0013] According to some embodiments of the present disclosure, a method is provided that can improve the accuracy and speed of an operation of removing an overlabeling on a data set or performing a relabeling on a data set.

[0014] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0015] Various aspects are now described with reference to the drawings, wherein like reference numerals are used to refer to similar elements generally. In the following examples, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate the description of one or more aspects.

[0016] FIG. 1 is a block diagram illustrating an example of a computing device according to some embodiments of the present disclosure.

[0017] FIG. 2 is a flowchart illustrating an example of a method by which a computing device performs removal or relabeling of an overlabeling for a first data set according to some embodiments of the present disclosure.

[0018] Figure 3 is a diagram illustrating an example of the k-NN algorithm.

[0019] FIG. 4 is a flowchart illustrating an example of a method by which a computing device performs removal or relabeling of an overlabeling for a first data set according to some embodiments of the present disclosure.

[0020] FIG. 5 is a flowchart illustrating an example of a method by which a computing device performs removal or relabeling of an overlabeling for a first data set according to some embodiments of the present disclosure.

[0021] FIG. 6 is a diagram illustrating an example of an algorithm performed by a computing device according to some embodiments of the present disclosure.

[0022] Figure 7 shows the results of an experiment in which overlabeling was removed or relabeled using the CIFAR10 data set and the CIFAR100 data set.

[0023] Figure 8 shows the results of an experiment in which overlabeling was removed or relabeled using the Clothing1M data set.

[0024] Figure 9 shows the experimental results showing the time required to remove or relabel overlabeling using the CIFAR10 data set, the CIFAR100 data set, and the Clothing1M data set.

[0025] Figure 10 is a diagram illustrating an example of an algorithm for a training method utilizing PES.

[0026] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0027] Terms such as "first," "second," "A," and "B" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component." The term "and / or" includes a combination of a plurality of related items described herein or any of a plurality of related items described herein.

[0028] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0029] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0031] In the present disclosure, a computing device can obtain a data set on which labeling has been performed. Labeling may be a process of processing data into a form that allows artificial intelligence to learn on its own. The computing device can label the data set using a pre-trained model. The computing device can compare the predicted labels generated by labeling the data set with the current labels (referring to labels currently assigned to the data) contained in the data set. Based on the comparison results, the computing device can remove mislabeling or relabel the data set. Below, a method for removing mislabeling or relabeling a data set by a computing device will be described with reference to FIGS. 1 to 6 .

[0032] FIG. 1 is a block diagram illustrating an example of a computing device according to some embodiments of the present disclosure.

[0033] Referring to FIG. 1, a computing device (100) may include a control unit (110), a storage unit (120), and a communication unit (130). However, the above-described components are not essential for implementing the computing device (100), and thus the computing device (100) may have more or fewer components than the components listed above.

[0034] The computing device (100) may include any type of computer system or computer device, such as, for example, a microprocessor, a mainframe computer, a digital processor, a portable device, or a device controller.

[0035] A computing device (100) may achieve desired system performance by utilizing a combination of typical computer hardware (e.g., devices that may include a computer processor, memory, storage, input devices and output devices, and other components of conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause the computing device to function in a particular manner).

[0036] The control unit (110) can typically process the overall operation of the computing device (100). The control unit (110) can process signals, data, information, etc. input or output through components of the computing device (100) or run application programs stored in the storage unit (120), thereby providing or processing appropriate information or functions to the user.

[0037] The control unit (110) may be composed of one or more cores and may include a processor for data analysis, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0038] In the present disclosure, the control unit (110) may extract features from a first data set using a pre-trained first model. Here, the first model may be a pre-trained model to extract features from a first data set including image data. The first model may be a model including a convolutional neural network (CNN), a recurrent neural network (RNN), an auto encoder, a generative adversarial network (GAN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a siamese network, a generative adversarial network (GAN), stable diffusion, neural radiance fields (NeRF), Nerfies, DragGan, or a PES.

[0039] The control unit (110) can obtain a pre-trained first model and extract features from a first data set. The first data set may be a data set including at least one of a clean data set and a noisy data set. The clean data set may be a data set without labeling errors. The noisy data set may be a data set including data that has been incorrectly labeled. The noisy data set may include normally labeled data and mislabeled data. The control unit (110) may perform a labeling operation to determine a label for a feature extracted through the first model using a pre-trained second model. By performing the labeling operation, the control unit (110) may determine a first probability value referred to as a soft label. The second model may be a model based on the k-Nearest Neighbor (k-NN) algorithm. The k-NN algorithm may be an algorithm that performs classification under the premise that data with similar characteristics belong to similar categories. The first probability value may be a value determined in such a way that the label corresponding to the correct answer for specific data has the highest probability, while classes that are not the correct answer also share small probabilities. The control unit (110) may repeat the labeling operation based on the first probability value. The control unit (110) may determine the second probability value through repetition of the labeling operation. The control unit (110) may perform removal or relabeling of mislabeling for the first data set using the first probability value, the second probability value, and the first data set. Hereinafter, an example of a method in which the control unit (110) performs removal or relabeling of mislabeling for the first data set will be described with reference to FIGS. 2 to 6.

[0040] The storage unit (120) may include memory and / or a permanent storage medium. The memory may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0041] In the present disclosure, a clean data set or a noise data set, etc. may be stored in the storage unit (120).

[0042] FIG. 2 is a flowchart illustrating an example of a method by which a computing device performs removal or relabeling of an overlabeling for a first data set according to some embodiments of the present disclosure.

[0043] Referring to FIG. 2, the control unit (110) of the computing device (100) can extract features from a first data set using a pre-trained first model (S110). The first model may be a pre-trained model to extract features from a first data set including image data. According to one embodiment, the image data may be an augmented image. For example, the image data may be data that has undergone augmentation, such as horizontal flipping, resizing, normalizing, color adjustment, or illumination adjustment. The first model for extracting features from the first data set may be a conventionally known model. The control unit (110) can extract features from the first data set by obtaining the pre-trained first model to extract features. Although the first data set is exemplified as image data, it will be understood by those skilled in the art that the type of data is not limited thereto.

[0044] The control unit (110) can determine a first probability value referred to as a soft label by performing a labeling operation for determining a label for a feature using a pre-learned second model (S120). The second model may be a model based on the k-Nearest Neighbor (k-NN) algorithm. The k-NN algorithm may be an algorithm that performs classification under the premise that data with similar characteristics belong to similar categories. The first probability value may be a value determined in such a way that the label corresponding to the correct answer for the target data has the highest probability, while classes that are not the correct answer also share a small probability. The second model may label samples included in the first data set by class.

[0045] A description of the k-NN algorithm and its applications can be easily obtained by those skilled in the art through non-patent literature 1 [Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training a model to predict. In International Conference on Machine Learning, pages 27412-27427. PMLR, 2022.]

[0046] The control unit (110) can determine a first predicted label based on the first probability value (S130). The first predicted label may be a result of labeling the features of a sample included in the first data set.

[0047] According to one embodiment, the control unit (110) can determine the first prediction label based on mathematical expressions 1, 2, and 3, which are performed repeatedly while m sequentially increases from 1 to M.

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] M is the number of epochs, k is the number of nearest neighbors, is a noisy dataset, is a feature extractor of the pre-trained first model, is a random augmentation function, silver The k-NN model learned using As a result of the prediction, is similarity, is the index, is the first probability value, is the result of soft voting using the first probability value, is the operation step of k-NN that finds the index of the k-nearest neighbor and finds the similarity between the input and the k-nearest neighbor. is the operation step of k-NN that generates probability values ​​(soft labels) for v and i, can be the first predicted label.

[0054] According to one embodiment, can be implemented as in mathematical formula 4.

[0055]

[0056] is a label Label from is one-hot encoding for , v is the similarity, and i can be an index.

[0057] The control unit (110) can determine the second prediction label by repeatedly performing a labeling operation using the first prediction label (S140).

[0058] Specifically, the control unit (110) can determine the second probability value by performing a labeling operation using the first predicted label through a pre-learned second model. The control unit (110) can determine the second predicted label based on the second probability value.

[0059] More specifically, the control unit (110) can determine the second prediction label based on mathematical expressions 5 and 6.

[0060]

[0061]

[0062] Here, is the second predicted label, may be the second probability value.

[0063] According to one embodiment, the control unit (110) may perform the labeling operation twice. In other words, the control unit (110) may perform the labeling operation twice through steps S120 and S140. Depending on the embodiment, the control unit (110) may utilize the first predicted label or the second predicted label, and depending on the embodiment, the labeling may be performed once or both times.

[0064] Specifically, the k-NN algorithm suffers from a locality problem, in that it is sensitive to the values ​​of neighboring nodes. Consequently, when performing a single run of the k-NN algorithm at a high noise level, there are cases where samples that should be labeled as class 1 are labeled as class 2. This can also be problematic when utilizing the transpose matrix, which utilizes the results of the k-NN algorithm. For convenience of explanation, reference may be made to Figure 3.

[0065] Figure 3 is a diagram illustrating an example of the k-NN algorithm.

[0066] Referring to FIG. 3, when the k-NN algorithm is iterated 0 times (Iteration 0), the four noisy samples in the center of the samples with a high noise ratio can be labeled as class 2. However, the four noisy samples in the center may be samples that should be labeled as class 1. This may be a problem caused by the locality problem of the k-NN algorithm and the high noise ratio of the samples. When the k-NN algorithm is iterated once (Iteration 1), three of the four noisy samples in the center can be labeled from class 2 to class 1. When the k-NN algorithm is iterated twice (Iteration 2), all samples can be labeled from class 2 to class 1. In other words, the control unit (110) can solve the locality problem of the k-NN algorithm by iterating the labeling operation twice.

[0067] Referring back to FIG. 2, the control unit (110) can perform removal or relabeling of overlabeling for the first data set using at least one of the first predicted label, the second predicted label, and the current label (meaning a label matching the first data) included in the first data set (S150).

[0068] Specifically, if the first predicted label and the re-evaluation label, which are the results of the first labeling operation, are identical, and the labels of the surrounding neighbors are identical during the re-evaluation, the control unit (110) may perform removal or re-labeling of the over-labeling for the first data set. The re-evaluation label may be a label that the control unit (110) re-evaluates based on at least one of the first predicted label, the second predicted label, and the current label. An example of a method by which the control unit (110) determines the re-evaluation label is described below with reference to FIG. 6.

[0069] According to one embodiment, the control unit (110) may generate a first transition matrix using the first predicted label and the current label. The control unit (110) may generate a second transition matrix using the second predicted label and the current label. The transition matrix may be a matrix representing the probability that a label corresponding to the correct answer is labeled with another label. The control unit (110) may generate the first and second transition matrices based on mathematical expression 7. The present invention can reduce the time required and solve the overestimation problem of other methods that utilize estimators by utilizing the transition matrix generated through simple label verification.

[0070]

[0071] Here, is the first predicted label or the second predicted label, may be a noisy label corresponding to the current label.

[0072] The first transpose matrix can be represented as a symmetric noise matrix or an asymmetric noise matrix. The second transpose matrix can be represented as a symmetric noise matrix or an asymmetric noise matrix. A symmetric noise matrix can be a matrix in which each class is equally orlabeled. For example, a symmetric noise matrix can be a matrix in which orlabeled classes are equally orlabeled, as shown in Table 1 below.

[0073] 0.60.20.20.20.60.20.20.20.6

[0074] An asymmetric noise matrix can be a matrix in which the labeled class is determined singly. For example, an asymmetric noise matrix can be a matrix in which the labeled class is single, as shown in Table 2 below.

[0075] 0.60.4000.60.40.400.6

[0076] The control unit (110) can perform removal or relabeling of overlabeling for the first data set based on the first transpose matrix and the second transpose matrix. Specifically, the control unit (110) can determine thresholds to be used for the first transpose matrix. For a symmetric matrix, if the noise ratio is higher than the first threshold value, or if the ratio of the second label of an asymmetric matrix (for example, 0.4, which is the ratio of labels to a single overlabeled class in the previous example) is higher than the second threshold value, the control unit (110) can perform removal or relabeling of overlabeling for the first data set using the result of repeatedly performing the labeling operation. Hereinafter, an example of a method for performing removal or relabeling of overlabeling for the first data set by the computing device (100) according to the present disclosure will be described with reference to FIG. 4.

[0077] FIG. 4 is a flowchart illustrating an example of a method by which a computing device performs removal or relabeling of an overlabeling for a first data set according to some embodiments of the present disclosure.

[0078] Referring to FIG. 4, the control unit (110) of the computing device (100) can determine a first score using a first predicted label and a current label, and can determine a second score using a second predicted label and a current label (S151).

[0079] Specifically, the control unit (110) can determine the first score and the second score based on mathematical expressions 8 and 9.

[0080]

[0081]

[0082] can be the first score or the second score. If this is the first score, is the first predicted label, may be the current label. If this is the second score, is the second predicted label, may be the current label. is currently labeled This may mean one-hot encoding.

[0083] The control unit (110) can determine a first threshold value for the noise ratio using at least one of the first transpose matrix and the second transpose matrix (S152).

[0084] Specifically, the control unit (110) can determine the first threshold value based on mathematical expressions 10 and 11.

[0085]

[0086]

[0087] Here, is the first threshold, is the first transpose matrix, can be the second transpose matrix. c can represent a specific class. is the threshold used when the noise type of a specific class is symmetrical, may be a threshold used when the noise type of a particular class is asymmetric. is the proportion of normal data among the data included in the data set related to the second transpose matrix, can represent the proportion of normal data among the data included in the data set related to the first transpose matrix.

[0088] is the transpose matrix, can be the ratio of normal data to symmetric noise. can be the ratio of the second label (e.g., Orabel) to the asymmetric noise.

[0089] At low noise ratios, the control unit (110) may need to perform fewer iterations of the labeling operation at low noise ratios because the iterations of the k-NN algorithm may overestimate the noise ratio. The control unit (110) may use a first threshold value to determine whether symmetric noise is present. If the noise ratio is higher, or the ratio of the second label to the asymmetric noise this If higher, the results of the second labeling operation can be used. If the noise ratio is higher, More normal ratio ( ) can also be understood as a low case.

[0090] The control unit (110) can perform removal or relabeling of overlabeling for the first data set based on at least one of the first threshold value, the first score, the second score, the first transpose matrix, the second transpose matrix, and the current label (S153).

[0091] More specifically, the control unit (110) may additionally determine a third score based on mathematical expression 12, and perform removal or relabeling of overlabeling for the first data set based on at least one of the first threshold value, the third score, and the current label.

[0092]

[0093] Specifically, the control unit (110) can determine a re-evaluation label based on at least one of a first threshold value, a third score, and a current label. If the first predicted label and the re-evaluation label are the same, and the labels of the k-nearest neighbors are the same as one during re-evaluation, the control unit (110) can perform removal or re-labeling of over-labeling for the first data set. Hereinafter, an example of a method in which the computing device (100) according to the present disclosure performs removal or re-labeling of over-labeling for the first data set will be described with reference to FIG. 5.

[0094] FIG. 5 is a flowchart illustrating an example of a method by which a computing device performs removal or relabeling of an overlabeling for a first data set according to some embodiments of the present disclosure.

[0095] Referring to FIG. 5, the control unit (110) of the computing device (100) can determine a reevaluation label based on at least one of a first threshold value, a first score, a second score, a first transpose matrix, a second transpose matrix, and a current label (S210).

[0096] Specifically, the control unit (110) may determine a re-evaluation label based on mathematical expressions 13 to 21. Mathematical expression 14 may be an operation in which the control unit (110) detects a high-scoring sample as a clean set and an operation in which the control unit (110) determines a high-scoring and balanced sample as a balanced clean set. Mathematical expression 14 may be performed iteratively until c sequentially increases from 1 to C. Mathematical expressions 15 to 21 relate to a series of operations in which the control unit (110) generates a damaged sample set and relabels the sample using the balanced clean set, and mathematical expression 17 may be performed iteratively until m sequentially increases from 1 to M.

[0097] A person skilled in the art can easily derive the operations related to a series of mathematical formulas based on the contents of Non-Patent Document 1 and the contents of the present invention.

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] Here, is the minimum number of samples to balance the clean set, is a set of samples labeled c, and I is the sorted third score of each is the index of, is a clean set of high-scoring samples with high scores, is a balanced clean set, is a function that outputs the indices of n and I, is a function that extracts the first element that satisfies the condition in the parentheses, is the number of samples labeled with c, N is the set of corrupted samples for testing, is a balanced clean label set, M is the number of epochs, is a reevaluation label, is the probability value related to the re-evaluation label, Is - k-NN soft label (probability value) obtained through norm, -norm can be the Euclidean distance used to measure the shortest distance between two points. The result of labeling the damaged set using the second model (k-NN algorithm) learned on the balanced clean set is as follows. is similarity, can be an index.

[0111] The control unit (110) can determine whether the first predicted label and the re-evaluation label are identical (S220). The control unit (110) can determine whether the first predicted label and the re-evaluation label represent the same class.

[0112] If the first predicted label and the re-evaluation label are the same, the control unit (110) can determine at least one first label classified as a first class included in the first predicted label (S230). Furthermore, if the control unit (110) determines that at least one first label and a plurality of adjacent labels are classified as a first class, the control unit (110) can perform removal or re-labeling of mislabeling for the first data set (S240). In other words, if the control unit (110) determines that at least one first label and a plurality of adjacent labels are not classified as a second class, the control unit (110) can perform removal or re-labeling of mislabeling for the first data set.

[0113] More specifically, the control unit (110) can perform removal or relabeling of the overlabeling for the first data set based on mathematical expressions 20 and 21.

[0114]

[0115]

[0116] Here, is a relabeled data set, may refer to an existing data set that does not correspond to an orabel.

[0117] According to the configuration described above with reference to FIGS. 2 to 5, the computing device (100) can repeatedly perform a labeling operation using the second model to determine a predicted label and a re-evaluation label. If the predicted label and the re-evaluation label are the same, and the labels of the surrounding neighbors are the same during re-evaluation, the computing device (100) can perform removal or re-labeling of mislabeling for the first data set. Since the logit function is used to perform re-labeling for the data set in the past, model learning was essentially required. On the other hand, the computing device (100) according to the present disclosure can perform removal or re-labeling of mislabeling without model learning by using only the features of the data set. Accordingly, the speed at which the computing device (100) performs removal or re-labeling of mislabeling can be improved.

[0118] FIG. 6 is a diagram illustrating an example of an algorithm performed by a computing device according to some embodiments of the present disclosure.

[0119] Referring to rows 1 through 4 of Figure 6, features of an arbitrarily increased image can be applied to k-NN. The results of k-NN can be aggregated into probability values ​​(soft voting).

[0120] Referring to lines 8 and 9, the first predicted label , prior similarity values and index can be reused in the iterative k-NN (step S140). Since is set to 2, the iterative k-NN loop can be omitted. In other words, the labeling operation by the second model can be performed only twice. The first predicted label , the second predicted label And the noise labels can be used to estimate two transpose matrices. The score for each k-NN iteration can be calculated with the soft and noise labels.

[0121] Referring to lines 10 to 13, the first transpose matrix at high noise ratio is used to solve the locality problem of k-NN. and first score Instead, the second transpose matrix and second score can be used. High scoring heads can be treated as clean sets.

[0122] Referring to lines 15 to 20, a balanced subset of the clean set can be generated for label correction. The corrupted set can be reevaluated by a k-NN trained on the balanced clean set.

[0123] Referring to lines 21 to 32, the predicted labels of the first k-NN This reevaluation label If the same and re-evaluated k-nearest neighbors consist of a single class, the sample belongs to the clean set. The label may be changed.

[0124] Below, the results of an experiment in which an overlabeling was removed or relabeled for a data set using a computing device (100) according to the present disclosure are described.

[0125] Figure 7 shows the results of an experiment in which overlabeling was removed or relabeled using the CIFAR10 data set and the CIFAR100 data set.

[0126] Experiments were conducted through comparisons with the LDMI, INCV, AUM, PES, and SimiFeat algorithms. The algorithms proposed through the method according to the present disclosure may be FastSimiFeat and FastSimiFeat+PES. PES may be a model used as the first model in the algorithm proposed through the method according to the present disclosure. The first model may be a model pretrained to extract features from a first data set including image data. The CIFAR10 and CIFAR100 data sets are artificially generated data sets. An algorithm for a training method utilizing PES is exemplarily presented through Fig. 10.

[0127] Referring to the CIFAR10 data set of Fig. 7, it can be confirmed that the error rate in 0.20 symmetric noise (0.20-Sym) is the lowest at 7.31% for FastSimiFeat+PES according to the present disclosure. It can be confirmed that the error rate in 0.60 symmetric noise (0.60-Sym) is the lowest at 13.02% for FastSimiFeat according to the present disclosure. It can be confirmed that the error rate in 0.20 asymmetric noise (0.20-Asym) is the lowest at 6.76% for FastSimiFeat+PES according to the present disclosure. It can be confirmed that the error rate in 0.45 asymmetric noise (0.45-Asym) is the lowest at 17.64% for FastSimiFeat+PES according to the present disclosure.

[0128] Referring to the CIFAR100 data set of Fig. 7, it can be confirmed that the error rate in 0.20 symmetric noise (0.20-Sym) is the lowest at 28.16% for FastSimiFeat+PES according to the present disclosure. It can be confirmed that the error rate in 0.60 symmetric noise (0.60-Sym) is the lowest at 39.78% for FastSimiFeat according to the present disclosure. It can be confirmed that the error rate in 0.20 asymmetric noise (0.20-Asym) is the lowest at 27.01% for FastSimiFeat+PES according to the present disclosure. It can be confirmed that the error rate in 0.45 asymmetric noise (0.45-Asym) is the lowest at 52.30% for FastSimiFeat+PES according to the present disclosure.

[0129] Figure 8 shows the results of an experiment in which overlabeling was removed or relabeled using the Clothing1M data set.

[0130] Experiments were conducted to compare the AUM and PES algorithms. The Clothing1M dataset was generated in a real-world environment.

[0131] Referring to Fig. 8, the AUM algorithm showed an error rate of 27.23% on the Clothing1M data set. The PES algorithm showed an error rate of 27.87% on the Clothing1M data set. The FastSimiFeat algorithm according to the present disclosure showed an error rate of 27.66% on the Clothing1M data set. In addition, the FastSimiFeat+PES algorithm according to the present disclosure was proven to have the best performance, showing an error rate of 26.88% on the Clothing1M data set.

[0132] Figure 9 shows the experimental results showing the time required to remove or relabel overlabeling using the CIFAR10 data set, the CIFAR100 data set, and the Clothing1M data set.

[0133] Referring to the CIFAR10 data set in FIG. 9, it can be confirmed that the time required to refine the CIFAR10 data set was the lowest at 0.04 h in the FastSimiFeat algorithm according to the present disclosure. Furthermore, it can be confirmed that the total time required to remove mislabeling or perform relabeling was the lowest at 0.59 h in the FastSimiFeat algorithm according to the present disclosure.

[0134] Referring to the CIFAR100 data set in FIG. 9, it can be confirmed that the time required to refine the CIFAR100 data set was the lowest at 0.06 h in the FastSimiFeat algorithm according to the present disclosure. Furthermore, it can be confirmed that the total time required to remove mislabeling or perform relabeling was the lowest at 0.82 h in the FastSimiFeat algorithm according to the present disclosure.

[0135] Referring to the Clothing1M data set of Fig. 9, it can be confirmed that the time required to refine the Clothing1M data set was the lowest at 0.93h in the FastSimiFeat algorithm according to the present disclosure.

[0136] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for performing removal or re-labeling of an over-labeling on a data set, the method being performed by a computing device including at least one processor, A step of extracting features from a first data set using a pre-trained first model; A step of determining a first probability value (soft label) by performing a labeling operation for determining a label for the feature using a pre-learned second model; A step of determining a first predicted label based on the first probability value; A step of determining a second prediction label by repeatedly performing the labeling operation using the first prediction label; and A step of performing removal or relabeling of overlabeling on the first data set using at least one of the first predicted label, the second predicted label, and the current label included in the first data set; Including, method.

2. In paragraph 1, The step of performing removal or relabeling of overlabeling for the first data set using at least one of the first predicted label, the second predicted label and the current label included in the first data set, A step of generating a first transition matrix using the first predicted label and the current label, and generating a second transition matrix using the second predicted label and the current label; and A step of performing removal or relabeling of overlabeling on the first data set based on the first transpose matrix and the second transpose matrix; Including, method.

3. In paragraph 2, The step of performing removal or relabeling of overlabeling for the first data set using at least one of the first predicted label, the second predicted label and the current label included in the first data set, A step of determining a first score using the first predicted label and the current label, and determining a second score using the second predicted label and the current label; A step of determining a first threshold value for noise ratio using the first transpose matrix and the second transpose matrix; and A step of performing removal or relabeling of overlabeling on the first data set based on the first threshold, the first score, the second score, the first transpose matrix, the second transpose matrix, and the current label; Including, method.

4. In paragraph 3, The above first threshold is, It is calculated by mathematical expressions 10 and 11, The above mathematical expression 10 is, And, The above mathematical expression 11 is, And, Above is the first threshold value, and the above is the first transpose matrix, and is the second transpose matrix, c is a specific class, and is a threshold value used when the noise type of a specific class is symmetrical, and the above is a threshold value used when the noise type of a specific class is asymmetric, and the above is the ratio of normal data among the data included in the data set related to the second transpose matrix, and is the proportion of normal data among the data included in the data set related to the first transpose matrix, and is the transpose matrix, and the above is the ratio of normal data to symmetric noise, and is the ratio of the second label to the asymmetric noise, method.

5. In paragraph 3, The step of performing removal or relabeling of overlabeling for the first data set using the first predicted label, the second predicted label, and the current label included in the first data set is as follows. A step of determining a reevaluation label based on the first threshold value, the first score, the second score, the first transpose matrix, the second transpose matrix and the current label; A step of determining whether the first predicted label and the re-evaluation label are the same; If the first predicted label and the re-evaluation label are the same, a step of determining at least one first label classified into the first class included in the first predicted label; and A step of performing removal or relabeling of the ora label for the first data set, if it is determined that at least one of the first labels and a plurality of adjacent labels are classified into the first class; Including, method.

Images