Adversarial Training Systems and Methods for Noise Labeling

Abstract

This invention relates to an adversarial training system and method for noisy labels. The system includes: a memory; and a processor configured to: train a first machine learning model based on first dataset labels; provide the trained first machine learning model with a second dataset to generate an updated second dataset including updated second dataset labels, determine a first difference between the updated second dataset labels and the second dataset labels; if the first difference is greater than a first threshold, train a second machine learning model based on the updated second dataset labels; provide the trained second machine learning model with the first dataset to generate an updated first dataset including updated first dataset labels, determine a second difference between the updated first dataset labels and the first dataset labels; and if the second difference is greater than a second threshold, train the first machine learning model based on the updated first dataset labels.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application Serial No. 62 / 783,150, filed December 20, 2018, entitled “Adversarial Training Method for Noise Tags,” the entire contents of which are hereby expressly incorporated by reference. Technical Field

[0003] One or more aspects of embodiments of this disclosure relate to adversarial training methods for noise labels. Background Technology

[0004] In recent years, the display industry has experienced rapid growth as new display technologies have been introduced to the market. Mobile devices, televisions, virtual reality (VR) headsets, and other displays are the constant forces driving displays to achieve higher resolutions and more accurate color reproduction. With the deployment of new display panel modules and production methods, surface defects have become more difficult to inspect using traditional methods.

[0005] The information above in the Background section is only used to enhance the understanding of the technical background and therefore should not be construed as an admission of the existence or relevance of prior art. Summary of the Invention

[0006] This summary is provided to introduce the selection of features and concepts of embodiments of this disclosure, which are further described below in a detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. One or more of the described features may be combined with one or more other described features to provide a feasible device.

[0007] An aspect of the exemplary embodiments of this disclosure relates to an adversarial training method for noise labels.

[0008] In some embodiments, an adversarial training system includes: a memory; and a processor configured to execute instructions stored in the memory, the instructions, when executed by the processor, causing the processor to: receive a first dataset including first dataset labels; receive a second dataset including second dataset labels; train a first machine learning model based on the first dataset labels; provide the trained first machine learning model with the second dataset to generate an updated second dataset including updated second dataset labels, the updated second dataset being generated by classifying the second dataset using the trained first machine learning model; determine a first difference between the updated second dataset labels and the second dataset labels; if the first difference is greater than a first threshold, train a second machine learning model based on the updated second dataset labels; provide the trained second machine learning model with the first dataset to generate an updated first dataset including updated first dataset labels, the updated first dataset being generated by classifying the first dataset using the trained second machine learning model; determine a second difference between the updated first dataset labels and the first dataset labels; and if the second difference is greater than a second threshold, train the first machine learning model based on the updated first dataset labels.

[0009] In some embodiments, the instructions further cause the processor to continue training the first machine learning model and the second machine learning model until a first difference is below a first threshold and a second difference is below a second threshold. In some embodiments, the first threshold and the second threshold are approximately equal to zero, wherein the first threshold is different from the second threshold. In some embodiments, the instructions further cause the processor to continue training the first machine learning model and the second machine learning model until both the first machine learning model and the second machine learning model produce the same dataset labeling results when applied to the updated second dataset and the updated first dataset. In some embodiments, the updated first dataset is generated by updating the labels of data samples from the first dataset that have been flipped to the new class after the trained second machine learning model is applied to the first dataset.

[0010] In some embodiments, the updated first dataset label differs from the first dataset label. In some embodiments, the first dataset is relabeled by a field human inspector based on the updated second dataset label to generate the updated first dataset. In some embodiments, the first machine learning model and the second machine learning model are classification algorithms. In some embodiments, the updated second dataset is generated by updating the labels of data samples from the second dataset that have been flipped to the new class after the trained first machine learning model has been applied to the second dataset. In some embodiments, the first dataset is classified or labeled by a field human inspector.

[0011] In some embodiments, an adversarial training system includes: a memory; and a processor configured to execute instructions stored in the memory, the instructions, when executed by the processor, causing the processor to: train a first machine learning model based on a first dataset label of a first dataset; provide the trained first machine learning model with a second dataset including a second dataset label to generate an updated second dataset including an updated second dataset label; train a second machine learning model based on the updated second dataset label; provide the trained second machine learning model with the first dataset to generate an updated first dataset including an updated first dataset label; and train the first machine learning model based on the updated first dataset label.

[0012] In some embodiments, the instructions further cause the processor to continue: determining a first difference between the updated second dataset label and the second dataset label, wherein if the first difference is greater than a first threshold, a second machine learning model is trained based on the updated second dataset label; determining a second difference between the updated first dataset label and the first dataset label, wherein if the second difference is greater than a second threshold, a first machine learning model is trained based on the updated first dataset label; and training the first machine learning model and the second machine learning model until the first difference is lower than the first threshold and the second difference is lower than the second threshold. In some embodiments, the first threshold and the second threshold are approximately equal to zero, wherein the first threshold is different from the second threshold.

[0013] In some embodiments, the updated first dataset is generated by classifying the first dataset using a trained second machine learning model, and the updated second dataset is generated by classifying the second dataset using the trained first machine learning model. In some embodiments, the instructions further cause the processor to continue training the first machine learning model and the second machine learning model until both the first machine learning model and the second machine learning model produce the same dataset labeling results when applied to the updated second dataset and the updated first dataset.

[0014] In some embodiments, the updated first dataset is generated by updating the labels of data samples from the first dataset that have been flipped to the new class after the trained second machine learning model is applied to the first dataset. The updated first dataset labels are different from the first dataset labels, and the first dataset is relabeled by a field human inspector based on the updated second dataset labels to generate the updated first dataset. In some embodiments, the first and second machine learning models are classification algorithms, wherein the first dataset is classified or labeled by a field human inspector, and wherein the updated second dataset is generated by updating the labels of data samples from the second dataset that have been flipped to the new class after the trained first machine learning model is applied to the second dataset.

[0015] In some embodiments, an adversarial training method includes: training a first machine learning model by a processor based on a first dataset label of a first dataset; providing the trained first machine learning model with a second dataset including a second dataset label to generate an updated second dataset including updated second dataset labels; training a second machine learning model by the processor based on the updated second dataset label; providing the trained second machine learning model with a first dataset to generate an updated first dataset including updated first dataset labels; and training the first machine learning model by the processor based on the updated first dataset label.

[0016] In some embodiments, the method further includes: determining a first difference between an updated second dataset label and a second dataset label by a processor, wherein if the first difference is greater than a first threshold, a second machine learning model is trained based on the updated second dataset label; determining a second difference between an updated first dataset label and a first dataset label by a processor, wherein if the second difference is greater than a second threshold, a first machine learning model is trained based on the updated first dataset label; and training the first machine learning model and the second machine learning model by a processor until the first difference is lower than the first threshold and the second difference is lower than the second threshold.

[0017] In some embodiments, the first threshold and the second threshold are approximately equal to zero, wherein the first threshold is different from the second threshold, and wherein the updated first dataset is generated by classifying the first dataset using a trained second machine learning model, and the updated second dataset is generated by classifying the second dataset using a trained first machine learning model. Attached Figure Description

[0018] These and other features of this disclosure will be appreciated and understood by referring to the specification, claims, and drawings, of some exemplary embodiments, wherein:

[0019] Figure 1A An overview of a system according to some embodiments of the present disclosure is shown;

[0020] Figure 1B An overview of a system for training a classifier according to some embodiments of the present disclosure is shown;

[0021] Figure 2 Example block diagrams of adversarial training methods according to some embodiments of the present disclosure are shown;

[0022] Figure 3 Illustrations of some embodiments according to this disclosure are shown. Figure 2 A flowchart of the adversarial training method;

[0023] Figure 4 An example block diagram of another adversarial training method according to some embodiments of the present disclosure is shown; and

[0024] Figure 5 An example block diagram of another adversarial training method according to some embodiments of the present disclosure is shown. Detailed Implementation

[0025] The following detailed description, set forth in conjunction with the accompanying drawings, is intended as a description of some exemplary embodiments of an adversarial training method for noise labeling provided in this disclosure, and is not intended to represent the only forms that can be constructed or utilized with respect to this disclosure. This description, in conjunction with the illustrated embodiments, illustrates the features of this disclosure. However, it should be understood that the same or equivalent functionality and structure can be implemented by different embodiments intended to also be included within the scope of this disclosure. As shown elsewhere herein, the same element numbers are intended to denote the same elements or features.

[0026] The mobile display industry has experienced rapid growth in recent years. With the deployment of new display panel modules and production methods, display resolution and pixel density have increased, making surface defects (e.g., defects in organic light-emitting diodes (OLEDs) within display panels) more difficult to inspect using traditional methods. Manual defect inspection is too time-consuming for modern manufacturing facilities, while automated inspection techniques are often ineffective. For example, in automated surface inspection, defects in uniform (e.g., non-textured) surfaces can be easily identified when a local anomaly has a clear contrast with its regular surrounding neighborhood. However, defects in low-contrast images are difficult to detect when there are no sharp edges between the defect and its surroundings and the background is unevenly illuminated.

[0027] One common type of display defect is "Mura." Mura is a large category of defects found in OLED manufacturing. Mura is a localized brightness non-uniformity that causes discomfort to human vision. Based on size and shape, Mura can be broadly categorized into line Mura, dot Mura, and area Mura. Each type of Mura may not have obvious edges and may not be very noticeable in an image. Therefore, a new method for identifying Mura defects is needed. Furthermore, the processing time of the new method or algorithm for identifying Mura should be considered, as it can directly impact OLED production efficiency. In addition, to achieve a fully automated surface inspection process, the classification of defect types (e.g., Mura) in display panels (e.g., OLED display panels) is necessary and will lead to increased production efficiency and a reduction in the need for human visual inspection.

[0028] Figure 1A An overview of a system according to some embodiments of the present disclosure is shown. Figure 1B An overview of a system for training a classifier according to some embodiments of the present disclosure is shown.

[0029] refer to Figure 1A and Figure 1B In some embodiments, the Mura detection system 101 receives an input image at a preprocessor 100. The input image may, for example, include an image of a display showing a test image. A camera can be used to generate the test image by capturing a picture of an OLED display showing the test image. In some embodiments, the test image may include an image of an instance of a Mura that might cause the display to exhibit a defect (e.g., white spot Mura). For example, the test image may be a uniform image exhibiting a low level of contrast. The input image may also have a sufficiently high resolution to show the individual pixels of the display being inspected for defects (e.g., white spot Mura). In some embodiments, the preprocessor 100 may be configured to receive the input image and perform smoothing to reduce noise in the image. After reducing noise in the input image, the preprocessor 100 may be configured to divide the image into multiple image blocks. Each of these image blocks can then be fed to a feature extractor 110.

[0030] In some embodiments, the feature extractor 110 is configured to compute various statistical features of the supplied image patch. For example, the statistical features may include one or more image moments (e.g., a weighted average of pixel intensities) and one or more texture measurements (e.g., texture analysis using a gray-level co-occurrence matrix (GLCM)). For example, in various embodiments, the feature extractor 110 extracts 37 statistical features comprising various image moments and GLCM texture features. In some embodiments, the feature extractor 110 may be configured to compute the mu30 moment (third centroid moment), contrast (GLCM), Hu5 moment (Hu moment), Hu1 moment (first Hu invariant moment), and correlation / dissimilarity (GLCM) for each image patch.

[0031] In some embodiments, the extracted statistical features of each image patch are supplied as input to classifier 120. In some embodiments, classifier 120 is a machine learning classifier that uses the extracted features (e.g., feature vectors) and label class information to identify instances of defects (e.g., muras). In some embodiments, the machine learning algorithm used in classifier 120 locates the position of the defect and classifies the defect type. In some embodiments, class information is supplied by training classifier 120.

[0032] In machine learning, classification can be defined as the process of predicting the class of a given data point in a dataset. The concept of classification in machine learning can include building models that classify data into different classes. In some embodiments, a class can be referred to as a target, label, or category. Classification prediction modeling can be an approximation of a mapping function from input variables to discrete output variables in a dataset. A classification model can be built by taking a set of training data pre-labeled with classes as input so that the algorithm can learn from it. The model can then be used with different datasets (e.g., a test dataset) that retain the classes as input, allowing the model to predict its class membership based on what it has learned from the training dataset. Because classification algorithms (e.g., decision trees, Naive Bayes, artificial neural networks (ANN), k-nearest neighbors (KNN)) require explicit class labels, classification is a form of supervised learning.

[0033] Many classification algorithms (e.g., decision trees, Naive Bayes, ANN, KNN) may be available, but it can be difficult to determine which algorithm is superior to the others. The choice of classification algorithm can be based on the application and nature of the available dataset. For example, if the classes or labels are linearly separable, then linear classifiers such as logistic regression or Fisher's linear discriminant might outperform complex models, and vice versa.

[0034] Since classifier 120 uses a supervised learning model, it is trained before it can function. In some embodiments, the supervised learning model used in classifier 120 is a support vector machine. The supervised learning model (e.g., a support vector machine) can be trained by providing classifier 120 with human inspection 130 during the training phase. For example, for each image patch, a field person can visually inspect the patch and label any instance of the white dot Mura. The image patches are also provided to feature extractor 110. The extracted feature vectors of the image patches and the corresponding human-inspected and labeled patches (e.g., the training dataset) are provided to classifier 120. Classifier 120 uses these provided patches (e.g., the training dataset) to generate class information (i.e., build a model) for later use in classification.

[0035] In some embodiments, the performance of a supervised learning model (e.g., a support vector machine) depends (e.g., heavily relies on) the amount of labeled examples in the training dataset and assumes that the labels are explicit and accurate. However, this assumption is generally not applicable to real-life datasets, such as training and / or test datasets with missing labels, subjective labeling, or images in the training dataset that are not exhaustively annotated. For example, in the training dataset used for a supervised learning model, class labels may be missing, objects in images may not be fully localized, or even on-site human experts (e.g., 130) may disagree on the class labels. As the training datasets for the classification algorithms used in supervised learning models become larger, the problem of missing labels and noisy labels in the test dataset becomes more acute. Noise caused by noisy labels, regardless of their source, can significantly degrade the learning performance of machine learning models.

[0036] In some embodiments, in order to use a machine learning model to classify a dataset with real-life data, it may be desirable to address the problem of noisy labels in the dataset. For example, test datasets with subjective labels and incomplete annotations make it difficult to evaluate the performance of machine learning models because such noisy labels may be detected as false detections or misclassifications.

[0037] In some example embodiments, datasets already labeled by multiple field human inspectors may suffer from subjective labeling issues because such labeling depends on the inspection results of individual inspectors. In some embodiments, test datasets may have subjective labeling issues for real-life problems; for example, making clear judgments in real-life problems (e.g., defect detection including Mura detection) can be challenging. In some embodiments, developing machine learning models (e.g., classification algorithms) can be challenging if there are conflicts between training and test datasets, which can frequently occur in the case of real-world datasets.

[0038] In some example implementations, it may be impossible to exhaustively label the entire dataset for all classified defects. Human inspectors can be used to label or classify only a few unresolved defects in the dataset. However, such an incompletely labeled dataset can lead to problems in the training dataset because such an incompletely labeled (e.g., underfitting) training dataset may have missing labels and inconsistent labeling. In some embodiments, an incompletely labeled dataset may make it difficult for machine learning models to be evaluated using a test dataset because noisy labels may be detected as false detections or misclassifications. For example, exhaustive labeling is required for all data to maintain the same detection criteria at all defect locations. This may not lead to subjective labeling problems. However, for a mixed dataset of exhaustively labeled and incompletely labeled datasets, it may be difficult to distinguish between subjective labeling problems and incomplete labeling problems.

[0039] As discussed above, classification is a form of supervised learning because classification algorithms require explicit class labels. The bias-variance tradeoff is a core problem in supervised learning. The bias-variance tradeoff can be a property of a set of predictive models, where a model with lower bias in parameter estimation may have higher variance in parameter estimation across samples, and vice versa. Bias can be an error arising from incorrect assumptions in the learning algorithm. In some embodiments, high bias can cause the algorithm to lose the correlation between features and the target output (e.g., underfitting). Variance can be an error arising from sensitivity to small fluctuations in the training set. In some embodiments, high variance can cause the algorithm to model random noise in the training data instead of the expected output (e.g., overfitting). In some embodiments, bias-variance decomposition is a way of decomposing the expected generalization error of a learning algorithm for a particular problem into a sum of three terms (bias, variance, and irreducible error) caused by noise in the problem itself. In some embodiments, the bias-variance problem can be a conflict in trying to minimize both sources of error (e.g., bias and variance) simultaneously, which may prevent a supervised learning algorithm from generalizing beyond its training set. The bias-variance tradeoff can be applied to all forms of supervised learning, such as classification, regression, structured output learning, or similar.

[0040] In some embodiments, it may be ideal to choose a model that both accurately captures regularities in its training data and generalizes well to unseen data. However, achieving both simultaneously can be challenging. High-variance learning methods may be able to represent their training sets well, but risk overfitting to noisy or unrepresentative training data. In contrast, algorithms with high bias may produce simpler models that may not tend to overfit but may underfit their training data and fail to capture important regularities. Although several classification algorithms exist, it may be difficult to reliably classify datasets representing defects in OLEDs due to the non-uniform nature of images displayed in OLED displays. Moreover, developing machine learning models (e.g., classification algorithms) can be challenging if there are conflicts between training and test datasets, which can often occur in the case of real-world datasets.

[0041] Most supervised learning algorithms, in their application to developing models and evaluating performance, are expected to identify basic factual labels on samples. This may not be the case for datasets suffering from subjective labeling issues. In defect detection, such as Mura detection, subjective and incomplete labeling is readily apparent on both training and test datasets. However, it may not be easy to correctly relabel the dataset (e.g., the test dataset) by simply re-examining it. Therefore, a method (or algorithm) may be desired to relabel the dataset with satisfactory performance metrics such as accuracy, precision, or recall to create effective machine learning models.

[0042] Various example embodiments of this disclosure can provide training methods for machine learning models to overcome problems associated with noisy labels in training and / or test datasets (e.g., overfitting or underfitting training data) and can create balanced models to achieve high accuracy for real-life datasets (e.g., defect detection in training or test datasets). Various example embodiments of this disclosure can also provide training methods to gradually reduce inconsistencies in classification between training and test datasets and create countable datasets for training and testing to meaningfully evaluate machine learning models. Some example embodiments of this disclosure can reduce the amount of review required to resolve subjective labeling problems from different field human inspectors, reduce the amount of review required to resolve pending defects from incompletely labeled datasets, and provide tools or architectures (e.g., machine learning models or classification algorithms) to help relabel data samples (e.g., training and / or test datasets) so that multiple inspectors can have a clear understanding of each data sample.

[0043] Some example embodiments of this disclosure can provide methods for training machine learning models on datasets with noisy labels (e.g., subjective labeling and incomplete labeling problems). Some example embodiments of this disclosure can include training two different machine learning models (e.g., classification algorithms) using two different datasets (e.g., a golden dataset or a training dataset and a new dataset or a test dataset), comparing results between other datasets (e.g., comparing dataset labels in the updated golden dataset for the current time period with dataset labels in the golden dataset for the previous time period, or comparing updated dataset labels in one dataset with previous dataset labels in the same dataset), and iteratively converging the results until a target performance metric such as accuracy, precision, or recall is reached. Some example embodiments of this disclosure can provide ways to show inconsistencies only from each side (training or testing) and reduce differences between two different datasets, and may also include ways to perform iterations by comparing the classification or detection results of the two machine learning models. In some embodiments, the method also provides solutions when one dataset (e.g., the training dataset) does not have sufficient data size to correctly classify the other dataset (e.g., the test dataset).

[0044] Figure 2 An example block diagram of the adversarial training method 200 is shown. Figure 3 The illustration is shown. Figure 2 The flowchart 300 of the adversarial training method 200 (i.e., adversarial training method 300). Figure 2 Adversarial training methods provide a way to update two datasets with different data distributions (e.g., dataset labels) and reach an agreement in the middle. Figure 2 The adversarial training method iterates between two machine learning models 210 and 218 (or two classification algorithms) until both models show the same results on the updated gold dataset 216 and the updated new dataset 206 (e.g., the same dataset labels the results). By iteratively reducing the inconsistency on the two datasets, the adversarial training method 200 will reach the point where the two machine learning models 210 and 218 generate the same results for both datasets.

[0045] The following paragraphs will be about Figure 2 Discussion of the block diagram Figure 3 300 adversarial training methods. Figure 3 The adversarial training method 300 can be executed by a processor in a system including memory, and the processor is configured to execute instructions stored in memory.

[0046] The adversarial training method 300 begins with 302.

[0047] At position 304, receive the new dataset 202 and the golden dataset 212.

[0048] In some embodiments, the Golden Dataset 212 refers to a dataset labeled by on-site human experts. Therefore, the Golden Dataset 212 is a trusted data source that will be used as a training dataset for machine learning algorithms used for OLED defect detection (e.g., Mura detection). The Golden Dataset 212 has different dataset labels than the New Dataset 202. For example, the Golden Dataset 212 may include a first dataset label, while the New Dataset 202 may include a second dataset label.

[0049] Optionally, at 306, a first machine learning model 218 can be trained based on the dataset labels (e.g., first dataset labels) of the gold dataset 212.

[0050] At 308, a new dataset 202 is provided to the trained first machine learning model 218 to relabel the new dataset 202. In some embodiments, the first distribution adjustment module 204 may apply the trained first machine learning model 218 to the new dataset 202.

[0051] At 310, an updated new dataset 206 is generated based on the trained first machine learning model 218. For example, the updated new dataset 206 is generated by classifying the new dataset 202 using the trained first machine learning model 218. The updated new dataset 206 may have a new criterion 208 or an updated dataset label (e.g., an updated second dataset label) that differs from the dataset label of the new dataset 202 (e.g., a second dataset label). For example, the updated new dataset 206 may include the updated second dataset label. In some embodiments, the new dataset 202 and the updated new dataset 206 may contain the same data but with different labels; for example, the new dataset 202 includes the second dataset label while the updated new dataset 206 includes the updated second dataset label. The updated new dataset 206 updates the data sample labels that have been flipped to the new class after the trained first machine learning model 218 has been applied to the new dataset 202.

[0052] At position 312, a first difference Δ1 is determined between the updated dataset label of the new dataset 206 (e.g., the updated second dataset label) and the dataset label of the new dataset 202 (e.g., the second dataset label), where,

[0053] Δ1 = (updated second dataset label - second dataset label).

[0054] At point 314, the first difference Δ1 is compared with the first threshold. At point 314, if it is determined that Δ1 is less than or equal to (approximately equal to zero) the first threshold, then at point 315, a second machine learning model 210 is trained based on the dataset labels of the new dataset 202 (e.g., the second dataset labels). Algorithm 300 proceeds from 315 to 318.

[0055] However, at 314, if it is determined that Δ1 is greater than the first threshold, then at 316, the second machine learning model 210 is trained based on the dataset label of the new standard 208 or the updated new dataset 206 (e.g., the updated second dataset label).

[0056] At 317, the dataset label of the new dataset 202 (e.g., the second dataset label) is updated to the dataset label of the new dataset 206 (e.g., the updated second dataset label).

[0057] At 318, the golden dataset 212 is provided to the trained second machine learning model 210. In some embodiments, the second distribution adjustment module 214 can apply the trained second machine learning model 210 to the golden dataset 212.

[0058] At 320, an updated gold dataset 216 is generated based on the trained second machine learning model 210. For example, the updated gold dataset 216 is generated by classifying the gold dataset 212 using the trained second machine learning model 210. The updated gold dataset 216 may have an updated dataset label or an updated first dataset label that differs from the dataset label (e.g., the first dataset label) of the gold dataset 212. In some embodiments, the gold dataset 212 and the updated gold dataset 216 may contain the same data but with different labels; for example, the gold dataset 212 includes the first dataset label while the updated gold dataset 216 includes the updated first dataset label. The updated gold dataset 216 updates the data sample labels that have been flipped to the new class after the trained second machine learning model 210 is applied to the gold dataset 212.

[0059] At position 322, a second difference Δ2 is determined between the dataset labels of the updated golden dataset 216 (e.g., the updated first dataset label) and the dataset labels of the golden dataset 212 (e.g., the first dataset label), where,

[0060] Δ2 = (updated first dataset label - first dataset label).

[0061] In some embodiments, at 322, the dataset label of the updated golden dataset 216 in the current time period is compared with the dataset label of the same dataset in the previous time period to determine Δ2. In some embodiments, the second distribution adjustment module 214 may compare the dataset label of the updated golden dataset 216 (e.g., the updated first dataset label) with the dataset label of the golden dataset 212 (e.g., the first dataset label) to determine the second difference Δ2.

[0062] At position 324, the second difference Δ2 is compared with the second threshold.

[0063] At 324, if it is determined that Δ2 is less than or equal to (approximately equal to zero) the second threshold, then at 325, the processor determines whether Δ1 is less than or equal to the first threshold.

[0064] If at 325, when it is determined at 324 that Δ2 is less than or equal to the second threshold, and Δ1 is determined to be less than or equal to the first threshold, then at 326, the adversarial training method 300 ends.

[0065] However, if at 325, when it is also determined at 324 that Δ2 is less than or equal to the second threshold, Δ1 is determined to be not less than or equal to the first threshold, then at 327, the first machine learning model 218 is trained based on the dataset labels of the new dataset 202 (e.g., the second dataset labels). Algorithm 300 returns from 327 to 308.

[0066] At 324, if it is determined that the second difference Δ2 is greater than the second threshold, then at 328, the first machine learning model 218 is trained (or updated) based on the dataset labels of the updated gold dataset 216 (e.g., the updated first dataset labels).

[0067] At position 329, the dataset label of the gold dataset 212 (e.g., the first dataset label) is updated to the dataset label of the updated gold dataset 216 (e.g., the updated first dataset label).

[0068] Algorithm 300 returns 308 from 329, and process 300 continues.

[0069] By iteratively running the training process for machine learning models 210 and 218, smaller differences or smaller Δ1 and Δ2 can be achieved for each iteration, which provides a clearer view of the boundaries because the inspector only reviews the data at the decision boundaries. The adversarial training method 300 can end when the two machine learning models 210 and 218 produce the same results for two datasets (e.g., the updated gold dataset 216 and the updated new dataset 206).

[0070] In some embodiments, the first distribution adjustment module 204 and the second distribution adjustment module 214 select data samples to flip decisions when generating an updated new dataset 206 or an updated golden dataset 216 (e.g., if unsure of making a decision, the sample is left for the next round).

[0071] In some embodiments, noisy or mislabeled labels in the Golden Dataset 212 or the new dataset 202 can be identified by a human inspector on-site, generating an updated Golden Dataset 216 and an updated new dataset 206. It is recommended that the same inspector be used to inspect both the Golden Dataset 212 and the new dataset 202 to reduce subjective labeling issues. By reducing discrepancies in the datasets, inspectors may develop an intuition about the actual decision boundaries. This could involve multiple iterations (of the adversarial training method 300) for updating the new criteria 208 and training machine learning models 210 and 218 until all inspectors agree on the results (e.g., dataset labeling) from the updated Golden Dataset 216 and the updated new dataset 206.

[0072] In some embodiments, a field human inspector may review the entire gold dataset 212 to relabel it according to a new standard 208 (e.g., an updated second dataset label) used for the updated new dataset 206, generating an updated gold dataset 216. However, the updated gold dataset 216 generated by a field human inspector relabeling the gold dataset 212 according to the new standard 208 (e.g., an updated second dataset label) may suffer from subjective labeling problems. Therefore, a machine or machine learning model (e.g., 210) is expected to be used to relabel the gold dataset 212 with the new standard 208 (e.g., an updated second dataset label).

[0073] The adversarial training method 300 can be extended to a new evaluation dataset by merging two test datasets (e.g., 202) into a new gold dataset. A new evaluation dataset can be received, which can be used as a new dataset, and the adversarial training method 300 can be applied.

[0074] In some example implementations, special samples that are not significantly different from other samples used for dataset labeling purposes may be included in two datasets (e.g., new dataset 202 and the golden dataset 212). This may occur for datasets that are too small to train on all the details. In such cases, a synthetic dataset can be generated to cover the sample, or a similar sample can be found in one dataset within a larger dataset and added to the other dataset.

[0075] In some example embodiments, training machine learning models 210 and 218 may imply training one or more of a locator, feature extractor (e.g., 110), and / or classifier (e.g., 120). In some embodiments, the classifier (e.g., 120) may apply a classification process to the dataset and find differences for use in a distribution adjustment process (e.g., 300). In some embodiments, the classifier (e.g., 120) and feature extractor (e.g., 110) may also update the feature set, which may be similar to finding differences. In some embodiments, the locator, feature extractor (e.g., 110), and classifier (e.g., 120) may generate new detection locations. Therefore, classification candidate locations may also be updated.

[0076] Figure 4 An example block diagram of another adversarial training method 400 is shown. Figure 4 The adversarial training method 400 can be executed by a processor in a system including memory, and the processor is configured to execute instructions stored in memory.

[0077] In the adversarial training method 400, a training dataset 401 and a test dataset 402 are received. The training dataset 401 may include a first dataset label, while the test dataset 402 may include a second dataset label. In some embodiments, the dataset label of the training dataset 401 (e.g., the first dataset label) can be used to train a machine learning model 410.

[0078] The test dataset 402 is fed to the trained machine learning model 410 to relabel the test dataset 402. An updated test dataset 415 is generated based on the trained machine learning model 410. For example, the updated test dataset 415 is generated by classifying the test dataset 402 using the trained machine learning model 410. The updated test dataset 415 may have an updated standard for dataset labeling or an updated second dataset label different from the dataset label used for the test dataset 402 (e.g., a second dataset label). The updated test dataset 415 is retained (e.g., stored in memory).

[0079] Next, the updated dataset labels of test dataset 415 (e.g., the updated second dataset labels) are compared with the dataset labels of test dataset 402 (e.g., the second dataset labels) to determine a first difference Δ1 between the updated second dataset labels and the second dataset labels, where,

[0080] Δ1 = (updated second dataset label - second dataset label).

[0081] If it is determined that Δ1 is less than or equal to (approximately equal to zero) a first threshold, then the machine learning model 420 is trained based on the dataset label (e.g., the second dataset label) of the test dataset 402.

[0082] However, if it is determined that Δ1 is greater than the first threshold, the machine learning model 420 is trained based on the updated dataset label of the test dataset 415 (e.g., the updated second dataset label). The dataset label of the test dataset 402 (e.g., the second dataset label) is updated to the updated dataset label of the test dataset 415 (e.g., the updated second dataset label).

[0083] Next, the training dataset 401 is fed to the trained machine learning model 420 to relabel the training dataset 401. An updated training dataset 425 is generated based on the trained machine learning model 420. For example, the updated training dataset 425 is generated by classifying the training dataset 401 using the trained machine learning model 420. The updated training dataset 425 may have an updated first dataset label that differs from the original dataset label or the first dataset label of the training dataset 401. The updated training dataset 425 is retained.

[0084] Next, the updated dataset labels for training dataset 425 (e.g., the updated first dataset labels) are compared with the dataset labels for training dataset 401 (e.g., the first dataset labels) to determine the second difference Δ2 between the updated first dataset labels and the first dataset labels, where,

[0085] Δ2 = (updated first dataset label - first dataset label).

[0086] In some embodiments, the dataset label of the updated training dataset 425 in the current time period (e.g., the updated first dataset label) is compared with the dataset label of the same dataset in the previous time period (e.g., the first dataset label) to determine Δ2.

[0087] If the second difference Δ2 is less than or equal to (approximately equal to zero) the second threshold, the processor determines whether Δ1 is less than or equal to the first threshold.

[0088] If it is determined that Δ1 is less than or equal to the first threshold and Δ2 is less than or equal to the second threshold, then the adversarial training method 400 ends.

[0089] However, if it is determined that Δ1 is not less than or equal to the first threshold and Δ2 is less than or equal to the second threshold, then the machine learning model 410 is trained based on the dataset label of the test dataset 402 (e.g., the second dataset label).

[0090] If it is determined that the second difference Δ2 is greater than the second threshold, the machine learning model 410 is updated or trained (or adjusted) based on the dataset labels of the updated training dataset 425 (e.g., the updated first dataset labels).

[0091] The dataset labels of training dataset 401 (e.g., the first dataset label) are updated to the dataset labels of the updated training dataset 425 (e.g., the updated first dataset label), and process 400 continues. The adversarial training method 400 can end when the two machine learning models 410 and 420 generate the same results for the two datasets (e.g., the updated training dataset 425 and the updated test dataset 415).

[0092] By iteratively running the training process for machine learning models 410 and 420, smaller differences or smaller Δ1 and Δ2 can be achieved for each iteration, which can provide a clearer view of the boundaries, since the inspector only reviews the data at the decision boundaries.

[0093] Figure 5 An exemplary block diagram of yet another adversarial training method 500 is shown. Figure 5 The adversarial training method 500 can be executed by a processor in a system including memory, and the processor is configured to execute instructions stored in memory.

[0094] In the adversarial training method 500, a golden dataset 502 is received. The golden dataset 502 may include the first dataset label. A result dataset 514 may also be received. The result dataset 514 may include the second dataset label.

[0095] Alternatively, the machine learning model 512 can be trained based on the dataset labels of the gold dataset 502 (e.g., the first dataset label).

[0096] The resulting dataset 514 is fed to the trained machine learning model 512 to relabel the resulting dataset 514. A new dataset 504 is generated based on the trained machine learning model 512. For example, the new dataset 504 is generated by classifying the resulting dataset 514 using the trained machine learning model 512. The new dataset 504 may have updated dataset labels (e.g., updated second dataset labels) that are different from the dataset labels (e.g., second dataset labels) of the resulting dataset 514. For example, the new dataset 504 may include updated second dataset labels. In some embodiments, the resulting dataset 514 and the new dataset 504 may contain the same data but with different labels; for example, the resulting dataset 514 includes the second dataset labels while the new dataset 504 includes updated second dataset labels. The new dataset 504 updates the labels of data samples that have been flipped to the new class after the machine learning model 512 is applied to the resulting dataset 514.

[0097] Determine the first difference Δ1 between the updated second dataset label and the second dataset label, where,

[0098] Δ1 = (updated second dataset label - second dataset label).

[0099] The first difference Δ1 is compared with a first threshold. If it is determined that Δ1 is less than or equal to (approximately equal to zero) the first threshold, the golden dataset 502 is provided to the distribution adjustment module 506 and an updated golden dataset 510 is generated based on the dataset label (e.g., the second dataset label) of the new standard 508 or the new dataset 504. The updated golden dataset 510 may have an updated dataset label that is different from the original dataset label or the first dataset label of the golden dataset 502.

[0100] Next, the updated dataset labels for the gold dataset 510 (e.g., the updated first dataset labels) are compared with the dataset labels for the gold dataset 502 (e.g., the first dataset labels) to determine the second difference Δ2 between the updated first dataset labels and the first dataset labels, where,

[0101] Δ2 = (updated first dataset label - first dataset label).

[0102] In some embodiments, the dataset label of the updated gold dataset 510 in the current time period (e.g., the updated first dataset label) is compared with the dataset label of the same dataset in the previous time period (e.g., the first dataset label) to determine Δ2.

[0103] If Δ2 is less than or equal to (approximately equal to zero) the second threshold, then adversarial training method 500 terminates. Otherwise, machine learning model 512 is trained based on the updated dataset labels of the golden dataset 510 (e.g., the updated first dataset labels), and process 500 continues. Adversarial training method 500 can terminate when machine learning model 512 produces the same results for both datasets (e.g., the updated golden dataset 510 and the new dataset 504).

[0104] By iteratively running the training process for the machine learning model 512, a smaller difference or a smaller Δ2 can be achieved for each iteration, which provides a clearer view of the boundaries, since the inspector only reviews the data at the decision boundaries.

[0105] It should be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Therefore, without departing from the spirit and scope of the inventive concept, the first element, component, region, layer, or portion discussed herein may be referred to as the second element, component, region, layer, or portion.

[0106] For ease of description, this document may use spatial relative terms such as “below,” “under,” “lower,” “bottom,” “above,” “higher,” and the like to describe the relationship of an element or feature as shown in the figure to one or more other elements or features. It should be understood that such spatial relative terms are intended to include different orientations of the device in use or operation, in addition to those shown in the figure. For example, if the device in the figure is flipped, an element described as “below,” “under,” or “bottom” of other elements or features would be oriented “above” other elements or features. Therefore, the example terms “below” and “bottom” can include both above and below orientations. The device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein should be interpreted accordingly. Additionally, it should be understood that when a layer is referred to as “between” two layers, it can be the only layer between those two layers, or there may be one or more intermediate layers.

[0107] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the inventive concept. As used herein, the terms “substantially,” “about,” and similar terms are used as approximate terms rather than terms of degree and are intended to take into account the inherent biases of measurements or calculations that will be recognized by one of ordinary skill in the art.

[0108] As used herein, unless the context clearly indicates otherwise, the singular form “a” is intended to include the plural form as well. It should be further understood that, when used in this specification, the term “comprising” and its variations specify the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of…” following a list of elements modify the entire list of elements without modifying any individual element in the list. Furthermore, when describing embodiments of the inventive concept, the use of “may” means “one or more embodiments of this disclosure.” Moreover, the term “exemplary” is intended to indicate an example or illustration. As used herein, the terms “use” and “being used” may be considered synonymous with the terms “utilize” and “be exploited,” respectively.

[0109] It should be understood that when a component or layer is referred to as being "on," "connected to," "coupled to," or "adjacent to" another component or layer, it may be directly on, connected to, coupled to, or adjacent to the other component or layer, or one or more intermediate components or layers may be present. Conversely, when a component is referred to as being "directly on," "directly connected to," "directly coupled to," or "immediately adjacent to" another component or layer, no intermediate components or layers are present.

[0110] Any numerical range described herein is intended to include all subranges containing the same numerical precision within the range described. For example, the range “1.0 to 10.0” is intended to include all subranges between (and including) the described minimum value of 1.0 and the described maximum value of 10.0, i.e., all subranges having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as, for example, 2.4 to 7.6. Any maximum numerical limit described herein is intended to include all lower numerical limits contained therein, and any minimum numerical limit described in this specification is intended to include all higher numerical limits contained therein.

[0111] In some embodiments, one or more outputs of different embodiments of the methods and systems of this disclosure may be transmitted to an electronic device coupled to or having a display device for displaying one or more outputs or information relating to one or more outputs of different embodiments of the methods and systems of this disclosure.

[0112] Electronic or electrical devices and / or any other related devices or components according to embodiments of the present disclosure described herein can be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuits), software, or a combination of software, firmware, and hardware. For example, various components of such devices may be formed on an integrated circuit (IC) chip or on a discrete IC chip. Furthermore, various components of such devices may be implemented on a flexible printed circuit film, a tape-on-a-package (TCP), a printed circuit board (PCB), or formed on a substrate. Additionally, various components of such devices may be processes or threads running on one or more processors in one or more computing devices, executing computer program instructions and interacting with other system components to perform the various functions described herein. The computer program instructions are stored in memory that can be implemented in the computing device using standard memory devices such as, for example, random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer-readable media such as, for example, CD-ROMs, flash drives, or the like. Furthermore, those skilled in the art will recognize that, without departing from the spirit and scope of exemplary embodiments of the present disclosure, the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices.

[0113] Although exemplary embodiments of the adversarial training method for noise labels have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that the adversarial training method for noise labels constructed according to the principles of this disclosure may be embodied in ways other than those specifically described herein. The inventive concept is further defined in the appended claims and their equivalents.

Claims

1. A Mura detection system, comprising: A preprocessor is configured to receive an input image; A feature extractor is configured to extract features from the received input image; A classifier uses extracted features and label class information to identify instances of defects; as well as An adversarial training system is used to train the classifier. The adversarial training system includes: Memory; and A processor configured to execute instructions stored in the memory, wherein, when executed by the processor, the processor: Receive the first dataset, which includes the first dataset label; Receive a second dataset including the tags of the second dataset; The first machine learning model is trained based on the labels of the first dataset. The second dataset is provided to the trained first machine learning model to generate an updated second dataset including updated second dataset labels, the updated second dataset being generated by classifying the second dataset using the trained first machine learning model; Determine the first difference between the updated second dataset label and the second dataset label; If the first difference is greater than the first threshold, then a second machine learning model is trained based on the updated second dataset labels; The first dataset is provided to the trained second machine learning model to generate an updated first dataset including updated first dataset labels, the updated first dataset being generated by classifying the first dataset using the trained second machine learning model; Determine the second difference between the updated first dataset label and the first dataset label; and If the second difference is greater than the second threshold, then the first machine learning model is trained based on the updated first dataset labels.

2. The Mura detection system according to claim 1, wherein the instructions further cause the processor to continue: Train the first machine learning model and the second machine learning model until the first difference is lower than the first threshold and the second difference is lower than the second threshold.

3. The Mura detection system according to claim 2, wherein one of the first threshold and the second threshold is equal to zero, wherein the first threshold is different from the second threshold.

4. The Mura detection system according to claim 1, wherein the instructions further cause the processor to continue training the first machine learning model and the second machine learning model until both the first machine learning model and the second machine learning model produce the same dataset labeling results when applied to the updated second dataset and the updated first dataset.

5. The Mura detection system of claim 1, wherein the updated first dataset is generated by updating the labels of data samples from the first dataset that have been flipped to the new class after the trained second machine learning model is applied to the first dataset.

6. The Mura detection system of claim 1, wherein the updated first dataset label is different from the first dataset label.

7. The Mura detection system of claim 6, wherein the first dataset is relabeled by a field human inspector based on the updated second dataset label to generate the updated first dataset.

8. The Mura detection system according to claim 1, wherein the first machine learning model and the second machine learning model are classification algorithms.

9. The Mura detection system of claim 1, wherein the updated second dataset is generated by updating the labels of data samples from the second dataset that have been flipped to the new class after the trained first machine learning model has been applied to the second dataset.

10. The Mura detection system of claim 1, wherein the first dataset is classified or labeled by a field human inspector.

11. A Mura detection system, comprising: A preprocessor is configured to receive an input image; A feature extractor is configured to extract features from the received input image; A classifier uses extracted features and label class information to identify instances of defects; as well as An adversarial training system is used to train the classifier. The adversarial training system includes: Memory; and A processor configured to execute instructions stored in the memory, wherein, when executed by the processor, the processor: The first machine learning model is trained based on the first dataset's first dataset labels; The trained first machine learning model is provided with a second dataset including the second dataset labels to generate an updated second dataset including the updated second dataset labels; A second machine learning model is trained based on the updated second dataset labels; The first dataset is fed into the trained second machine learning model to generate an updated first dataset that includes updated labels for the first dataset; and The first machine learning model is trained based on the updated first dataset labels.

12. The Mura detection system of claim 11, wherein the instructions further cause the processor to continue: Determine a first difference between the updated second dataset label and the second dataset label, wherein if the first difference is greater than a first threshold, train the second machine learning model based on the updated second dataset label; Determine a second difference between the updated first dataset label and the first dataset label, wherein if the second difference is greater than a second threshold, train the first machine learning model based on the updated first dataset label; and Train the first machine learning model and the second machine learning model until the first difference is lower than the first threshold and the second difference is lower than the second threshold.

13. The Mura detection system of claim 12, wherein one of the first threshold and the second threshold is equal to zero, wherein the first threshold is different from the second threshold.

14. The Mura detection system of claim 11, wherein the updated first dataset is generated by classifying the first dataset using a trained second machine learning model, and the updated second dataset is generated by classifying the second dataset using a trained first machine learning model.

15. The Mura detection system of claim 11, wherein the instructions further cause the processor to continue training the first machine learning model and the second machine learning model until both the first machine learning model and the second machine learning model produce the same dataset labeling results when applied to the updated second dataset and the updated first dataset.

16. The Mura detection system according to claim 11, wherein: The updated first dataset is generated by updating the labels of data samples from the first dataset that have been flipped to the new class after the trained second machine learning model is applied to the first dataset. The updated first dataset label is different from the first dataset label, and The first dataset was relabeled by on-site human inspectors based on the updated second dataset labels to generate the updated first dataset.

17. The Mura detection system of claim 11, wherein the first machine learning model and the second machine learning model are classification algorithms, wherein the first dataset is classified or labeled by a field human inspector, and wherein the updated second dataset is generated by updating the labels of data samples from the second dataset that have been flipped to the new class after the trained first machine learning model is applied to the second dataset.

18. A Mura detection method, comprising: The input image is received via a preprocessor. The features of the received input image are extracted using a feature extractor; Instances of defects are identified by using the extracted features and label class information through a classifier; as well as The classifier is trained using the following adversarial training method, which includes: The processor trains the first machine learning model based on the first dataset's first dataset labels; The processor provides the trained first machine learning model with a second dataset including the second dataset label to generate an updated second dataset including the updated second dataset label; The processor trains a second machine learning model based on the updated second dataset labels; The processor provides the first dataset to the trained second machine learning model to generate an updated first dataset including updated first dataset labels; and The processor trains the first machine learning model based on the updated first dataset labels.

19. The Mura detection method according to claim 18, further comprising: The processor determines a first difference between the updated second dataset label and the second dataset label, wherein if the first difference is greater than a first threshold, the second machine learning model is trained based on the updated second dataset label; The processor determines a second difference between the updated first dataset label and the first dataset label, wherein if the second difference is greater than a second threshold, the first machine learning model is trained based on the updated first dataset label; and The processor trains the first machine learning model and the second machine learning model until the first difference is lower than the first threshold and the second difference is lower than the second threshold.

20. The Mura detection method of claim 19, wherein one of the first threshold and the second threshold is equal to zero, wherein the first threshold is different from the second threshold, and wherein the updated first dataset is generated by classifying the first dataset using a trained second machine learning model, and the updated second dataset is generated by classifying the second dataset using a trained first machine learning model.

Images