Manufacturing defect classification system and method

Abstract

To accurately predict a manufacturing defect.SOLUTION: A manufacturing defect classification method according to one embodiment of the present invention includes the steps of: identifying a first data sample satisfying a first criterion from a training data set; removing the first data sample from the training data set to output the filtered training data set including a second data sample; training a first machine learning model with the filtered training data set; training a second machine learning model on the basis of at least one of the first data sample and the second data sample; receiving first product data associated with a first manufactured product; applying the second machine learning model to confidence prediction of the first product data; and in response to the confidence prediction of the first product data, applying the first machine learning model to generation of classification based on the first product data.SELECTED DRAWING: Figure 2

Description

【Technical Field】 【0001】 The present invention relates to a manufacturing defect classification system and method. 【0002】 This application claims the priority of U.S. Patent Application No. 63 / 169,621, filed with the U.S. Patent and Trademark Office on April 1, 2021 (Title of Invention: IDENTIFY MANUFACTURING DISPLAY IMAGE DEFECT TYPES WITH TWO-STAGE REJECTION-BASED METHOD), and the entire content of U.S. Patent Application No. 63 / 169,621 is incorporated herein by reference. 【Background Art】 【0003】 In recent years, the mobile display device industry has grown rapidly, and by using new types of display panel modules and production methods, it has become difficult to find surface defects only with existing methods. Therefore, for example, it is preferable to automatically predict whether a manufactured display panel module is defective using artificial intelligence (AI). 【0004】 In fact, it is preferable to predict defects using artificial intelligence not only for display panel modules but also for other hardware products. 【0005】 The information described in the background art is for enhancing the understanding of the present invention and includes information that does not pertain to the prior art already known to those skilled in the art. 【Prior Art Documents】 【Patent Documents】 【0006】 【Patent Document 1】 U.S. Patent No. 10436702 【Patent Document 2】 U.S. Patent Application Publication No. 2020 / 0311615 【Summary of the Invention】 [Problems that the invention aims to solve] 【0007】 The problem that this invention aims to solve is to accurately predict manufacturing defects. [Means for solving the problem] 【0008】 A manufacturing defect classification method according to one embodiment of the present invention includes the steps of: identifying a first data sample that satisfies a first criterion from a training dataset; removing the first data sample from the training dataset to output a filtered training dataset that includes a second data sample; training a first machine learning model with the filtered training dataset; training a second machine learning model based on at least one of the first and second data samples; receiving first product data associated with a first manufactured product; applying the second machine learning model to predict the confidence of the first product data; and applying the first machine learning model to generate a classification based on the first product data in response to the confidence prediction of the first product data. 【0009】 The first criterion may be a confidence level below a set threshold. 【0010】 The second data sample may be associated with a confidence level above the set threshold. 【0011】 The step of training the second machine learning model includes a step of applying unsupervised learning based on the second data sample, and the second data sample may be associated with a specific class. 【0012】 The step of training the second machine learning model further includes the step of identifying a cluster associated with the particular class, and the step of adjusting the boundary of the cluster based on an adjustment threshold, and in response to the determination that the first product data falls within the boundary of the cluster, the first machine learning model may be applied to generate the classification. 【0013】 The step of training the second machine learning model includes a step of applying supervised learning based on the first and second data samples, wherein the first data sample may be identified as a first data type and the second data sample may be identified as a second data type. 【0014】 The step of training the second machine learning model further includes identifying a decision boundary that separates the first data type from the second data type, and adjusting the decision boundary based on the adjustment threshold, wherein the first machine learning model may be applied to generate the classification in response to a determination that the first product data belongs to the second data type. 【0015】 The process may further include the steps of identifying second product data associated with a second production product, applying the second machine learning model to predict the confidence level of the second product data, and processing the second product data as defective based on the confidence level of the second product data. 【0016】 The process further includes a step of generating a signal based on the aforementioned classification, the signal being used to trigger an action. 【0017】 A manufacturing defect classification system according to one embodiment of the present invention includes a processor and memory. The memory stores instructions, and when the processor executes the instructions, the processor identifies a first data sample that satisfies a first criterion from a training dataset, removes the first data sample from the training dataset, outputs a filtered training dataset including a second data sample, trains a first machine learning model on the filtered training dataset, trains a second machine learning model based on at least one of the first and second data samples, receives first product data associated with a first manufactured product, applies the second machine learning model to predict the confidence of the first product data, and in response to the confidence prediction of the first product data, applies the first machine learning model to generate a classification based on the first product data. 【0018】 The first criterion may be a confidence level below a set threshold. 【0019】 The second data sample may be associated with a confidence level above the set threshold. 【0020】 The training of the second machine learning model involves applying unsupervised learning based on the second data sample, and the second data sample may be associated with a specific class. 【0021】 The training of the second machine learning model further includes identifying clusters associated with the particular class and adjusting the boundaries of the clusters based on adjustment thresholds, and in response to the determination that the first product data falls within the boundaries of the clusters, the first machine learning model may be applied to generate the classification. 【0022】 Training of the second machine learning model includes applying supervised learning based on the first and second data specimens, where the first data specimen is identified as a first data type and the second data specimen may be identified as a second data type. 【0023】 Training of the second machine learning model further includes identifying a decision boundary that separates the first data type from the second data type and adjusting the decision boundary based on the adjustment threshold, and the first machine learning model may be applied to generate the classification in response to a determination that the first product data belongs to the second data type. 【0024】 When the processor executes the instructions, the processor additionally identifies second product data associated with a second production product, applies the second machine learning model to predict the confidence of the second product data, and rejects the second product data based on the confidence of the second product data. 【0025】 When the processor executes the instructions, the processor additionally generates a signal based on the classification, and the signal may be for triggering an operation. 【0026】 A manufacturing defect classification system according to another embodiment of the present invention includes a data collection circuit that collects an input data set, and a processing circuit that is connected to the data collection circuit and has logic. The logic identifies a first data sample that meets a first criterion from a training data set, removes the first data sample from the training data set, outputs a filtered training data set that includes a second data sample, trains a first machine learning model with the filtered training data set, trains a second machine learning model based on at least one of the first data sample and the second data sample, receives product data related to a manufactured product, applies the second machine learning model to predict the confidence of the product data, and in response to the prediction of the confidence of the product data, applies the first machine learning model to generate a classification based on the product data. 【0027】 The first criterion may be a confidence level below a set threshold. 【Advantages of the Invention】 【0028】 By filtering data samples with low confidence in the training and inference processes in this way, the influence of out-of-distribution samples can be minimized, thereby improving the prediction accuracy of the target data samples (covered data samples). 【Brief Description of the Drawings】 【0029】 [Figure 1] It is a block diagram of a system for making predictions regarding products produced through a manufacturing process according to an embodiment of the present invention. [Figure 2] It is a flowchart of a method for making predictions regarding products produced through a manufacturing process according to an embodiment of the present invention. [Figure 3]This is a more detailed flowchart of the confidence learning process according to one embodiment of the present invention. [Figure 4] This is an example of a confusion matrix according to one embodiment of the present invention. [Figure 5] This is a block diagram of a defect detection model implemented as a simultaneous fusion model according to one embodiment of the present invention. [Figure 6] This is a conceptual diagram of the data filtered by the anomaly filter during the prediction phase according to one embodiment of the present invention. [Figure 7] This graph illustrates the trade-off between accuracy and range when selecting an adjustment threshold using one embodiment of the present invention. [Figure 8] This graph shows an example of an adjustment threshold calculated according to a function of range and accuracy by one embodiment of the present invention. [Modes for carrying out the invention] 【0030】 Embodiments of the present invention will be described in detail below with reference to the attached drawings. However, the present invention can be realized in various different forms and is not limited to the embodiments described herein. By illustrating such embodiments, the detailed description of the invention will become clearer and it will be possible to fully convey the diverse aspects and features of the invention to those skilled in the art. Therefore, processes, apparatus, techniques, etc. that can be omitted in order for those skilled in the art to fully understand the diverse aspects and features of the present invention will be omitted from the description. Unless otherwise specified, the same reference numerals throughout the drawings and specification indicate the same components, and redundant descriptions may be omitted. Also, in the drawings, the size of parts, layers, and areas may be exaggerated or simplified for clearer understanding. 【0031】 As new types of display modules and manufacturing methods are used, and product specifications become more stringent, improvements to equipment and quality control methods are needed to maintain quality. For example, it is necessary to monitor manufacturing defects during the production process. 【0032】 One method of monitoring manufacturing defects is to employ human inspectors who possess the expertise to identify defects. In connection with this, high-resolution (sub-micron level) images can be obtained around the defective area. Human inspectors examine the obtained images and categorize defects based on their type and their impact on production yield. More specifically, human inspectors can sample numerous defect images, spend considerable time searching for features, and categorize unclassified defect images. However, training human inspectors is time-consuming. Furthermore, even after training, it can take several weeks for human inspectors to identify manufacturing defects in the current batch of images, making it difficult to scale the human inspector's work to various instances simultaneously. 【0033】 Machine learning (ML) models can be used for rapid manufacturing defect detection that can be extended to various cases simultaneously. However, for such machine learning models to be useful, the predictions they make must be accurate. Furthermore, the models must be generalizable so that they can make accurate predictions even for new data sets that have not been encountered before. 【0034】 However, various factors can degrade the performance of machine learning models. One such factor is mislabeling of training data, known as label noise. Mislabeling can start, for example, with human error. For instance, images used for training might be labeled to indicate a certain type of manufacturing defect, when in reality the defect is virtually nonexistent, or the type of defect identified by the human is incorrect. Training a machine learning model with mislabeled data will reduce the predictive accuracy of the machine learning model. 【0035】 Another problem that arises when using machine learning models is the use of small data sets when training the model. A sparse data set compared to the high dimensionality of the data can lead to model overfitting. If the model is overfitted, mislabeled data may not be properly processed and may be learned by the model. This can lead to predictions being made during deployment based on mislabeled data, and as a result, the model may not function correctly on new, unfamiliar data. 【0036】 Generally, embodiments of the present invention relate to manufacturing defect identification using deep learning machine learning models. According to one embodiment of the present invention, a two-stage approach is used to filter out unconfident (also known as noisy) data. In this view, data samples with noise labels in the training data set are removed to train a first machine learning model so that the defect detection model is trained using a training data set that is confident (also known as clean). The unconfident (also known as noisy) data may then be used to train a second deep learning machine learning model (also known as an outlier detection model or outlier filter) which is used while filtering out the noisy / unconfident data samples. In this way, the data predicted by the defect detection model can be moved into the high-confident prediction region, thereby improving the prediction accuracy of the defect detection model. 【0037】 According to one embodiment of the present invention, a tuning threshold hyperparameter is used to adjust the boundary that an anomaly filter uses to filter out uncertain data. The threshold hyperparameter can be determined by considering the tradeoff between the rejection rate [or amount of coverage of the data] and the predictive accuracy of the defect detection model. According to one embodiment of the present invention, a smaller coverage leads to higher predictive accuracy. In this example, the threshold hyperparameter can be selected based on the identification of current requirements in terms of accuracy and / or coverage. 【0038】 Figure 1 is a block diagram of a system for making predictions about products produced through a manufacturing process according to one embodiment of the present invention. The system includes, but is not limited to, one or more data collection circuits 100 and an analysis system 102. The data collection circuit 100 may include, for example, one or more imaging systems that obtain image data of products during the manufacturing process, such as an X-ray machine, a magnetic resonance imaging (MRI) device, a transmission electron microscope (TEM) device, a scanning electron microscope (SEM) device, and / or similar devices. The image data collected by the data collection circuit 100 may be, for example, spectroscopy images such as energy-dispersive X-ray spectroscopy (EDS) images and / or high-angle annular dark-field (HAADF) images, microscopy images such as transmission electron microscope (TEM) images, thermal images, and / or similar images. The collected data samples are not limited to still images, but may also include video, text, lidar data, radar data, image fusion data, temperature data, pressure data, and / or similar data. 【0039】 The data acquisition circuit 100 can be placed, for example, on a conveyor belt that transports products during the production process. The data acquisition circuit 100 can acquire data samples (e.g., image data) of the product multiple times during one manufacturing cycle (e.g., every second or at intervals of a few seconds). 【0040】 The analysis system 102 may include a training module 106 and an inference module 108. Although the training module 106 and the inference module 108 are described as separate functional units, those skilled in the art will understand that the functions of the two modules 106 and 108 can be combined or integrated into a single module, or further subdivided into smaller modules, without departing from the spirit and scope of the present invention. The components of the analysis system 102 can be realized by one or more processors [e.g., processor 101 in Figure 1] having associated memory [e.g., memory 103 in Figure 1], and the processor 101 may include, for example, application-specific integrated circuits (ASICs), general-purpose or special-purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and field-programmable gate arrays (FPGAs). 【0041】 The training module 106 can generate and train multiple machine learning models used to classify product manufacturing defects. The generation and training of multiple machine learning models may be based on training data provided by the data acquisition circuit 100. According to one embodiment of the present invention, two machine learning models are trained in two separate stages. In the first stage, a defect detection model can be trained using only the flawless training data set. In this view, in order to generate the flawless training data set, labels with noisy / uncertain data that are identified as false are removed from the test data set. 【0042】 According to one embodiment of the present invention, the defect detection model is a joint fusion model trained using a set of flawless test data from different types of data acquisition circuits 100 that are integrated and trained together. The defect detection model does not have to be a joint fusion model, but could be any deep neural network known to those skilled in the art that is trained using information from a single source. 【0043】 In the second stage, the outlier filter may be trained to filter out noisy / uncertain data samples while it is being deployed. According to one embodiment of the present invention, the outlier filter is trained using unsupervised learning based on the flawless training data samples identified in the first stage. According to one embodiment of the present invention, the outlier filter is trained using supervised learning based on data samples labeled as noisy / uncertain in the first stage. The size of the classification cluster or decision boundary may vary depending on the identified adjustment threshold hyperparameter. 【0044】 The inference module 108 can classify product manufacturing defects during the placement process at the inference stage. In this example, the data acquisition circuit 100 can provide the obtained data samples to an outlier filter to identify whether the data samples are confident or not. According to one embodiment of the present invention, the outlier filter determines whether a data sample is an outlier. For example, if a data sample does not fall into one of the classification groups generated based on flawless training data, that data sample can be identified as an outlier. According to another embodiment of the present invention, if a data sample matches the characteristics of data labeled as noisy / uncertain, that data sample can be recognized as an outlier. 【0045】 According to one embodiment of the present invention, data samples identified as outliers are removed. In this example, the removed data samples are not provided to the defect detection model for prediction. Therefore, the data provided to the defect detection model can be considered confidence data, thereby improving the classification accuracy of the defect detection model. The classification performed by the defect detection model may include the presence or absence of defects in a product, the classification of defects in defective products, and / or similar classifications. According to one embodiment of the present invention, the analysis system 102 can generate a signal based on the classification result. For example, the signal may prompt a human inspector to take some action in response to classifying a product as defective. That action may be to remove the product from the production line for reinspection. 【0046】 Figure 2 is a flowchart of a method for making predictions about products produced through a manufacturing process according to one embodiment of the present invention. The order of the steps in this process is not limited and may be changed to other orders recognized by those skilled in the art. 【0047】 In block 200, one or more data acquisition circuits 100 capture data on products produced during the manufacturing process. The captured data may be, for example, image data. According to one embodiment of the present invention, two or more completely different data acquisition circuits 100 can simultaneously capture image data of a specific product. For example, a first data acquisition circuit 100 can capture a TEM image of the product, and a second data acquisition circuit 100 can capture a HAADF image of the same product. 【0048】 The data acquired by the data acquisition circuit 100 can be used to train a machine learning model. In this example, a human inspector can examine and label images of the area around a defect in the product obtained by the data acquisition circuit 100 for defect identification. However, humans are prone to errors, and sometimes the labels attached to the images may be incorrect. Labeling errors can be a problem because the accuracy of the model depends on the accuracy of the training data. 【0049】 According to one embodiment of the present invention, training module 106 performs confidence learning in block 202 to identify and remove noisy data samples from the training data set. Noisy data samples may include image data that a human inspector would expect to have mislabeled. According to one embodiment of the present invention, confidence learning is based on an estimation of a joint distribution between noisy [given] labels and uncorrupted [true] labels, which is specifically described in Northcutt et al., “Confident Learning: Estimating Uncertainty in Dataset Labels” (2021) (available at https: / / arxiv.org / abs / 1911.00068v4), which is herein by reference. 【0050】 According to one embodiment of the present invention, in response to confidence learning in block 202, the training module 106 identifies data samples in the training data set that it expects to be noisy, labels the identified data samples as noisy, and removes such data samples from the training data set. 【0051】 In block 204, the training module 106 trains a defect detection model based on flawless data samples from the filtered training data set. The trained defect detection model may be, for example, a concurrent fusion model, as described in U.S. Patent Application No. 16 / 938,812 (title: “Image-Based Defects Identification and Semi-Supervised Localization”) filed on 24 July 2020, or U.S. Patent Application No. 16 / 938,857 (title: “Fusion Model Training Using Distance Metrics”) filed on 24 July 2020, which are herein by reference. According to one embodiment of the present invention, the defect detection model is a single machine learning model (rather than a concurrent fusion model), consisting of, for example, a random forest, XGBoost (extreme gradient boosting), SVM (support-vector machine), DNN (deep neural network), and / or similar machine learning algorithms. 【0052】 In block 206, the training module 106 trains an outlier filter using noisy data samples and / or flawless data samples from confidence learning block 202. Training the outlier filter can employ either supervised learning or unsupervised learning. In embodiments employing supervised learning, flawless data samples, noisy data samples, and labeled samples can be used to teach the outlier filter to classify data into noisy / uncertain or flawless / confident. A decision boundary can be identified during the training process to determine the boundary separating flawless data from noisy data. For example, a machine learning algorithm such as logistic regression can be used to identify the decision boundary. 【0053】 In embodiments employing unsupervised learning, the training module 106 uses a clustering algorithm to find similarities among the training data samples and groups similar data samples into a single cluster. Clusters can be generated using a clustering algorithm such as the K-means clustering algorithm. 【0054】 According to one embodiment of the present invention, the training module 106 adjusts the position of the crowd boundary or decision boundary based on the adjustment threshold hyperparameter. The adjustment threshold can control the degree to which the decision boundary or crowd is close to the unfiltered, noisy data. The closer the boundary is to the noisy data, the larger the coverage of the data samples to be retained for defect prediction. However, a larger coverage may reduce the prediction accuracy. According to one embodiment of the present invention, the desired coverage and / or accuracy are taken as input, and the training module 106 selects an appropriate adjustment threshold as a function of the incoming input. 【0055】 In block 208, a trained anomaly filter and a trained defect detection model are used during deployment to identify defects in products such as display panels. According to one embodiment of the present invention, the inference module 108 applies an anomaly filter to predict the confidence level of data samples captured by the data acquisition circuit 100 during the production process. According to one embodiment of the present invention, the anomaly filter identifies data samples that (if the filter is trained through unsupervised learning) cannot be confidently placed into one of the known classification groups, or (if the filter is trained through supervised learning) have features / parameters that cause the data to be classified as noisy / uncertain, and removes such data samples from the captured data set. The removed uncertain data samples can be seen as anomaly data that may cause a degradation in the quality of equipment used in the production process. 【0056】 According to one embodiment of the present invention, the inference module 108 applies a defect detection model to generate flawless, high-confidence data samples. In this manner, the predictive accuracy of the defect detection model can be improved compared to conventional defect detection models. 【0057】 Figure 3 is a more specific flowchart of confidence learning in block 202 according to one embodiment of the present invention. In block 300, the training module 106 calculates an error matrix / confusion matrix between the predicted (true / fitted) labels and the (human-given) labels of the test data set. A deep learning model can be applied to predict the true / fitted labels of data samples in the test data set. The confusion matrix can be generated based on a comparison of predicted labels to given labels. The confusion matrix can be a joint distribution between the predicted labels and the given labels for each predicted label. For example, given three possible label classes, apple, pear, and orange, the confusion matrix could have the first item representing the probability that a data sample predicted to be apple is actually displayed as apple, the second item representing the probability that a data sample predicted to be apple is actually displayed as pear, and the third item representing the probability that a data sample predicted to be apple is actually displayed as orange. A similar joint distribution can be calculated for the predictions of pear and orange. 【0058】 In block 302, the training module 106 calculates a threshold based on the confusion matrix for each predicted label. According to one embodiment of the present invention, a joint probability value is used as the threshold. According to one embodiment of the present invention, the threshold may be based on the peak signal-to-noise ratio (PSNR) for the predicted class, which can be calculated based on the joint probability distribution for the predicted class. According to one embodiment of the present invention, the threshold for a particular predicted class may be based on the difference between the probability of the predicted true label and the probability of that class. An example of pseudocode for calculating the threshold may be as follows: 【0059】 Obtain a set of prediction probabilities(a matrix of size:n_samples*n_classes) For each class c in n_classes: Calculate(difference of class c)=(probability of the predicted true label)-(the probability of the class c);(size:n_samples*1) Find the k-th smallest value of the difference of class c, as the threshold of the class c 【0060】 In block 304, the training module 106 identifies noisy and uncertain data in the training data set based on a calculated threshold. For example, if the joint probability distribution for apples being labeled as pears is 14%, the training module 106 can identify the 14% of data samples labeled as pears that have the highest probability of being apples as noisy data. According to one embodiment of the present invention, samples are identified as noisy data samples if the difference between the predicted true label and the probability of the class is smaller than a threshold set for that class. 【0061】 In block 306, the training module 106 labels and filters out noisy data in the training data set. For example, the training module 106 can label noisy data as "noisy" or a similar label. 【0062】 Figure 4 shows an example of a confusion matrix according to one embodiment of the present invention. In the example in Figure 4, the joint probability 400 for a data sample that is predicted to be an apple and is actually labeled as an apple is 0.25. Also, the joint probability 402 for a data sample that is predicted to be an apple but is actually labeled as a pear is 0.14. 【0063】 Figure 5 is a block diagram of a defect detection model implemented as a simultaneous fusion model according to one embodiment of the present invention. The simultaneous fusion model includes a first neural network branch 500 that receives a first set of cleaned data that has undergone confidence learning, and a second neural network branch 502 that receives a second set of cleaned data that has undergone confidence learning. According to one embodiment of the present invention, the training module 106 trains each branch independently and combines the first branch 500 and the second branch 502 as a simultaneous fusion model 504 through a convolutional layer. The first data set is internally sorted, and the second data set is also internally sorted, but the first data set and the second data set may not be sorted relative to each other. According to one embodiment of the present invention, the first data set may include spectrometer images such as energy-dispersive X-ray spectroscopy (EDS) images used in conjunction with high-angle annular dark-field (HAADF) images, and the second data set may include microscopic images such as transmission electron microscope (TEM) images. 【0064】 According to one embodiment of the present invention, each of the first branch 500 and the second branch 502 includes an attention module. The attention module of the neural network branch [e.g., the first neural network branch 500 or the second neural network branch 502] can overlay spatial attention onto the image received by the neural network branch and display areas where defects may occur. For example, the first attention module of the first branch 500 overlays a first special attention heat map onto a first data set received by the first branch 500, and the second attention module of the second branch 502 overlays a second special attention heat map onto a second data set received by the second branch 502. The attention module may include a spatial map network that is adjusted based on the final predicted label (error type / no error) of the input image (e.g., corresponding to the spatial attention heat map). The spatial map network can show the spatial relationship between the input data and the final predicted label. 【0065】 The first data set may be a set of spectrometer images, which can be entered through various channels (X channels in this example), and each channel may represent data associated with a specific chemical element or composition. Each neural network branch may include a channel recognition module and a spatial recognition module in the form of a CBAM (convolutional block attention module) (described later). Branches that use multiple image sources, such as the first branch 500, may also include an extra channel recognition module. The extra channel recognition module indicates which element input channels to focus on. According to one embodiment of the present invention, in a simultaneous fusion model, product information obtained from completely different data acquisition circuits 100 can be integrated and trained together, allowing the information to complement each other and make predictions for product manufacturing defects. 【0066】 According to one embodiment of the present invention, the spatial recognition module and the channel recognition module are networks that are trained in a semi-supervised manner to ensure that larger neural networks (e.g., neural network branches) give higher weights to data coming from selected channels or spatial regions. During training, the spatial / channel recognition module learns which features are associated with errors and, ultimately, which spatial regions or channels are associated with errors through the associated features. Once trained, the module operates within a larger neural network structure to "pay more attention" to the network in order to select regions / channels (for example, by setting one or more weights associated with those regions / channels). In one embodiment of the present invention, the recognition module is contained within a CBAM, which is an effective recognition module for feed-forward convolutional neural networks. Spectrometer branches and microscope branches can both include CBAMs that provide spatial and channel recognition. Spatial recognition may be a spatial heatmap associated with error locations, and channel recognition may be associated with color / grayscale channels of data. 【0067】 As mentioned above, within the first branch 500, there may be additional channel recognition modules in addition to CBAM. CBAM provides spatial heatmaps and color-channel recognition properties. Therefore, additional channel recognition modules can focus on channels associated with target elements related to specific defect types. 【0068】 Figure 6 is a conceptual layout diagram of the data filtered by the anomaly filter 602 in prediction step 208 according to one embodiment of the present invention. In the embodiment shown in Figure 6, data samples 600 obtained by the data acquisition circuit 100 for the produced products are provided to the anomaly filter 602, and the confidence level of the data samples is predicted (604). The data samples may include first type (e.g., flawless / confident) data samples 600a-600d and second type (e.g., noisy / uncertain) data samples 600e. 【0069】 According to one embodiment of the present invention, the anomaly filter 602 is an unsupervised learning filter 602a that identifies the class to which a data sample belongs based on the data sample and associated data mediation variables. In the embodiment shown in Figure 6, data samples of type 1, 600a-660d, are crowd-classified into appropriate classes 606a-606d. Class boundaries can be set based on adjustment thresholds. When a data sample is predicted to fall within the boundaries of an identified class, it can be considered confidence data, and a defect detection model can use it for defect prediction. 【0070】 In the embodiment shown in Figure 6, the second type of data sample 600e does not belong to any group and can therefore be considered noisy / uncertain data. According to one embodiment of the present invention, the second type of data sample 600e is treated as defective, and the defect detection model does not use that data sample for defect prediction. 【0071】 According to one embodiment of the present invention, the anomaly filter is a supervised learning filter 602b that classifies a data sample 600 as either flawless / certain 610 or noisy / uncertain 612. As with the unsupervised learning filter, the decision boundary 614 that separates the certain data from the uncertain data can be set by an adjustment threshold. According to one embodiment of the present invention, if a data sample is predicted to be noisy / uncertain, that data sample is treated as a fault, and the defect detection model does not use them for defect prediction. 【0072】 Figure 7 is a graph illustrating the trade-off between accuracy and scope when selecting an adjustment threshold according to one embodiment of the present invention. This graph shows that the predictive accuracy of the defect detection model increases as the scope decreases (i.e., as the data sample is filtered more). 【0073】 Figure 8 is a graph showing an example of an adjustment threshold calculated as a function of range and accuracy by one embodiment of the present invention. This graph shows that at 95% accuracy, the accuracy cutoff line 802 includes a certain accuracy curve 800. The 95% accuracy point occurs at intersection 804. The range cutoff line 806, which intersects the 95% accuracy at intersection 804, intersects the range curve 808 at approximately 65% ​​range and has an adjustment threshold of approximately 0.17. Therefore, in this example, an adjustment threshold of 0.16 yields 95% predicted accuracy and 65% range. On the other hand, as shown through the range cutoff line 810, lowering the range to 50% [intersection 812] increases the predicted accuracy to approximately 97% [intersection 814]. The corresponding adjustment threshold also increases to approximately 0.57. According to one embodiment of the present invention, the training module 106 executes a function that outputs an adjustment threshold based on the input of expected accuracy values ​​and / or range values. 【0074】 According to one embodiment of the present invention, the aforementioned manufacturing defect identification system and method can be implemented with one or more “processors.” The term processor can mean one or more processors and / or one or more processing cores. One or more processors may be located in a single device or distributed across multiple devices [e.g., a cloud system]. Processors can include, for example, programmable logic devices such as application-specific integrated circuits (ASICs), general-purpose or dedicated central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and FPGAs. In such processors as used herein, each function can be performed by hardware that performs the function (physically embedded hardware) or by general-purpose hardware such as a CPU that executes instructions stored in a non-transitory storage medium (e.g., memory). Processors can be fabricated on a single printed circuit board (PCB) or distributed across interconnected printed circuit boards. Processors can include other processors, for example, FPGAs and CPUs interconnected on a printed circuit board. 【0075】 Terms such as “first,” “second,” and “third” are used for various elements, components, regions, layers, and parts, but these various elements, components, regions, layers, and parts are not limited by modifiers such as “first,” “second,” and “third.” Terms such as “first,” “second,” and “third” are used to distinguish one element, component, region, layer, or part from other elements, components, regions, layers, and parts. Therefore, for example, the first element, first component, first region, first layer, and first part may also be called the second element, second component, second region, second layer, and second part, to the extent that they do not depart from the spirit and scope of the present invention. 【0076】 The terms used herein are for the purpose of describing specific embodiments and are not intended to limit the invention. Here, “substantially,” “about,” “approximately,” and similar expressions are merely approximations and not expressions of “degree,” but are used to indicate the inherent error of a measured or calculated value that a person skilled in the art would know. 【0077】 Here, unless otherwise specified, numbers include both singular and plural forms. The expression “includes” a certain feature, integer, stage, action, part, component, etc., may also include other features, integers, stages, actions, parts, components, etc., in addition to the part in question. The expression “and / or” includes all combinations of one or more of the listed items. Expressions such as “at least one” that precede the listed items modify the listed items as a whole, not each individual item within them. Furthermore, the expression “may be” or “may be” used when describing one embodiment of the present invention means that it is applicable to “one or more embodiments of the present invention.” Also, the term “exemplary” indicates one example. “Use,” “utilize,” etc., can be used with similar meanings together with other similar expressions. 【0078】 Embodiments of a manufacturing defect identification system and method have been described and illustrated above, but those skilled in the art can modify and alter such embodiments. Therefore, other manufacturing defect identification systems and methods configured according to the principles presented herein are also included in the present invention. The present invention is defined by the following claims and their equivalents. [Explanation of Symbols] 【0079】 100: Data acquisition circuit 101: Processor 102: Analysis System 103: Memory 106: Training Module 108: Inference Module 500: First neural network branching 502: Second neural network branching 504: Simultaneous Fusion Model

Claims

[Claim 1] A step in which a processor identifies a first data sample that satisfies a first criterion from a training data set, The processor removes the first data sample from the training data set and outputs a filtered training data set that includes the second data sample. The processor trains a first machine learning model with the filtered training data set. The processor trains a second machine learning model based on at least one of the first data sample and the second data sample. The processor receives first product data associated with a first manufactured product. The processor applies the second machine learning model to the confidence prediction of the first product data, and The processor includes the step of applying the first machine learning model to generate classifications based on the first product data in response to the confidence prediction of the first product data. The step of training the second machine learning model includes the step of the processor applying unsupervised learning based on the second data sample, The second data sample is associated with a specific class. The stage of training the second machine learning model is as follows: The processor identifies a cluster associated with the specific class, and The processor adjusts the boundary of the crowd based on the adjustment threshold. It further includes, The processor, in response to a determination that the first product data falls within the boundaries of the crowd, applies the first machine learning model to generate the classification. Method for classifying manufacturing defects. [Claim 2] The processor identifies a first data sample from the training data set that satisfies a first criterion, The processor removes the first data sample from the training data set and outputs a filtered training data set that includes the second data sample. The processor trains a first machine learning model with the filtered training data set. The processor trains a second machine learning model based on at least one of the first data sample and the second data sample. The processor receives first product data associated with a first manufactured product. The processor applies the second machine learning model to the confidence prediction of the first product data, and The processor includes the step of applying the first machine learning model to generate classifications based on the first product data in response to the confidence prediction of the first product data. The step of training the second machine learning model includes the step of the processor applying supervised learning based on the first and second data samples, The first data sample is identified as the first data type, and the second data sample is identified as the second data type. The stage of training the second machine learning model is as follows: The processor identifies a decision boundary that separates the first data type from the second data type, and The processor adjusts the decision boundary based on the adjustment threshold. It further includes, The processor applies the first machine learning model to generate the classification in response to a determination that the first product data belongs to the second data type. Method for classifying manufacturing defects. [Claim 3] The manufacturing defect classification method according to claim 1 or 2, wherein the first criterion is a confidence level below a set threshold. [Claim 4] The manufacturing defect classification method according to claim 1 or 2, wherein the second data sample is associated with a confidence level above a set threshold. [Claim 5] The processor identifies second product data associated with a second production product, The step of applying the second machine learning model to the confidence prediction of the second product data, and The processor processes the second product data as defective based on the confidence level of the second product data. A manufacturing defect classification method according to claim 1 or 2, further comprising: [Claim 6] The processor further includes the step of generating a signal based on the classification, The manufacturing defect classification method according to claim 1 or 2, wherein the signal triggers an action. [Claim 7] Processor, and Memory for storing instructions Includes, When the processor executes the instruction, the processor: From the training data set, identify the first data sample that satisfies the first criterion. From the aforementioned training data set, the first data sample is removed, and a filtered training data set containing the second data sample is output. The first machine learning model is trained on the filtered training data set. A second machine learning model is trained based on at least one of the first and second data samples. Receive first product data related to the first manufactured product. The second machine learning model is applied to the confidence prediction of the first product data. In response to the confidence prediction of the first product data, the first machine learning model is applied to generate classifications based on the first product data. The training of the second machine learning model includes applying unsupervised learning based on the second data sample, The second data sample is associated with a specific class. The training of the second machine learning model described above is Identify the cluster associated with the aforementioned specific class, Further includes adjusting the boundary of the crowd based on an adjustment threshold, In response to the determination that the first product data falls within the boundaries of the crowd, the first machine learning model is applied to generate the classification. Manufacturing defect classification system. [Claim 8] A processor, and Memory for storing instructions Includes, When the processor executes the instruction, the processor: From the training data set, identify the first data sample that satisfies the first criterion. From the aforementioned training data set, the first data sample is removed, and a filtered training data set containing the second data sample is output. The first machine learning model is trained on the filtered training data set. A second machine learning model is trained based on at least one of the first and second data samples. Receive first product data related to the first manufactured product. The second machine learning model is applied to the confidence prediction of the first product data. In response to the confidence prediction of the first product data, the first machine learning model is applied to generate classifications based on the first product data. The training of the second machine learning model includes applying supervised learning based on the first and second data samples, The first data sample is identified as the first data type, and the second data sample is identified as the second data type. The training of the second machine learning model described above is Identify a decision boundary that separates the first data type from the second data type. Further includes adjusting the decision boundary based on an adjustment threshold, In response to the determination that the first product data belongs to the second data type, the first machine learning model is applied to generate the classification. Manufacturing defect classification system. [Claim 9] The manufacturing defect classification system according to claim 7 or 8, wherein the first criterion is a confidence level below a set threshold. [Claim 10] The manufacturing defect classification system according to claim 7 or 8, wherein the second data sample is associated with a confidence level above a set threshold. [Claim 11] If the processor executes the instruction, the processor additionally: Identify second product data related to the second production product, The second machine learning model is applied to the confidence prediction of the second product data, A manufacturing defect classification system according to claim 7 or 8, wherein the second product data is processed as defective based on the confidence level of the second product data. [Claim 12] If the processor executes the instruction, the processor additionally generates a signal based on the classification, The manufacturing defect classification system according to claim 7 or 8, wherein the signal triggers an action. [Claim 13] A data acquisition circuit for collecting an input data set, and A processing circuit having logic is connected to the aforementioned data acquisition circuit. Includes, The aforementioned logic is, From the training data set, identify the first data sample that satisfies the first criterion. From the aforementioned training data set, the first data sample is removed, and a filtered training data set containing the second data sample is output. The first machine learning model is trained on the filtered training data set. A second machine learning model is trained based on at least one of the first and second data samples. We receive product data related to manufactured products. The second machine learning model is applied to the confidence prediction of the product data, In response to the confidence prediction of the product data, the first machine learning model is applied to generate classifications based on the product data. The training of the second machine learning model includes applying unsupervised learning based on the second data sample, The second data sample is associated with a specific class. The training of the second machine learning model described above is Identify the cluster associated with the aforementioned specific class, Further includes adjusting the boundary of the crowd based on an adjustment threshold, In response to the determination that the product data falls within the boundaries of the crowd, the first machine learning model is applied to generate the classification. Manufacturing defect classification system. [Claim 14] A data acquisition circuit for collecting an input data set, and A processing circuit having logic is connected to the aforementioned data acquisition circuit. Includes, The aforementioned logic is, From the training data set, identify the first data sample that satisfies the first criterion. From the aforementioned training data set, the first data sample is removed, and a filtered training data set containing the second data sample is output. The first machine learning model is trained on the filtered training data set. A second machine learning model is trained based on at least one of the first and second data samples. We receive product data related to manufactured products. The second machine learning model is applied to the confidence prediction of the product data, In response to the confidence prediction of the product data, the first machine learning model is applied to generate classifications based on the product data. The training of the second machine learning model includes applying supervised learning based on the first and second data samples, The first data sample is identified as the first data type, and the second data sample is identified as the second data type. The training of the second machine learning model described above is Identify a decision boundary that separates the first data type from the second data type. Further includes adjusting the decision boundary based on an adjustment threshold, In response to the determination that the product data belongs to the second data type, the first machine learning model is applied to generate the classification. Manufacturing defect classification system. [Claim 15] The manufacturing defect classification system according to claim 13 or 14, wherein the first criterion is a confidence level below a set threshold.

Images