System and method for classifying products

Abstract

A system and method for classifying products. A processor generates a first instance and a second instance of a first classifier and trains the instances based on an input dataset. A second classifier is trained based on an input, wherein the second classifier is configured to learn a representation of a latent space associated with the input. A first supplemental dataset is generated in the latent space, wherein the first supplemental dataset is an unlabeled dataset. A first prediction for labeling the first supplemental dataset is generated based on the first instance of the first classifier; and a second prediction for labeling the first supplemental dataset is generated based on the second instance of the first classifier. A labeled annotation of the first supplemental dataset is generated based on the first prediction and the second prediction. A third classifier is trained based on at least the input dataset and the annotated first supplemental dataset.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 62 / 909,053, filed October 1, 2019, entitled “KNOWLEDGE DISTILLATION FOR DATA INSUFFICENCECY,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] One or more aspects of embodiments of this disclosure relate to classifiers, and more specifically to systems and methods for training machine learning (ML) classifiers in situations where training data is scarce and imbalanced. Background Technology

[0004] In recent years, the mobile display industry has developed rapidly. With the deployment of new types of display panel modules and production methods, it has become increasingly difficult to inspect surface defects using only traditional mechanisms. There is a growing expectation to use artificial intelligence (AI) to automatically predict whether manufactured display panel modules have defects (e.g., whether they are classified as "defective" or "good"). In fact, there is a desire to use AI to predict defects in other hardware products, not just display panel modules.

[0005] AI / ML (Artificial Intelligence / Machine Learning) models that can be generalized to new, previously unseen data (referred to as "generalized ML models") are well-suited for such tasks. However, generating generalized ML models is challenging when the training dataset is insufficient and highly imbalanced. Due to the nature of creating datasets, insufficient datasets and class imbalance are unavoidable problems, making the generation of generalized machine learning models a daunting task.

[0006] Therefore, there is a need for an improved system and method to steadily and stably build AI / ML models from a limited amount of training data that may be imbalanced on binary classes. Summary of the Invention

[0007] Embodiments of the present disclosure are directed to a method of classifying products manufactured via a manufacturing process. A processor receives an input dataset, generates at least a first instance and a second instance of a first classifier, and trains the first instance and the second instance of the first classifier based on the input dataset. A second classifier is also trained based on the input dataset, where the second classifier is configured to learn a representation of a latent space associated with the input dataset. The processor further generates a first supplemental dataset in the latent space, where the first supplemental dataset is an unlabeled dataset. The processor generates a first prediction for labeling the first supplemental dataset based on the first instance of the first classifier, and further generates a second prediction for labeling the first supplemental dataset based on the second instance of the first classifier. The processor generates a labeled annotation of the first supplemental dataset based on the first prediction and the second prediction. The processor further trains a third classifier based on at least the input dataset and the annotated first supplemental dataset. The trained third classifier is configured to receive data of a product to be classified to output a prediction of the product based on the received data.

[0008] In one embodiment, each of the first classifier, the second classifier, and the third classifier is a neural network.

[0009] In one embodiment, the second classifier is a variational autoencoder.

[0010] In one embodiment, the latent space provides a compressed representation of the input dataset.

[0011] In one embodiment, the generation of the first supplemental dataset includes generating random data elements in the latent space.

[0012] In one embodiment, the generation of the labeled annotation includes: determining a majority class based on the first prediction and the second prediction; and labeling the first supplemental dataset based on the majority class.

[0013] In one embodiment, the generation of the labeled annotation includes: determining a first probability of the first prediction and a second probability of the second prediction; calculating an average of the first probability and the second probability; and identifying a class of the first supplemental dataset based on the calculated average.

[0014] In one embodiment, the generation of the labeled annotation includes: determining a first probability of a majority class and a second probability of a minority class based on the first prediction; determining a third probability of the majority class and a fourth probability of the minority class based on the second prediction; calculating a first average of the first probability and the third probability; calculating a second average of the second probability and the fourth probability; and labeling the first supplemental dataset based on the calculated first average and the second average.

[0015] In one embodiment, the method for classifying products further comprises: generating a second supplemental data set based on oversampling of the latent space; and training a third classifier based on the input data set, the annotated first supplemental data set, and the second supplemental data set.

[0016] Embodiments of the present disclosure are also directed to a system for classifying products manufactured via a manufacturing process. The system includes a processor and a memory, wherein the memory has stored therein instructions that, when executed by the processor, cause the processor to: receive an input data set; generate at least a first instance and a second instance of a first classifier; train the first instance and the second instance of the first classifier based on the input data set; train a second classifier based on the input data set, wherein the second classifier is configured to learn a representation of a latent space associated with the input data set; generate a first supplemental data set in the latent space, wherein the first supplemental data set is an unlabeled data set; generate a first prediction for labeling the first supplemental data set based on the first instance of the first classifier; generate a second prediction for labeling the first supplemental data set based on the second instance of the first classifier; generate a labeled annotation of the first supplemental data set based on the first prediction and the second prediction; and train a third classifier based on at least the input data set and the annotated first supplemental data set. The trained third classifier is configured to receive data of a product to be classified to output a prediction of the product based on the received data.

[0017] Embodiments of the present disclosure are also directed to a system for classifying products manufactured via a manufacturing process. The system includes a processor and a memory, wherein the memory has stored therein instructions that, when executed by the processor, cause the processor to: receive an input data set; generate at least a first instance and a second instance of a first classifier; train the first instance and the second instance of the first classifier based on the input data set; train a second classifier based on the input data set, wherein the second classifier is configured to learn a representation of a latent space associated with the input data set; generate a first supplemental data set in the latent space, wherein the first supplemental data set is an unlabeled data set; generate a first prediction for labeling the first supplemental data set based on the first instance of the first classifier; generate a second prediction for labeling the first supplemental data set based on the second instance of the first classifier; generate a labeled annotation of the first supplemental data set based on the first prediction and the second prediction; and train a third classifier based on at least the input data set and the annotated first supplemental data set. The trained third classifier is configured to receive data of a product to be classified to output a prediction of the product based on the received data. BRIEF DESCRIPTION OF DRAWINGS

[0018] These and other features and advantages of the present disclosure will be appreciated and understood by those skilled in the art from the detailed description, claims, and drawings, where:

[0019] Figure 1 This is a system for classifying parts as defective or non-defective, according to an exemplary embodiment;

[0020] Figure 2 This is based on an exemplary embodiment. Figure 1 A conceptual block diagram of the classifier engine in the system;

[0021] Figure 3 According to an exemplary embodiment, by Figure 2 A flowchart illustrating the process executed by the classifier engine to generate and train student models;

[0022] Figure 4 This is based on an exemplary embodiment. Figure 1 A more detailed block diagram of the variational autoencoder in the system;

[0023] Figure 5 This is a more detailed flowchart of a process for latent space oversampling according to an exemplary embodiment;

[0024] Figure 6A It is a graph showing the distribution of example input datasets that can be used to train deep learning models;

[0025] Figures 6B to 6E Based on Figure 6A Example diagram showing how the input dataset produces multiple classifier instances with different decision boundaries;

[0026] Figure 7A It is a graph showing the distribution of the example input dataset that has been augmented with additional samples; and

[0027] Figure 7B Depicting based on Figure 7A The expanded training dataset produces a graph of multiple classifier instances with different decision boundaries. Detailed Implementation

[0028] The detailed description set forth below with reference to the accompanying drawings is intended as a description of exemplary embodiments of the systems and methods for data expansion provided in this disclosure, and is not intended to represent the only forms in which this disclosure may be constructed or utilized. The description, in conjunction with the illustrated embodiments, illustrates the features of this disclosure. However, it should be understood that the same or equivalent functions and structures may be implemented through different embodiments, which are also intended to be covered within the scope of this disclosure. As indicated elsewhere herein, the same reference numerals are intended to indicate the same elements or features.

[0029] Manufacturing processes, such as mobile display manufacturing processes, can acquire digital tracking data during the manufacturing of mobile display products. Although mobile display products are used as an example, one skilled in the art will recognize that embodiments of the present disclosure can be applied to manufacturing processes of other glass and non-glass products, including, for example, the manufacturing of wafers and semiconductor glass.

[0030] Tracking data is collected via one or more sensors that can be placed, for example, on a conveyor belt that carries the product during production. The sensors are configured to record any sensed activity as tracking data. The sensors can be, for example, a plurality of temperature and pressure sensors configured to capture measurements of temperature and pressure over time in the manufacturing process. Each sensor can be sampled multiple times (e.g., every few seconds over the glass or wafer being manufactured in the chamber).

[0031] It is desirable to use a classifier to automatically analyze the tracking data to predict manufactured products that are flawed (“bad”) from products that are not flawed (“good”). Thus, the classifier needs to be trained to assign data samples to one of two classes (“good” and “bad”). However, the training data used to train such a classifier can be highly imbalanced. For example, in a manufacturing process for manufacturing electronic parts, it is likely that the majority of the parts are acceptable or “good” and a small number of the parts are defective or “bad” to some extent. The number of “good” parts can be in the range of 100 to 200 times more than the number of bad parts. To this end, when data is obtained during the manufacturing and testing process, the majority of the data can come from good devices, resulting in an imbalance of the training data. Furthermore, it is often difficult to expect a sufficient data set to train a deep learning model due to the rapid change of product models. The life of a product model is short, so it is difficult to collect a sufficient number of data samples. It is also often necessary to generate a new AI / ML model for a new product model to detect the flaw condition of the new product model. Thus, generating a generalized model with a limited data set is a general challenge.

[0032] There are data augmentation techniques that can be used to artificially expand the size of the training data set. One technique can be, for example, to generate all possible data sample cases to use as training data. However, considering all possible cases generates a data set that is too large. In addition, although samples can be expanded, they can not all be meaningful when training a model.

[0033] Another technique often used for visual recognition / image classification is data distillation. Data distillation is a general approach for omni-supervised learning that distills knowledge from unlabelled data by generating annotations for unlabelled data. The annotations are generated using a model trained on a large amount of labelled data. The model is then retrained using the extra generated annotations. The extra unlabelled data can be generated via geometric / spatial transformations of the labelled data (e.g., flipping, shifting, and / or rotating). Additional information on data distillation is provided by Ilija Radosavovic et al., “Data Distillation: Towards Omni-Supervised Learning,” in Computer Vision and Pattern Recognition (2018), available at https: / / arxiv.org / abs / 1712.04440, the contents of which are incorporated by reference herein.

[0034] However, geometric transformations commonly used to augment data for image classification are not applicable to tracking data because it is not known which transformations will allow the characteristics of the input data to be maintained after the transformations are applied. Additionally, for defect samples, there is often a correlation between the manufacturing task and the defect pattern generation, so it cannot be guaranteed that the quality of the classifier will be improved by applying known data augmentation techniques. Thus, conventional data augmentation techniques are not suitable for expanding defect or tracking data for training purposes. Data distillation suggests using unlabelled datasets in Internet-level datasets, but similar data samples for a particular company’s manufacturing dataset cannot be found on the Internet.

[0035] In view of the deficiencies in the prior art, it is desirable to have a system and method for an omni-supervised learning approach that generates generalized ML models that address the data insufficiency and class imbalance problems in labelled datasets. According to various embodiments, the insufficiency and class imbalance of the input dataset is addressed via data augmentation using deep generative models. Specifically, the data augmentation is performed in the manifold (latent) space rather than the input data space.

[0036] Additionally, since one of the main problems in generating generalized AI / ML models is data insufficiency, this problem can be addressed by using deep generative model data generation to incorporate more variations into the dataset. In some embodiments, the augmented / complemented data is an unlabelled dataset generated from the latent space using random inputs. In some embodiments, the augmented data consists of synthetic data samples obtained from oversampling the latent space.

[0037] According to one embodiment, knowledge is distilled from multiple instances of a baseline machine learning model to annotate an unlabeled dataset and generate a new annotated dataset. By feeding the unlabeled dataset to each of the multiple instances of the trained ML model, knowledge from the multiple instances can be distilled. The unlabeled data can then be labeled based on the output of each of the ML model instances using an ensemble mechanism. Thus, knowledge distillation can be implemented from data-dense models (e.g., individual ML model instances) while broadening the ML model to achieve better classification accuracy than a traditional model alone. Once trained, the generalized ML model can be used to predict defective parts resulting from a manufacturing process.

[0038] Figure 1 is a system for classifying a part as defective or non-defective according to one example embodiment. In some embodiments, the system includes one or more data collection circuits 105 (which can include, for example, temperature and pressure sensors, amplifiers, and / or analog-to-digital converters), a data pre-processing circuit 110 (which can reformat data, as discussed in further detail later), and a classifier engine 115 (which can generate a deep learning (DL) neural network).

[0039] The data pre-processing circuit 110 can receive raw trace data (e.g., multiple time traces as noted above) from the data collection circuit 105 and can reformat the data into, for example, a two-dimensional array (e.g., a 224 x 224 array). In one embodiment, the data pre-processing circuit 110 includes one or more data storage devices for storing the raw trace data. The size of the two-dimensional array can be selected to be comparable to the size of images typically classified by a neural network. The reformatting can then make it possible to re-use particular portions of code used to implement a neural network classifier for images, for use in some embodiments.

[0040] The reformatted input data is provided to the classifier engine 115 for training or making predictions via a DL neural network. In this regard, the classifier engine 115 can be configured with logic or instructions stored on one or more tangible machine-readable media that can be executed by one or more processing devices for generating, training, and performing predictions via a DL neural network.

[0041] Figure 2 is a conceptual block diagram of the classifier engine 115 according to one example embodiment. During training, a labeled input dataset 200 is received from the pre-processing circuit 110. The labeled input dataset 200 includes a first number of data samples (e.g., thousands of data elements) each labeled as “good” (or “G”) and a second number of data samples (e.g., between 10 and 100 data elements) each labeled as “not good” (or “NG”).

[0042] The labeled input dataset 200 is used to train a classifier, which can be a deep learning neural network such as, for example, a variational autoencoder (VAE) 202. Although a VAE is used as an example, one skilled in the art will recognize that other deep generative models such as, for example, a generative adversarial network (GAN) or an autoregressive model, among others, can be used in place of the VAE. In embodiments that employ a VAE, the VAE 202 is configured to learn a latent / hidden / manifold space representation of the input dataset 200. Generally speaking, the latent space is composed of latent vectors, which are simpler / compressed (e.g., smaller dimensional) representations of the input dataset 200.

[0043] In some embodiments, the trained VAE 202 is invoked for statistically generating additional training data (also referred to as synthetic data samples). In this regard, the VAE 202 can be coupled to an oversampling module 204 and a random generation module 206. The oversampling module 204 can be invoked for oversampling the latent space learned by the VAE 202 to generate additional “bad” samples. The random generation module 206 can be invoked for generating supplemental samples in the latent space using random inputs. According to one embodiment, the randomly generated data samples are unlabeled data samples.

[0044] In one embodiment, the classifier engine 115 is further configured to create multiple (e.g., two or more) classifier instances 208a-c (collectively referred to as 208) of the binary baseline classifier. Each classifier instance 208 can be, for example, a neural network.

[0045] The randomly generated data samples from the random generation module 206 are provided to each of the different classifier instances 208 for knowledge distillation. In this regard, the integration of the predictions from the classifier instances 208 on the randomly generated data samples is used to train the student classifier / model 210. That is, the aggregation of the predictions made by the different classifier instances 208 generates new knowledge that can be distilled and used to train the student classifier 210.

[0046] In one embodiment, in addition to the distilled knowledge from the classifier instances 208, the original labeled input dataset 200 and / or additional samples from the oversampling module 204 are used to train the student classifier 210. Once trained, the student classifier 210 can be fed new tracking data obtained for a newly manufactured product to predict whether the product is “good” or “bad.” In one embodiment, the trained student classifier 210 can be used by an inspection system (not shown) to make this type of prediction for a newly manufactured product. In one embodiment, if the product is predicted to be “bad,” the inspection system can output a message or recommendation regarding an action to be taken. For example, the action can be to remove the flawed product for further testing, fixing, or discarding. Experiments have shown that the trained student model 210 will run stably and within a threshold level of classification accuracy (e.g., 90% or higher accuracy) for new tracking data.

[0047] Figure 3 is a flowchart of a process performed by the classifier engine 115 for generating and training a student model 210 according to one example embodiment. In block 300, a labeled input dataset 200 is received by the classifier engine 115.

[0048] In block 302, two or more classifier instances 208 of a baseline classifier are trained (e.g., using supervised learning) using the input dataset 200. Each classifier instance 208 can be generated by initializing the baseline classifier into a random initial state. For example, the initial weights of the baseline classifier can be randomly set at initialization. By training instances of a single model, rather than training separate and independent models that are often large and complex, can result in more convenient and more economical training (in terms of computational resources).

[0049] In block 308, the input dataset 200 received in block 300 is also used to train the VAE 202. In one embodiment, the VAE 202 is trained to learn a representation of the input dataset 200 in a latent space. Once trained, the VAE 202 can be used to generate augmented data in blocks 310 and 314. In this regard, in block 310, the classifier engine 115 invokes the trained VAE model to generate additional complementary unlabeled data samples (sometimes simply referred to as additional dataset) 312 in the latent space. The additional data samples can be generated using, for example, random inputs.

[0050] In blocks 304a through 304c, each of the classifier instances 208 receives the additional complementary unlabeled data samples 312 and generates predictions for the received samples.

[0051] In block 306, the integration of the prediction results from the different classifier instances 208 is used to generate annotations for the additional supplemental unlabeled data samples 312 and generate an annotated dataset 316. In this way, the knowledge obtained by invoking each classifier instance 208 can be distilled into the student classifier 210. Considering the aggregation of the predictions of the different classifier instances 208 helps to reduce the errors of the trained student model 210, even if each of the individual classifier instances 208 can be prone to errors when considered in isolation. Thus, it can be desirable to use an integration mechanism to stabilize the stability of the trained student model 210.

[0052] One of various integration methods can be used to generate annotations for the additional dataset 312. Such integration methods include, but are not limited to, hard voting, soft voting, and knowledge distillation. Methods based on hard voting integration take the majority of the predictions of the classifier instances 208 and label the additional dataset based on the majority decision. For example, if classifier instance A 208a and classifier instance B 208b predict the “good” class for a particular data sample, and classifier instance C 208c predicts the “bad” class, the data sample is labeled in the “good” class.

[0053] Methods based on soft voting integration take the average of the prediction probabilities of the different classifier instances 208, as follows:

[0054] where K is the number of classifier instances 208.

[0055] For example, if classifier instance A 208a predicts a 99% probability of “good”, classifier instance B 208b predicts a 49% probability of “good”, and classifier instance C 208c predicts a 49% probability of “good”, the average probability of the data sample being “good” across the different classifier instances 208 is (99% + 49% + 49%) / 3 = 65.67%. Thus, the integrated decision for the data sample is “good”.

[0056] Methods based on knowledge distillation integration take the average probability of multiple instances of a model, considering not only the majority class (e.g., “good”), but also the minority class (e.g., “bad”). In this regard, the average probability of the majority and minority classes is output as the integrated decision for the data. In some embodiments, knowledge distillation can extend to multiple class models. In such embodiments, the average probability can be obtained across multiple instances of the multiple class model.

[0057] According to one embodiment, in block 314, the classifier engine 115 also generates additional data samples for training the student model 210. In this regard, an oversampling algorithm is applied to generate additional data samples in the minority class (e.g., the “bad” class) to produce an oversampled dataset 318. In one embodiment, the oversampling is from the VAE latent space, rather than the original space or other latent spaces.

[0058] In block 320, the classifier engine 115 trains the student classifier 210 using: i) the original input dataset 200; ii) the annotated dataset 316; and / or iii) the oversampled dataset 318. The trained student classifier 210 can then be used as a binomial classifier to classify newly manufactured products as, for example, “good” or “bad” based on new tracking data acquired for the products.

[0059] Figure 4 is a more detailed block diagram of the VAE 202 according to one example embodiment. Generally speaking, the VAE 202 is a neural network that can include an encoder network (simply “encoder”) 400 and a decoder network (simply “decoder”) 402. The encoder 400 can map or encode each received data element (such as the input dataset 200) into a latent vector 404 that is constrained to have a distribution that approximates a unit Gaussian distribution (i.e., a vector distribution where the elements of the vector are, for example, independent Gaussian distributions each having a mean and a variance). In this regard, the VAE 202 employs a mean vector 406 and a standard deviation vector 408 to encode each received data element into a latent vector 404. As those skilled in the art will appreciate, the latent vector 404 is a compressed, low-dimensional representation of the input dataset 200. The space of all latent vectors 404 can be referred to as the latent space.

[0060] The decoder 402 can perform an approximate inverse operation of the encoder 400; it can map each latent vector 404 produced by the encoder 400 into a data element that approximates the (synthetic) data element that the encoder 400 would have mapped it to as a latent vector 404. The encoder 400 and the decoder 402 can be trained together using the input dataset 200.

[0061] Once the encoder 400 and the decoder 402 are trained, block 310 Figure 3generating supplemental data in the latent space. For example, random sampling can be performed on the latent space to generate random vectors (generated to have a unit Gaussian distribution) that can be fed into the decoder 402 to generate synthetic data elements (e.g., to generate an additional data set 312). Since the VAE 202 is trained using data elements from both classes (i.e., with both“good” data elements and“bad” data elements), the additional data set 312 is unlabelled.

[0062] Figure 5 is a more detailed flowchart of the process of the block 314 Figure 3 ) for latent space oversampling according to one example embodiment. In block 500, the latent space data set obtained by training the VAE model in block 308 is input to an oversampling module 502. In one embodiment, the oversampling module 502 is configured to sample attributes from instances in the minority class (“bad” class) to generate synthetic samples (e.g., an oversampled data set 318). Unlike traditional oversampling that uses the original input data set to increase the minority class in the original input data set, embodiments of the present disclosure allow for increasing the minority class data from the learned latent space.

[0063] Oversampling can be implemented using algorithms such as SMOTE (Synthetic Minority Over-sampling Technique) or ADASYN (Adaptive Synthetic) oversampling. Generally, SMOTE is an oversampling method that creates synthetic samples from the minority class rather than creating duplicates. According to one embodiment, the algorithm selects two or more similar instances in the latent space (using a distance measure) and perturbs the instances one attribute at a time by a random amount within the difference from the neighboring instances.

[0064] ADASYN can offset each of the resulting data elements by adding a small random vector (or“offset”) to it to reduce the likelihood that the synthetic samples from the minority class can interfere with other classes (e.g., the majority class which can be the“good” class).

[0065] In block 504, the oversampled data set 318 in the latent space is output.

[0066] Figure 6A is a plot of the distribution of an example input data set (similar to the input data set 200) that can be used to train a deep learning model. In Figure 6A the example, the input data set 200 is small and consists of data labeled as“good” 600 and data labeled as“bad” 602.

[0067] Figures 6B to 6E is based on Figure 6AAn example plot of multiple classifier instances (e.g., classifier instances 208) producing different decision boundaries 604a-d based on the input dataset 200. In one example, the varying decision boundaries are a result of the small amount of training data input. Due to the variation in decision boundaries 604a-d, each of the models is expected to make different classification decisions on future data.

[0068] Figure 7A An example input dataset augmented with additional training data 700 utilizing the data augmentation mechanism in accordance with various embodiments of the present disclosure. (Similar to Figure 6A An example plot of the distribution of the augmented training dataset based on

[0069] Figure 7B An example plot of multiple classifier instances (e.g., classifier instances 208) producing different decision boundaries 704a-c based on the augmented Figure 7A training dataset based on Figure 7B As depicted in

[0070] Experiments show that by applying knowledge distillation from the model instances, a generalized ML model can be steadily and stably generated without knowledge of the full test dataset. For example, some of the individual model instances have validation accuracy as low as 80% on the “bad” data. However, the generalized ML model generated using the data augmentation and knowledge distillation mechanisms of the present embodiments generates 90% or higher validation accuracy on the “bad” data.

[0071] In some embodiments, the data pre-processing circuit 110, the classifier engine 115, and one or more of the various modules and models / classifiers discussed above are implemented with one or more processing circuits. The term "processing circuit" is used herein to mean any combination of hardware, firmware, and software for processing data or digital signals. Processing circuit hardware can include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). In processing circuits as used herein, each function can be performed by hardware configured (i.e., hardwired) to perform that function, or by more general purpose hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. Processing circuits can be fabricated on a single printed circuit board (PCB), or can be distributed across several interconnected PCBs. Processing circuits can contain other processing circuits; for example, a processing circuit can include two processing circuits (an FPGA and a CPU) interconnected on a PCB.

[0072] It will be understood that, although the terms "first", "second", "third", and the like can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed herein could be termed a second element, component, region, layer or section without departing from the spirit and scope of the inventive concept.

[0073] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concept. As used herein, the terms "substantially", "approximately" and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art.

[0074] As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of," when preceding a list of two or more items, modify the list of items as a whole and do not modify the individual items of the list. Further, the use of "can", "may" and "might" when describing embodiments of the inventive concept indicates that one or more embodiments of the disclosure are so described. In addition, the term "exemplary" is intended to mean an example or illustration. As used herein, the terms "use", "in use", and "used" can be considered synonymous with the terms "utilize", "utilizing", and "utilized", respectively.

[0075] It will be understood that when an element or layer is referred to as being "on", "connected to", "coupled to" or "adjacent to" another element or layer, it can be directly on, connected, coupled or adjacent the other element or layer, or one or more intervening elements or layers can be present. In contrast, when an element or layer is referred to as being "directly on", "directly connected to", "directly coupled to" or "immediately adjacent to" another element or layer, then there are no intervening elements or layers present.

[0076] Any numerical range recited herein is intended to include all sub-ranges of the same numerical precision subsumed within the recited range. For example, a range of "1.0 to 10.0" is intended to include all sub-ranges between (and including) the recited minimum value of 1.0 and the recited maximum value of 10.0, that is, having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as, for example, 2.4 to 7.6. Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed therein and any minimum numerical limitation recited in this specification is intended to include all higher numerical limitations subsumed therein.

[0077] While example embodiments of systems and methods for knowledge distillation for model instances have been described and illustrated herein, a wide variety of modifications and changes will become apparent to those skilled in the art. Accordingly, it is the intention that the systems and methods for knowledge distillation constructed according to principles of the present disclosure can be embodied in a variety of other forms that may be apparent to those skilled in the art. The present disclosure is therefore intended to cover any and all modifications and changes as may be apparent to those skilled in the art. It is also intended to cover any and all equivalents.

Claims

1. A method for classifying products manufactured via a manufacturing process, the method comprising: receiving, by a processor, an input dataset from a plurality of sensors, the plurality of sensors comprising temperature sensors and pressure sensors; generating, by the processor, at least a first instance and a second instance of a first classifier; training, by the processor, the first instance and the second instance of the first classifier based on the input dataset; training, by the processor, a second classifier based on the input dataset, wherein the second classifier is configured to learn a representation of a latent space associated with the input dataset; generating, by the processor, a first supplementary dataset in the latent space, wherein the first supplementary dataset is an unlabeled dataset; generating, by the processor, a first prediction for labeling the first supplementary dataset based on the first instance of the first classifier; generating, by the processor, a second prediction for labeling the first supplementary dataset based on the second instance of the first classifier; generating, by the processor, a labeled annotation of the first supplementary dataset based on the first prediction and the second prediction; and training, by the processor, a third classifier based on at least the input dataset and the labeled first supplementary dataset, wherein the trained third classifier is configured to receive data of a product to be classified to output a prediction of the product based on the received data, the prediction being a classification of "good" or "bad".

2. The method of claim 1, wherein, each of the first classifier, the second classifier, and the third classifier is a neural network.

3. The method of claim 2, wherein, the second classifier is a variational autoencoder.

4. The method of claim 1, wherein, the latent space provides a compressed representation of the input dataset.

5. The method of claim 1, wherein, the generation of the first supplementary dataset comprises generating random data elements in the latent space.

6. The method of claim 1, wherein, the generation of the labeled annotation comprises: determining a majority class based on the first prediction and the second prediction; and labeling the first supplementary dataset based on the majority class.

7. The method of claim 1, wherein, the generation of the labeled annotation comprises: determining a first probability of the first prediction and a second probability of the second prediction; computing a mean of the first probability and the second probability; and identifying a class of the first supplementary dataset based on the computed mean.

8. The method of claim 1, wherein, the generation of the labeled annotation comprises: determining a first probability of a majority class and a second probability of a minority class based on the first prediction; determining a third probability of the majority class and a fourth probability of the minority class based on the second prediction; computing a first mean of the first probability and the third probability; computing a second mean of the second probability and the fourth probability; and labeling the first supplementary dataset based on the computed first mean and the second mean.

9. The method of claim 1, further comprising: generating a second supplementary dataset based on oversampling of the latent space; and training the third classifier based on the input dataset, the labeled first supplementary dataset, and the second supplementary dataset.

10. A system for classifying products manufactured via a manufacturing process, the system comprising: ​ a processor; and a memory, wherein the memory has stored therein instructions that, when executed by the processor, cause the processor to: receive an input dataset from a plurality of sensors, the plurality of sensors comprising a temperature sensor and a pressure sensor; generate at least a first instance and a second instance of a first classifier; train the first instance and the second instance of the first classifier based on the input dataset; train a second classifier based on the input dataset, wherein the second classifier is configured to learn a representation of a latent space associated with the input dataset; generate a first supplementary dataset in the latent space, wherein the first supplementary dataset is an unlabeled dataset; generate a first prediction for labeling the first supplementary dataset based on the first instance of the first classifier; generate a second prediction for labeling the first supplementary dataset based on the second instance of the first classifier; generate a labeled annotation of the first supplementary dataset based on the first prediction and the second prediction; and train a third classifier based on at least the input dataset and the labeled first supplementary dataset, wherein the trained third classifier is configured to receive data of a product to be classified to output a prediction of the product based on the received data, the prediction being a classification of "good" or "bad".

11. The system of claim 10, wherein, Each of the first classifier, the second classifier, and the third classifier is a neural network.

12. The system of claim 11, wherein, The second classifier is a variational autoencoder.

13. The system of claim 10, wherein, The latent space provides a compressed representation of the input dataset.

14. The system of claim 10, wherein, The instructions that cause the processor to generate the first supplementary dataset comprise instructions that cause the processor to generate random data elements in the latent space.

15. The system of claim 10, wherein, The instructions that cause the processor to generate the labeled annotation comprise instructions that cause the processor to: determine a majority class based on the first prediction and the second prediction; and label the first supplementary dataset based on the majority class.

16. The system of claim 10, wherein, The instructions that cause the processor to generate the labeled annotation comprise instructions that cause the processor to: determine a first probability of the first prediction and a second probability of the second prediction; compute an average of the first probability and the second probability; and identify a class of the first supplementary dataset based on the computed average.

17. The system of claim 10, wherein, The instructions that cause the processor to generate the labeled annotation comprise instructions that cause the processor to: determine a first probability of a majority class and a second probability of a minority class based on the first prediction; determine a third probability of the majority class and a fourth probability of the minority class based on the second prediction; compute a first average of the first probability and the third probability; compute a second average of the second probability and the fourth probability; and label the first supplementary dataset based on the computed first average and second average.

18. The system of claim 10, wherein, The instructions further cause the processor to: generate a second supplementary dataset based on oversampling of the latent space; and training the third classifier based on the input dataset, the first supplemental dataset of annotations, and the second supplemental dataset.

19. A system for classifying a manufactured part as a good or a bad, the system comprising: a data collection circuit including a plurality of sensors configured to collect an input dataset, the plurality of sensors including a temperature sensor and a pressure sensor; and a processing circuit coupled to the data collection circuit, the processing circuit having logic to: receive the collected input dataset; generate at least a first instance and a second instance of a first classifier; train the first instance and the second instance of the first classifier based on the input dataset; train a second classifier based on the input dataset, wherein the second classifier is configured to learn a representation of a latent space associated with the input dataset; generate a first supplemental dataset in the latent space, wherein the first supplemental dataset is an unlabeled dataset; generate a first prediction for labeling the first supplemental dataset based on the first instance of the first classifier; generate a second prediction for labeling the first supplemental dataset based on the second instance of the first classifier; generate labeled annotations of the first supplemental dataset based on the first prediction and the second prediction; and train a third classifier based on at least the input dataset and the first supplemental dataset of annotations, wherein the trained third classifier is configured to receive data of a product to be classified to output a prediction of the product based on the received data, the prediction being a classification of "good" or "bad."

Images