Product classification system and method of classifying products

Abstract

To provide a product classification system and a product classification method.SOLUTION: A method includes the steps of: generating at least a first and a second instances of a first classifier; training an instance based on an input data set; training a second classifier that learns representation of a latent space associated with the input data set based on the input data set; generating a first additional data set that is an unclassified data set in the latent space; generating first prediction for classifying the first additional data set based on a first instance of the first classifier; generating second prediction for classifying the first additional data set based on the second instance of the first classifier; generating a classification annotation for the first additional data set based on the first prediction and the second prediction; and training a third classifier based on at least the input data set and annotated first data set.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a product classification system and method, and in particular to a system and method for training a classifier.

[0002] This application claims priority to U.S. Patent Application No. 62 / 909,053, filed in the United States Patent and Trademark Office on October 1, 2019, the entire contents of which are incorporated herein by reference. [Background technology]

[0003] The mobile display industry has grown exponentially in the last few years. With the use of new types of display modules and production methods, it is becoming more difficult to detect surface defects using existing methods alone. It would be desirable to use artificial intelligence (AI) to automatically predict whether a manufactured display module is defective (i.e., whether it will be classified as "no good" or "good"). In fact, it would be desirable to use AI to predict defects not only in display modules but also in other hardware products. Summary of the Invention [Problem to be solved by the invention]

[0004] AI / ML (machine learning) models that can be applied well to new, previously unseen data (called "generalized ML models") are suitable for such tasks. However, it is difficult to create generalized ML models when the training dataset is insufficient and highly imbalanced. Due to the nature of the dataset, insufficiency and class imbalance are inevitable challenges, making it difficult to generate generalized machine learning models.

[0005] Therefore, there is a need for improved systems and methods for sustainably and stably building AI / ML models from a limited amount of training data that is imbalanced towards two classes. [Means for solving the problem]

[0006] A method for classifying products according to an embodiment of the present invention is a method for classifying products produced through a manufacturing process, the method comprising the steps of: receiving an input data set by a processor; generating at least first and second instances of a first classifier by the processor; training the first and second instances of the first classifier based on the input data set by the processor; and generating a latent space associated with the input data set by the processor based on the input data set. generating, by the processing device, a first additional data set, the first additional data set being an unlabeled data set in the latent space; generating, by the processing device, a first prediction for classifying the first additional data set based on the first instance of the first classifier; generating, by the processing device, a second prediction for classifying the first additional data set based on the second instance of the first classifier; generating, by the processing device, classification annotations for the first additional data set based on the first and second predictions; and training, by the processing device, a third classifier based on at least the input data set and the annotated first additional data set, the trained third classifier receiving data of a product to be classified and outputting a prediction for the product based on the received data.

[0007] According to one embodiment of the invention, each of the first, second and third classifiers may be a neural network.

[0008] According to one embodiment of the present invention, the second classifier may be a variational autoencoder (VAE).

[0009] According to one embodiment of the present invention, the latent space may provide a condensed representation of the input data set.

[0010] According to one embodiment of the invention, generating the first additional data set may include generating arbitrary data elements in the latent space.

[0011] According to one embodiment of the invention, the generating of the classification annotation may include determining a plurality of classes based on the first prediction and the second prediction, and classifying the first additional data set based on the plurality of classes.

[0012] According to one embodiment of the invention, generating the classification annotation may include determining a first probability of the first prediction and a second probability of the second prediction, calculating an average of the first and second probabilities, and defining a class for the first additional data set based on the calculated average.

[0013] According to one embodiment of the present invention, the generating of the classification annotation may include determining a first probability of a majority class and a second probability of a minority class based on the first prediction, determining a third probability of the majority class and a fourth probability of the minority class based on the second prediction, calculating a first average of the first and third probabilities, calculating a second average of the second and fourth probabilities, and classifying the first additional data set based on the calculated first and second averages.

[0014] According to an embodiment of the invention, the method may further include generating a second additional data set based on oversampling of the latent space, and training the third classifier based on the input data set, the annotated first additional data set, and the second additional data set.

[0015] In accordance with one embodiment of the present invention, a classification system for classifying a product produced during a manufacturing process includes a processor and a memory having instructions stored therein, the processor executing the instructions to receive an input data set, generate at least first and second instances of a first classifier, train the first and second instances of the first classifier based on the input data set, and compute a latent space associated with the input data set based on the input data set. train a second classifier to learn a representation of the input data set (a latent space); generate a first additional data set in the latent space, the first additional data set being an unlabeled data set; generate a first prediction for classifying the first additional data set based on the first instance of the first classifier; generate a second prediction for classifying the first additional data set based on the second instance of the first classifier; generate classification annotations for the first additional data set based on the first and second predictions; train a third classifier based on at least the input data set and the annotated first additional data set, wherein the trained third classifier receives data of a product to be classified and outputs a prediction for the product based on the received data.

[0016] In accordance with one embodiment of the present invention, a classification system for classifying manufactured parts as good or bad includes a data collection circuit for collecting an input data set, and a processing circuit coupled to the data collection circuit and including logic, the processing circuit's logic receiving the collected input data set, generating at least first and second instances of a first classifier, training the first and second instances of the first classifier based on the input data set, and computing a latent space associated with the input data set based on the input data set. train a second classifier to learn a representation of the input data set (a latent space); generate a first additional data set in the latent space, the first additional data set being an unlabeled data set; generate a first prediction for classifying the first additional data set based on the first instance of the first classifier; generate a second prediction for classifying the first additional data set based on the second instance of the first classifier; generate classification annotations for the first additional data set based on the first and second predictions; train a third classifier based on at least the input data set and the annotated first additional data set, wherein the trained third classifier receives data of a product to be classified and outputs a prediction for the product based on the received data. Effect of the Invention

[0017] This allows us to expand the insufficient and imbalanced training data for the two classes to build a classifier sustainably and stably. [Brief description of the drawings]

[0018] [Figure 1] 1 is a system for classifying parts as bad or good according to one embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram of the classifier engine concept of FIG. 1 according to one embodiment of the present invention. [Diagram 3] 3 is a flowchart of the process performed by the classifier engine of FIG. 2 to generate and train student models according to one embodiment of the present invention. [Figure 4]FIG. 3 is a detailed block diagram of the variational autoencoder (VAE) of FIG. 2 in accordance with one embodiment of the present invention. [Diagram 5] 4 is a flowchart detailing a latent space oversampling process according to an embodiment of the present invention. [Figure 6] (a) is a distribution graph showing an example of an input data set that can be used to train a deep learning model, and (b)-(e) are graphs showing examples of multiple classifier instances that generate different decision ranges based on the input data set in (a). [Figure 7] (a) is a distribution graph showing an example of an input data set extended with additional data, and (b) shows a graph of multiple classifier instances generating different decision ranges based on the extended training data set of (a). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The following detailed description, with reference to the accompanying drawings, relates to embodiments of a data expansion system and method, and does not represent all aspects embodied or utilized by the present invention. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the same or equivalent functions and structures embodied in different embodiments are also included within the scope of the present invention. The same reference numerals are used throughout the specification to refer to the same or similar components.

[0020] During the manufacturing process of the mobile display device, digital trace data can be obtained during the production of the mobile display device. Although a mobile display device is given as an example, those skilled in the art should appreciate that the embodiments of the present invention may also be applied to processes for producing other glass and non-glass products, such as wafers and semiconductor glass.

[0021] The trace data can be collected, for example, through one or more sensors located on a conveyor that transports the product through the production process. The sensors record any activity they sense. The sensors can be, for example, multiple temperature and pressure sensors that track temperature and pressure measurements as a function of time through the production process. Each sensor can be sampled multiple times (every few seconds for one glass or wafer being manufactured in the chamber).

[0022] Trace data can be automatically analyzed using a classifier that predicts bad products from "good" products. Therefore, a classifier needs to be trained to assign data samples to one of two classes, namely "good" and "bad". However, the training data used for such classifiers is seriously imbalanced. For example, in a manufacturing process that produces electronic components, the majority of the components are acceptable or "good" and only a small proportion are defective or "bad" in some way. There are 100 to 200 times as many "good" parts as there are bad ones. For this reason, when data is obtained during the manufacturing and inspection process, the majority of the data comes from good products, resulting in an imbalance in the training data. Also, since product models change rapidly, it is generally difficult to expect a sufficient amount of data set to train a deep learning model. The short lifespan of product models makes it difficult to collect sufficient data samples. It is also common to need to generate new AI / ML models to detect the bad conditions of new product models. Therefore, generating a generalized model with a limited data set is a common challenge.

[0023] There are data augmentation techniques that are used to artificially expand the size of the training data set. For example, all possible data sample cases can be generated and used as training data. However, all possible cases are too large to create in a single data set. Also, even if the sample is expanded, not all samples are meaningful for model training.

[0024] Another example is data distillation, a technique often used in visual recognition / image classification. Data distillation is a common omni-supervised learning method that distills knowledge from unlabeled data by generating annotations for the unlabeled data. The annotations are generated using a model trained on a large amount of classified data. The model is then retrained using the extra generated annotations. The extra unlabeled data can be generated through geometric / spatial transformations (e.g., flip, shift, and / or rotation) of the classified data. Additional information on data distillation can be found in Ilija Radosavovic et al., “Data Distillation: Towards Omni-Supervised Learning,” Computer Vision and Pattern Recognition (2018), available at https: / / arxiv.org / abs / 1712.04440, which is incorporated herein by reference.

[0025] However, geometric transformations, which are primarily used to augment data for video classification, cannot be applied to trace data because it is not known whether the characteristics of the input data will remain after applying the transformation. Also, there is generally a correlation between manufacturing issues and defect pattern generation for defect samples, so there is no guarantee that applying known data augmentation techniques will improve the quality of the classifier. Therefore, existing data augmentation techniques are inappropriate for augmenting defect or trace data for training purposes. Data distillation suggests using unclassified data sets from internet-level data sets, but it is not possible to find similar data samples on the internet for a specific company's manufacturing data set.

[0026] In response to the deficiencies of the prior art, what is anticipated is an omni-supervised learning system and method for generating generalized ML models that address the problems of data insufficiency and class imbalance in classification data sets. In accordance with embodiments of the present invention, we address input data set insufficiency and class imbalance through data augmentation via a deep generative model. In particular, we perform data augmentation in manifold (latent) space instead of input data space.

[0027] In addition, one of the main problems when generating a generalized M / L model is an insufficient data set, and this problem is solved by incorporating more variance into the data set using deep generative model data generation. According to one embodiment of the present invention, the augmented / supplemental data is an unclassified data set generated in the latent space using random input. According to one embodiment of the present invention, the augmented data consists of synthetic data samples obtained from latent space oversampling.

[0028] According to an embodiment of the present invention, knowledge is distilled from multiple instances of a baseline machine learning model to annotate an unlabeled data set to generate a new annotated dataset. Knowledge from the multiple instances can be distilled by feeding the unlabeled data set to each of multiple instances of a trained ML model. An ensemble mechanism can then be used to classify the unlabeled data based on the output of each of the ML model instances. In this way, knowledge distillation can be achieved from less data intense models (e.g., individual ML model instances), and the ML model can be generalized to have improved classification accuracy over conventional models. Once trained, the generalized ML model can be used to predict defective parts that occur during manufacturing.

[0029] 1 illustrates a system for classifying parts as bad or good according to one embodiment of the present invention. According to one embodiment of the present invention, the system includes one or more data collection circuits 105, data preprocessing circuits 110, and a classifier engine 115. The data collection circuits 105 can include, for example, temperature and pressure sensors, amplifiers, and / or analog-to-digital converters, the data preprocessing circuits 110 can reformat the data, as described in more detail below, and the classifier engine 115 can generate a deep learning (DL) neural network.

[0030] The data pre-processing circuit 110 receives the raw trace data (e.g., the number of time traces discussed above) from the data collection circuit 105 and can convert the received data into, for example, a two-dimensional array (e.g., a 224x224 array). According to one embodiment of the invention, the data pre-processing circuit 110 includes one or more data storage devices for storing the raw trace data. The size of the two-dimensional array can be selected to be on the order of the size of the images that a neural network typically classifies. Reformatting allows certain portions of the code that runs the image neural network classifier to be reused, which can be used in some embodiments of the invention.

[0031] The format-converted input data may be provided to the classifier engine 115 and used for training or predicting through the DL neural network. In this regard, the classifier engine 115 may comprise logic or instructions stored on one or more tangible, machine-readable media and executed by one or more processing devices to perform prediction generation, prediction training, and prediction through the neural network.

[0032] 2 is a conceptual block diagram of the classifier engine 115 according to one embodiment of the present invention. During training, a classification input data set 200 is received from the pre-processing circuit 110. The classification input data set 200 includes a first number of data samples (e.g., several thousand data elements) that are classified as "good" (or "G") and a second number of data samples (e.g., between 10 and 100 data elements) that are classified as "bad" (or "NG").

[0033] The classification input data set 200 is used to train a classifier, which may be a deep learning neural network, such as a variational autoencoder (VAE) 202. Although a VAE is used as an example, those skilled in the art will recognize that other deep generative models, such as generative adversarial networks (GANs), autoregression models, etc., may be used instead of a VAE. In an embodiment using a VAE, the VAE 202 learns a latent / hidden / manifold space representation for the input data set 200. The latent space typically consists of latent vectors that are simpler / compressed (e.g., lower dimensional) representations of the input data set 200.

[0034] According to one embodiment of the present invention, the trained VAE 202 is implemented to statistically generate additional training data (also referred to as synthetic data samples). In this regard, the VAE 202 may be coupled with an oversampling module 204 and an arbitrary generation module 206. The oversampling module 204 is implemented to oversample the latent space on which the VAE 202 learns, to generate additional "bad" samples. The arbitrary generation module 206 is implemented to generate additional samples in the latent space using arbitrary inputs. According to one embodiment of the present invention, the randomly generated data samples are unlabeled data samples.

[0035] According to one embodiment of the invention, the classifier engine 115 also generates multiple (e.g., two or more) instances 208a-208c of the binary criteria classifier (collectively denoted as instances 208). Each classifier / model instance 208 can be, for example, a neural network.

[0036] The randomly generated data samples from the random generation module 206 are provided to the multiple classifier instances 208 for knowledge distillation. In this regard, an ensemble of predictions from the classifier instances 208 for the randomly generated data samples is used to train a student classifier / model 210. That is, the collection of predictions by the multiple classifier instances 208 can be used to generate new knowledge and distill the generated new knowledge to train the student classifier 210.

[0037] According to an embodiment of the present invention, in addition to the distilled knowledge from the classifier instance 208, additional samples from the original classification data set 200 and / or the oversampling module 204 can be used to train the student classifier 210. The trained student classifier 210 can be provided with new trace data from a newly manufactured product to predict whether the product is "good" or "bad". According to an embodiment of the present invention, the trained student classifier 210 can be used by a prediction system (not shown) to make such prediction types for the newly manufactured products. According to an embodiment of the present invention, the prediction system can output a recommendation or message for an action to be taken if the product is predicted to be "bad". Examples of actions include removing the defective product for further inspection, repair, or disposal. Experiments have shown that the trained model 210 consistently performs within a classification accuracy threshold (e.g., 90% or higher accuracy) on new trace data.

[0038] 3 is a flow diagram of the process performed by the classifier engine 115 to generate and train a student model 210 according to one embodiment of the present invention. At block 300, the classifier engine 115 receives the classified input data set 200.

[0039] At block 302, the input data set 200 is used to train (e.g., using supervised learning) two or more instances 208 of a reference classifier. Each classifier instance 208 may be generated by initializing the reference classifier to an arbitrary initial state. For example, the initial weights of the reference classifier may be arbitrarily set at initialization. Training an instance of a single model, as opposed to separate independent models that are sometimes large and complex, may result in more convenient and cost-effective learning (in terms of computational resources).

[0040] The input data set 200 received at block 300 may also be used to train a VAE 202 at block 308. According to one embodiment of the present invention, the VAE 202 is trained to learn a representation of the input data set 200 in a latent space. Once trained, the VAE 202 can be used to generate augmented data at blocks 310, 314. In this regard, at block 310, the classifier engine 115 executes the trained VAE model to generate additional unclassified data samples 312 in the latent space. The additional data samples may be generated using, for example, arbitrary input.

[0041] At blocks 304a-304c, each of the classifier instances 208 receives additional unclassified data samples 312 and generates predictions for the received samples.

[0042] At block 306, the ensemble of prediction results generated by the multiple classifier instances 208 is used to generate annotations for additional unclassified data samples 312 to generate an annotated dataset 316. In this manner, knowledge gained from running each classifier instance 208 can be distilled into a student classifier 210. By considering the ensemble of predictions from the multiple classifier instances 208, errors in the trained student model 210 can be reduced, even though errors may occur when considering each model instance 208 independently. The use of an ensemble mechanism is preferable to ensure the stability of the trained student model 210 over time.

[0043] One of a variety of ensemble methods may be used to generate annotations for the additional unclassified data set 312. Such ensemble methods include, but are not limited to, hard voting, soft voting, and knowledge distillation. A hard voting ensemble-based method performs predictions from multiple classifier instances 208 and classifies the additional data set based on a majority vote. For example, if classifier instance A 208a and classifier instance B 208b predict a "good" class for a particular data sample, and classifier instance C 208c predicts a "bad" class, then the data sample is classified as a "good" class.

[0044] The soft-voting ensemble-based method takes the average of the predicted probabilities of many classifier instances 208 as follows:

number

[0045] For example, if classifier instance A 208a predicts "good" 99% of the time, classifier instance B 208b predicts "good" 49% of the time, and classifier instance C 208c predicts "good" 49% of the time, then the probability that the data sample is "good" is (99+49+49) / 3=65.67%. Therefore, the ensemble prediction for the data sample is "good".

[0046] The knowledge distillation based ensemble method takes the average probability of multiple instances of a model considering not only the majority class (e.g., "good") but also the minority class (e.g., "bad"). In this way, the average probability of both the majority class and the minority class is output as an ensemble judgment for the data. According to one embodiment of the present invention, knowledge distillation can be extended to multiple class models. In such an embodiment, the average probability can be calculated for multiple instances of a model for multiple classes.

[0047] According to one embodiment of the present invention, the classifier engine 115 also generates additional data samples for training the student model 210 in block 314. In this regard, an oversampling algorithm is applied to generate additional data samples for the minority class (e.g., the "bad" class) to create an oversampled data set 318. According to one embodiment of the present invention, oversampling is done from the VAE latent space instead of the original or other latent space.

[0048] In block 320, the classifier engine 115 trains the student classifier 210 using i) the original input dataset 200, ii) the annotated dataset 316, and / or iii) the oversampled dataset 318. The trained student classifier 210 can then be used as a binary classifier to classify newly manufactured products as “good” or “bad”, etc., based on new trace data obtained for the products.

[0049] FIG. 4 is a detailed block diagram of the VAE 202 according to an embodiment of the present invention. In general, the VAE 202 is a neural network including an encoder network (or "encoder" for short) 400 and a decoder network (or "decoder" for short) 402. The encoder 400 can map or encode each received data element, such as the input data 200, into a latent vector 404, under the restriction that the latent vector 404 has a distribution that approximates a unit Gaussian distribution (i.e., the elements of the vector are independent Gaussian distributions, e.g., with a mean and a variance, respectively). Thus, the VAE 202 applies a mean vector 406 and a standard deviation vector 408 to encode each received data element into the latent vector 404. Those skilled in the art should know that the latent vector 404 is a compressed, low-dimensional source representation of the input data 200. The space of all latent vectors 404 can be referred to as a latent space.

[0050] The decoder 402 may perform roughly the inverse operation of the encoder 400, but may map each latent vector 404 generated by the encoder 400 with a (synthetic) data element that approximates the data element that the encoder 400 mapped the latent vector 404 to. Both the encoder 400 and the decoder 402 may be trained on the input data 200.

[0051] Once the encoder 400 and decoder 402 have been trained, block 310 (FIG. 3) can be executed to generate additional data in the latent space. For example, the latent space can be randomly sampled to generate random vectors (with unit Gaussian distribution) that can be provided to the decoder 402, which generates synthetic data elements (e.g., generating the additional data set 312). The additional data set 312 is unclassified because the VAE 202 is trained on data elements from two classes (i.e., "good" data elements and "bad" data elements).

[0052] 5 is a detailed flow chart of the latent space oversampling process of block 314 (FIG. 3) according to an embodiment of the present invention. In block 500, the latent space data set obtained by training the VAE model in block 308 is input to an oversampling module 502. According to an embodiment of the present invention, the oversampling module 502 samples attributes from instances in the minority class (the "bad" class) to generate synthetic samples (e.g., oversampled data set 318). Unlike conventional oversampling, which uses the original input data set to expand the minority class in the original input data set, embodiments of the present invention are able to expand the minority class data in the learned latent space.

[0053] Oversampling can be done using algorithms such as Synthetic Minority Over-sampling Technique (SMOTE) and Adaptive Synthetic oversampling (ADASYN). In general, SMOTE is an oversampling method that generates synthetic samples from minority classes instead of generating duplicates. According to one embodiment of the present invention, the algorithm selects two or more similar instances in the latent space (using a distance measure) and varies the instances by any amount within the difference between adjacent instances, one attribute at a time.

[0054] ADASYN varies these by adding a small arbitrary vector [or "offset"] to each data element, reducing the chance that synthetic samples from a minority class will interfere with other classes (e.g. a majority class such as a "good" class).

[0055] Block 504 outputs an oversampled data set of the latent space.

[0056] Figure 6(a) is a distribution graph of an example of an input data set (similar to input data set 200) that may be used to train a deep learning model. In the example of Figure 6(a), input data set 200 is small and includes data 600 classified as "good" and data 602 classified as "bad."

[0057] 6(b)-(e) are graphs illustrating an example of multiple classifier instances (e.g., classifier instance 208) generating different decision ranges 604a-604d based on the input data set 200 of FIG. 6(a). According to one embodiment of the present invention, the difference in decision ranges 604a-604d is due to a small amount of input training data. Due to the difference in decision ranges 604a-604d, each model is expected to make a different classification decision on subsequent data.

[0058] 7(a) is a distribution graph of an example input data set (similar to the data set of FIG. 6(a)) that has been augmented with additional training data 700 by a data augmentation mechanism according to an embodiment of the present invention. The additional examples fill gaps in the input data due to the sparseness of the input data set 200. The additional examples are generated through data augmentation as described above using a deep generative model.

[0059] Figure 7(b) shows a graph of multiple classifier instances (e.g., classifier instance 208) generating different decision ranges 704a-704c based on the expanded training data set of Figure 7(a). As shown in Figure 7(b), even though we fill the gaps in the input data with additional samples 700, each of the multiple classifier instances still tends to generate different decision ranges 704a-704c. However, by applying knowledge distillation from the multiple classifier instances as indicated by arrows 706a-706c, we can define new student classifier decision ranges 704d that are more stable and accurate for each of the individual model instances.

[0060] Experimental results show that by applying knowledge distillation from model instances, a generalized ML model can be generated consistently and stably for the entire test data set without knowledge. For example, the effective accuracy of some individual model instances for "bad" data is as low as 80%. However, the generalized ML model generated by the data augmentation and knowledge distillation mechanism according to an embodiment of the present invention shows an effective accuracy of over 90% for "bad" data.

[0061] According to an embodiment of the present invention, one or more of the data pre-processing circuitry 110, the classifier engine 115, the various modules and models / classifiers described above may be implemented by one or more processing circuits. The "processing circuitry" may be embodied using hardware, firmware, software, or a combination thereof. The processing circuitry may include, for example, an application-specific integrated circuit (ASIC), a general-purpose or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a programmable logic device such as an FPGA, etc. Each function in the processing circuitry may be performed by hardwired hardware that performs that function, or by general-purpose hardware such as a CPU that executes instructions stored in a non-transitory storage medium. The processing circuitry may be fabricated on a single printed circuit board (PCB) or distributed across multiple PCBs that are interconnected. The processing circuitry may include other processing circuits, such as an FPGA and a CPU that are interconnected on a PCB.

[0062] Terms such as "first," "second," and "third" are used to refer to various elements, components, regions, layers, sections, etc., but are not limited by such modifiers. Such terms are used to distinguish one element, component, region, layer, section from another element, component, region, layer, section, without departing from the spirit and scope of the present invention.

[0063] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the present invention. In this specification, the terms "substantially," "about," "generally," and similar terms are merely used to indicate approximations and do not indicate "degrees," but are used to indicate the inherent error of measurements or calculations that can be known by those skilled in the art.

[0064] Unless a number is specifically stated herein, all cases of singular or plural are included. The expression "comprising" a feature, step, operation, part, component, etc. means that other features, steps, operations, parts, components, etc. may be included in addition to the part in question. The term "and / or" includes all combinations of one or more of the listed items. The expression "at least one" and the like before a list of items modifies the list as a whole, and does not modify each item in the list. In addition, the expression "may" used in describing an embodiment of the present invention means that it is applicable to "one or more embodiments of the present invention." The term "exemplary" refers to an example or drawing. The terms "use," "utilization," and the like may be used in a similar sense with other similar expressions.

[0065] When a part, layer, region, component, etc. is described as being "on" or "connected" to another part, layer, region, component, it includes not only the case where it is "directly" on or "directly" connected, but also the case where there is another part, layer, region, component, etc. in between. However, when it is described as being "directly on" or "directly connected", it means that there is no other part in between.

[0066] Numerical ranges recited herein include all sub-ranges of the same precision contained within the range. For example, a range "1.0 to 10.0" includes a minimum value of 1.0 and a maximum value of 10.0, and all sub-ranges therebetween, i.e., sub-ranges having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as 2.4 to 7.6. Maximum values ​​recited herein include all numerical limits lower than them, and minimum values ​​recited herein include all numerical limits higher than them.

[0067] Although the above describes and illustrates embodiments of a system and method for knowledge distillation of a model instance, those skilled in the art may change and modify such embodiments. Accordingly, other knowledge distillation systems and methods constructed according to the principles presented herein are also encompassed by the present invention. The present invention is defined by the following claims and their equivalents. [Explanation of symbols]

[0068] 105: Data acquisition circuit 110: Pre-processing circuit 115: Classifier engine 200: Input data (set) 202:VAE(variational autoencoder) 204: Oversampling module 208: Instance 210: Classifiers / Models 312: Additional Data Sets / Unclassified Data Samples 316: Annotated Data Sets 318: Oversampling Data Set 400: Encoder 402: Decoder 404: Latent Vector 406: Mean Vector 408: Standard deviation vector 502: Oversampling module 600: Good data 602: Bad Data 700: Additional samples / additional training data 604a-604d, 704a-704d: Judgment range

Claims

1. A classification method for classifying products manufactured through an industrial manufacturing process into good or bad products, comprising: receiving, by a processor, a set of input data from at least one sensor detecting temperature or pressure measurements as a function of time during the manufacturing process; generating, by the processing device, at least a first instance and a second instance of a first classifier; training, by the processing device, the first and second instances of the first classifier based on the input data set; training, by the processing unit, a second classifier based on the input data set to learn a representation of a latent space associated with the input data set; generating, by the processing device, a first additional data set in the latent space, the first additional data set being an unclassified data set; generating, by the processing unit, a first prediction for classifying the first additional data set based on the first instance of the first classifier; generating, by the processing unit, a second prediction for classifying the first additional data set based on the second instance of the first classifier; generating, by the processing device, classification annotations for the first additional data set based on the first prediction and the second prediction; and training, by said processing device, a third classifier based on at least said input data set and said first additional data set annotated with classifications. Including, The third classifier, which has already been trained, receives data of a product to be classified and outputs a prediction for the product based on the received data. Classification method.

2. The method of claim 1 , wherein each of the first classifier, the second classifier, and the third classifier is a neural network.

3. The method of claim 2 , wherein the second classifier is a variational autoencoder (VAE).

4. The method of claim 1 , wherein the latent space provides a condensed representation of the input data set.

5. The method of claim 1 , wherein generating the first additional data set comprises generating arbitrary data elements in the latent space.

6. The step of generating classification annotations includes: determining a majority class based on the first prediction and the second prediction; and classifying said first additional data set based on said multiple classes Includes The classification method according to claim 1 .

7. The step of generating classification annotations includes: determining a first probability of the first prediction and a second probability of the second prediction; calculating an average of the first probability and the second probability; and defining classes for the first additional data set based on the calculated averages. Includes The classification method according to claim 1 .

8. The step of generating classification annotations includes: determining a first probability of a majority class and a second probability of a minority class based on the first prediction; determining a third probability of the majority class and a fourth probability of the minority class based on the second prediction; calculating a first average of the first probability and the third probability; calculating a second average of the second probability and the fourth probability; and sorting said first additional data set based on said calculated first and second averages. Includes The classification method according to claim 1 .

9. generating a second additional data set based on oversampling in the latent space; and training the third classifier based on the input data set, the first additional data set annotated with the classification annotations, and the second additional data set. The method of claim 1 further comprising:

10. A classification system for classifying products produced through an industrial manufacturing process into good and bad products, comprising: A processing device, and Memory Including, the memory having instructions stored therein; The processing device executes the instructions to receiving an input data set from at least one sensor detecting temperature or pressure measurements as a function of time during the manufacturing process; generating at least first and second instances of a first classifier; training the first and second instances of the first classifier based on the input data set; training a second classifier based on the input data set to learn a representation of a latent space associated with the input data set; generating a first additional data set in the latent space, the first additional data set being an unclassified data set; generating a first prediction for classifying the first additional data set based on the first instance of the first classifier; generating a second prediction for classifying the first additional data set based on the second instance of the first classifier; generating classification annotations for the first additional data set based on the first prediction and the second prediction; training a third classifier based on at least the input data set and the first additional data set annotated with the classification; The third classifier, which has already been trained, receives data of a product to be classified and outputs a prediction for the product based on the received data. Classification system.

11. The classification system of claim 10 , wherein each of the first classifier, the second classifier, and the third classifier is a neural network.

12. The classification system of claim 11 , wherein the second classifier is a variational autoencoder (VAE).

13. The classification system of claim 10 , wherein the latent space provides a condensed representation of the input data set.

14. The classification system of claim 10 , wherein the processing unit, when executing the instructions to generate the first additional data set, generates arbitrary data elements in the latent space.

15. The processing device, when executing the instructions to generate the classification annotations, determining a majority class based on the first prediction and the second prediction; Classifying the first additional data set based on the multiple classes. The classification system of claim 10.

16. The processing device, when executing the instructions to generate the classification annotations, determining a first probability of the first prediction and a second probability of the second prediction; calculating an average of the first probability and the second probability; Defining classes for the first additional data set based on the calculated averages. The classification system of claim 10.

17. The processing device, when executing the instructions to generate the classification annotations, determining a first probability of a majority class and a second probability of a minority class based on the first prediction; determining a third probability of the majority class and a fourth probability of the minority class based on the second prediction; calculating a first average of the first probability and the third probability; calculating a second average of the second probability and the fourth probability; sorting said first additional data set based on said calculated first and second averages; The classification system of claim 10.

18. The processing device executes the instructions to generating a second additional data set based on oversampling in the latent space; training the third classifier based on the input data set, the first additional data set annotated with the classification annotations, and the second additional data set; The classification system of claim 10.

19. A classification system for classifying products produced through an industrial manufacturing process into good or bad products, comprising: a data acquisition circuit for acquiring an input data set from at least one sensor that detects temperature or pressure measurements as a function of time during the manufacturing process; and a processing circuit coupled to the data acquisition circuit and including logic; Including, The logic of the processing circuitry comprises: receiving the collected set of input data; generating at least first and second instances of a first classifier; training the first and second instances of the first classifier based on the input data set; training a second classifier based on the input data set to learn a representation of a latent space associated with the input data set; generating a first additional data set in the latent space, the first additional data set being an unclassified data set; generating a first prediction for classifying the first additional data set based on the first instance of the first classifier; generating a second prediction for classifying the first additional data set based on the second instance of the first classifier; generating classification annotations for the first additional data set based on the first prediction and the second prediction; training a third classifier based on at least the input data set and the first additional data set annotated with the classification; The third classifier, which has already been trained, receives data of a product to be classified and outputs a prediction for the product based on the received data. A classification system for classifying manufactured parts as good or bad.

Images