Systems and methods for classification

Abstract

A system and method for classification. In some embodiments, the method includes: forming a first training dataset and a second training dataset from a labeled input dataset; training a first classifier using the first training dataset; training a variational autoencoder using the second training dataset, the variational autoencoder including an encoder and a decoder; generating a third dataset by feeding a pseudo-random vector into the decoder; labeling the third dataset using the first classifier to form a third training dataset; forming a fourth training dataset based on the third dataset; and training a second classifier using the fourth training dataset.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority and the benefit of U.S. Provisional Application No. 62 / 830,131, filed on April 5, 2019, entitled "Systems and Methods for Data Augmentation for Tracking Data Sets", the entire content of which is incorporated herein by reference. Technical Field

[0003] One or more aspects in accordance with embodiments of the present disclosure relate to classifiers, and more particularly to systems and methods for data augmentation used in training classifiers. Background Art

[0004] When training using data with data imbalance for binary classes, or when the amount of training data is relatively small given the input data dimensions, an automatic classifier may exhibit relatively poor performance.

[0005] Therefore, there is a need for improved systems and methods for data augmentation. Summary of the Invention

[0006] According to an embodiment of the present invention, there is provided a method for classification, the method comprising: forming a first training data set and a second training data set from a labeled input data set; training a first classifier using the first training data set; training a variational auto - encoder using the second training data set, the variational auto - encoder including an encoder and a decoder; generating a third data set by feeding a pseudo - random vector into the decoder; using the first classifier to label the third data set to form a third training data set; forming a fourth training data set based on the third data set; and training a second classifier using the fourth training data set.

[0007] In some embodiments, the first training data set is the labeled input data set.

[0008] In some embodiments, the second training data set is the labeled input data set.

[0009] In some embodiments, forming the first training data set includes: oversampling the labeled input data set to produce a first supplementary data set; and combining the labeled input data set and the first supplementary data set to form the first training data set.

[0010] In some embodiments, oversampling the labeled input data set includes using the Synthetic Minority Over - sampling Technique.

[0011] In some embodiments, oversampling the labeled input data set includes using the Adaptive Synthetic Over - sampling Technique.

[0012] In some embodiments, the fourth training dataset is the same as the third training dataset.

[0013] In some embodiments, forming the fourth training dataset includes: combining a first portion of the labeled input dataset and the third training dataset to form the fourth training dataset.

[0014] In some embodiments, forming the fourth training dataset includes: combining a first portion of the labeled input dataset, a first supplementary dataset, and the third training dataset to form the fourth training dataset.

[0015] In some embodiments, the method further includes: validating the second classifier using a second portion of the labeled input dataset that is different from the first portion of the labeled input dataset.

[0016] In some embodiments, forming the second training dataset includes: oversampling the labeled input dataset to produce a first supplementary dataset; and combining the labeled input dataset and the first supplementary dataset to form the second training dataset.

[0017] In some embodiments, the labeled input dataset includes: a majority class data including a first number of data elements and a minority class data including a second number of data elements, the first number exceeding the second number by at least five times.

[0018] In some embodiments, the first number exceeds the second number by at least fifteen times.

[0019] According to an embodiment of the present invention, there is provided a classification system, including: a processing circuit configured to: form a first training dataset and a second training dataset from a labeled input dataset; train a first classifier using the first training dataset; train a variational autoencoder using the second training dataset, the variational autoencoder including an encoder and a decoder; generate a third dataset by feeding a pseudo-random vector into the decoder; label the third dataset using the first classifier to form a third training dataset; form a fourth training dataset based on the third dataset; and train a second classifier using the fourth training dataset.

[0020] In some embodiments, the first training dataset is the labeled input dataset.

[0021] In some embodiments, the second training dataset is the labeled input dataset.

[0022] In some embodiments, the processing circuit is configured to: oversample the labeled input dataset to produce a first supplementary dataset; and combine the labeled input dataset and the first supplementary dataset to form the first training dataset.

[0023] In some embodiments, the processing circuit is configured to oversample the labeled input data set using the synthetic minority over-sampling technique.

[0024] In some embodiments, the processing circuit is configured to oversample the labeled input data set using the adaptive synthetic over-sampling technique.

[0025] According to an embodiment of the present invention, there is provided a system for classifying a manufactured part as a good product or a defective product, the system comprising: a data collection circuit; and a processing circuit configured to: form a first training data set and a second training data set from the labeled input data set; train a first classifier using the first training data set; train a variational autoencoder using the second training data set, the variational autoencoder including an encoder and a decoder; generate a third data set by feeding a pseudo-random vector into the decoder; label the third data set using the first classifier to form a third training data set; form a fourth training data set based on the third data set; and train a second classifier using the fourth training data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims, and drawings, wherein:

[0027] Figure 1 is a block diagram of a system for classification according to an embodiment of the present disclosure;

[0028] Figure 2 is a flowchart of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0029] Figure 3A is a flowchart of a part of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0030] Figure 3B is a flowchart of a part of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0031] Figure 3C is a flowchart of a part of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0032] Figure 3D is a flowchart of a part of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0033] Figure 3E is a flowchart of a part of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0034] Figure 3Fis a flowchart that is part of a method for training and validating a classifier according to an embodiment of the present disclosure;

[0035] Figure 4 is a table of test results according to an embodiment of the present disclosure; and

[0036] Figure 5 is a table of test results according to an embodiment of the present disclosure. Detailed Description

[0037] The following detailed description, presented in conjunction with the accompanying drawings, is intended to describe exemplary embodiments of a system and method for data augmentation provided in accordance with the present disclosure and is not intended to represent the only form in which the present disclosure may be constructed or utilized. The description sets forth the features of the present disclosure in connection with the illustrated embodiments. However, it should be understood that the same or equivalent functions and structures may be accomplished by different embodiments that are also intended to be encompassed within the scope of the present disclosure. As referred to elsewhere herein, like reference numerals are intended to indicate like elements or features.

[0038] A classifier for a binary class can have the task of assigning data samples to one of two classes, and there may be a significant imbalance in the training data used to train such a classifier. For example, in a manufacturing process for making electronic components, it may be the case that the majority of the components are acceptable or "good", while a minority of the components are defective or "not good" in some aspect. For this reason, when data is obtained during the manufacturing and testing processes, most of the data may come from good devices, i.e., there may be an imbalance in the data. This imbalance can be an obstacle when training an automatic classifier to classify components as "good" or "not good".

[0039] In addition, the number of measurements obtained for each component may be large, i.e., the dimension of each data sample (a data element, which is a set of measurements of an item to be classified, such as a manufactured component) may be large. When training an automatic classifier, this can be a further obstacle, especially in view of the dimension of each data element when the number of training data elements for either class is small.

[0040] For example, when manufacturing a mobile display, trace data can be acquired during the manufacturing process of the display panel. The trace data can include, for example, measurements of temperature and pressure during the manufacturing process as a function of time. Multiple temperature sensors and pressure sensors can be used, and each sensor can be sampled multiple times (e.g., three or four times a day over a period of several days). The trace data generated from these measurements can include, for example, approximately 64 traces, each trace having approximately 304 measurements, e.g., totaling over 19,000 measurements, such that each data element has over 19,000 dimensions.

[0041] As described in further detail below, various methods can be used to address some of the above-mentioned obstacles. Referring to Figure 1 , in some examples, a system for detecting a faulty component includes one or more data collection circuits 105 (the data collection circuits 105 can include, for example, temperature sensors and pressure sensors, amplifiers, and analog-to-digital converters), data preprocessing circuits 110 (the data preprocessing circuits 110 can reformat the data, as discussed in further detail below), and a classifier 115 that can be a deep learning (DL) neural network.

[0042] The data preprocessing circuits 110 can receive raw trace data (e.g., multiple traces as mentioned above) from the data collection circuits 105 and can reformat the raw trace data into, for example, a two-dimensional array (e.g., a 224×224 array). The size of the two-dimensional array can be selected to be comparable to the size of an image typically classified by a neural network. Then, the reformatting can make it possible to reuse a specific portion of the code implementing the neural network classifier for images for use in some embodiments.

[0043] Figure 2 A flowchart for training and validating a classifier in some embodiments is shown. A labeled input data set (or “input data set”) 205 is received from the data preprocessing circuits 110. The labeled input data set 205 includes a first quantity of data samples (e.g., several thousand data elements) each labeled as “good” (or “G”) and a second quantity of data samples (e.g., between 10 and 100 data elements) each labeled as “not good” (or “NG”). Oversampling (as discussed in further detail below) can be employed at 210 to generate a first supplementary data set including additional data samples of one or both classes. Then, the labeled input data set 205 and the first supplementary data set can be used to train a first classifier (or “baseline binary classifier”) at 215 using supervised learning with a first training data set (the first training data set can be, can include a combination (or “union”) of both the labeled input data set 205 and the first supplementary data set). In some embodiments, the step of oversampling or the result of oversampling at 210 can be omitted, and only the labeled input data set 205 can be used to train the first classifier.

[0044] Then, at 220, a model generated from the training of a first classifier (e.g., a trained first classifier or a copy of its neural network programmed with the weights generated from the training of the first classifier) can be used to label a third data set to form a third training data set. The machine learning model can be any of a variety of forms including classifiers, regressors, autoencoders, etc. At 225, the third data set can be generated by a data augmentation method using a variational autoencoder, as discussed further in detail below. At 225, the data augmentation method can use a second training data set as input, which can be, for example, the labeled input data set 205, or a combination of the labeled input data set 205 and a first supplementary data set.

[0045] Then, at 230, a second classifier can be trained using a combination of (i) a first portion 235 of the labeled input data set 205 (generated from the labeled input data set 205 by a data set separator 240), (ii) a first supplementary data set, and (iii) one or more portions of the third training data set. Then, at 245, a second portion 250 of the labeled input data set 205 (also generated from the labeled input data set 205 by the data set separator 240) can be used to validate a model generated from the training of the second classifier (e.g., a trained second classifier or a copy of its neural network programmed with the weights generated from the training of the second classifier). The second portion 250 (for validation) can be different from the first portion 235 (for training), e.g., the second portion 250 can be the remaining portion of the labeled input data set 205.

[0046] In the validation step at 245, the performance of the trained second classifier (i.e., the performance of the model generated from the training of the second classifier) can be used to evaluate whether the second classifier is suitable for production, e.g., for determining whether each manufactured part is to be used or discarded (or reworked).

[0047] Figure 3A is an oversampling step (at Figure 2Flowchart at 210 in []. At 210, the labeled input data set 205 is oversampled to form a first supplementary data set (or "oversampled data set") 305. Oversampling can be performed using SMOTE (Synthetic Minority Over-sampling Technique) or ADASYN (Adaptive Synthetic) oversampling, and each of SMOTE and ADASYN oversampling can be used to create a data set of data elements in the minority class (e.g., the "not good" class). ADASYN can offset each data element in the resulting data elements by adding a small random vector (or "offset") to each data element to reduce the likelihood that the data elements of the first supplementary data set 305 may interfere with other classes (e.g., the majority class which may be the "good" class).

[0048] Figure 3B is the flowchart of the training of the first classifier at 215 ( Figure 2 ). At 215, the first classifier is trained using one or both of (i) the labeled input data set 205 and (ii) the first supplementary data set 305, resulting in a first classifier model (or "baseline binary classifier model") 310.

[0049] Figure 3C is the flowchart of data augmentation (at Figure 2 225). A variational autoencoder can include an encoder and a decoder. The encoder can map or "encode" each received data element into a vector or "eigenvector" that satisfies the following constraint: the eigenvector has a distribution approximating a unit Gaussian distribution (i.e., a vector distribution where the elements of the vector are independent Gaussian distributions, e.g., each element has a mean and a variance). The decoder can perform an approximate inverse of the operations of the encoder; the decoder can map each eigenvector produced by the encoder to a (synthetic) data element that approximates the data element that the encoder would map to that eigenvector. The encoder and decoder can be trained together using regularization of the Gaussian distribution, along with a training set representing the data elements (e.g., the second training set mentioned above) and a cost function that measures the difference between the input to the encoder and the output of the decoder. Once the encoder and decoder are trained, pseudo-random eigenvectors (generated to have a unit Gaussian distribution) can be fed into the decoder to generate synthetic data elements (e.g., to generate a third data set 315 (or "VAE data set")). If the variational autoencoder is trained using data elements from two classes (i.e., having both "good" data elements and "not good" data elements), the synthetic data elements can be unlabeled.

[0050] Figure 3D is the flowchart of labeling (at Figure 2Flowchart of forming a third training dataset from 220 in. The third dataset 315 is classified by the first classifier model 310, and each data element of the third dataset 315 is labeled to form the third training dataset (or "labeled VAE dataset") 320. Figure 3E is a flowchart for training a second classifier (at Figure 2 230 in). At 230, the second classifier is trained using one or more (or corresponding parts of one or more) of (i) the first part 235 of the labeled input dataset 205 (generated from the labeled input dataset 205 by the dataset separator 240), (ii) the first supplementary dataset 305, and (iii) the third training dataset 320, resulting in the second classifier model (or "binary classifier model") 325. Figure 3F is a flowchart for classifier validation (at Figure 2 245 in). Each data element of the second part 250 of the labeled input dataset 205 is fed into the trained second classifier (or "second classifier model") 325, and at 330, each resulting classification is compared with the label of the data element. Then, the performance of the second classifier is evaluated based on the degree of agreement between the classification and the label.

[0051] Figure 4 The table of shows the results of tests performed using oversampling and 80% of the original G dataset, with one embodiment using the baseline binary classifier model 310. According to Figure 2 The method illustrated in processes the labeled input dataset 205 including 3936 data elements in the "good" class and 22 data elements in the "not good" class. Oversampling (at Figure 2 210 in) is used to increase the ratio of (i) "not good" data elements to (ii) "good" data elements to 0.1:1 or 1:1. Figure 4 The table (in the first column) shows the classification accuracies of the "good" and "not good" data elements of the training dataset used, and (in the second column) shows the classification accuracies of the "good" and "not good" data elements of the validation dataset.

[0052] Figure 5 The table shows the results of a performance test of the second classifier model 325 in one embodiment. The second classifier is trained using (i) synthetic samples from the VAE (third training dataset 320), including 3238 G samples and 6762 NG samples, (ii) 2000 NG samples generated by oversampling, and (iii) 2000 real G samples randomly selected from the G samples in the input dataset 205.

[0053] It can be seen thatFigure 5 The performance shown in Figure 4 is significantly better than the performance shown in Figure 4 , i.e., the second classifier significantly outperforms the first (baseline) classifier in the tests corresponding to Figure 5 and Figure 5 respectively. In addition, a smaller portion of the G samples of the input dataset 205 achieved Figure 4 ; 80% of the G samples of the input dataset 205 were used to train the classifier in the test that produced Figure 5 , while only 50.8% of the G samples (2000 G samples) of the input dataset 205 were used to train the classifier in the test that produced

[0054] In some embodiments, k-fold cross-validation is used to obtain a more reliable assessment of the accuracy of the classifier 115 constructed according to the methods described herein.

[0055] In some embodiments, each of the first classifier (or “first classifier model” 310) and the second classifier (or “second classifier model”) 325 can be a suitably trained SqueezeNet, ResNet, or VggNet neural network as described herein. A variational autoencoder can be constructed as described in “Auto-Encoding Variational Bayes” by D. Kingma and M. Welling, available at arxiv.org / abs / 1312.6114, the entire content of which is incorporated herein by reference.

[0056] In some embodiments, the data preprocessing circuit 110, the classifier 115, and one or more of the systems that perform the Figure 2 illustrated methods are implemented with one or more processing circuits. The term “processing circuit” is used herein to refer to any combination of hardware, firmware, and software employed to process data or digital signals. Processing circuit hardware can include, for example, application-specific integrated circuits (ASICs), general-purpose or special-purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field-programmable gate arrays (FPGAs). In a processing circuit, as used herein, each function is performed by hardware configured (i.e., hardwired) to perform that function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. The processing circuit can be fabricated on a single printed circuit board (PCB) or distributed across several interconnected PCBs. The processing circuit can contain other processing circuits; for example, the processing circuit can include two processing circuits, an FPGA, and a CPU interconnected on a PCB.

[0057] As used herein, a "portion" of a thing means all or less than all of the thing. Thus, a portion of a data set means a proper subset or the entire data set.

[0058] It will be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Thus, a first element, first component, first region, first layer, or first portion discussed herein may be referred to as a second element, second component, second region, second layer, or second portion without departing from the spirit and scope of the inventive concept.

[0059] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the inventive concept. As used herein, the terms "substantially", "about", and similar terms are used as approximate terms and not as terms of degree, and are intended to account for the inherent deviations in the values of measurements or calculations that would be recognized by one of ordinary skill in the art. As used herein, the term "major ingredient" means an ingredient present in a composition, polymer, or product in an amount greater than that of any other single ingredient in the composition, polymer, or product. In comparison, the term "essential ingredient" means an ingredient that constitutes at least 50% by weight or more of a composition, polymer, or product. As used herein, the term "major portion", when applied to a plurality of items, means at least half of the items.

[0060] As used herein, the singular form "a" is also intended to include the plural form unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of...", when following a list of elements, modify the entire list of elements and not individual elements of the list. Further, when describing embodiments of the inventive concept, the use of "may" refers to "one or more embodiments of the present disclosure". Additionally, the term "exemplary" is intended to mean an example or illustration. As used herein, the term "use" may be considered synonymous with the term "utilize".

[0061] It will be understood that when an element or layer is referred to as being "on", "connected to", "coupled to", or "adjacent to" another element or layer, the element or layer can be directly on, directly connected to, directly coupled to, or directly adjacent to the other element or layer, or there can be one or more intervening elements or intervening layers. In contrast, when an element or layer is referred to as being "directly on", "directly connected to", "directly coupled to", or "directly adjacent to" another element or layer, there are no intervening elements or intervening layers.

[0062] Any numerical range recited herein is intended to include all sub-ranges of the same numerical precision subsumed within the recited range. For example, a range of "1.0 to 10.0" is intended to include all sub-ranges between (and including) the recited minimum value of 1.0 and the recited maximum value of 10.0, that is, having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as, for example, 2.4 to 7.6. Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed within the maximum numerical limitation, and any minimum numerical limitation recited in this specification is intended to include all higher numerical limitations subsumed within the minimum numerical limitation.

[0063] Although exemplary embodiments of systems and methods for data augmentation have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Accordingly, it should be understood that the systems and methods for data augmentation constructed in accordance with the principles of the present disclosure may be embodied in a manner different from that specifically described herein. The invention is also defined in the appended claims and their equivalents.

Claims

1. A method for classifying manufactured parts as good or defective, the method comprising: forming a first training dataset and a second training dataset from a labeled input dataset; training a first classifier using the first training dataset; training a variational autoencoder using the second training dataset, the variational autoencoder including an encoder and a decoder; generating a third dataset by feeding a pseudo-random vector into the decoder; labeling the third dataset using the first classifier to form a third training dataset; forming a fourth training dataset based on the third dataset; and training a second classifier using the fourth training dataset, wherein the first training dataset is the labeled input dataset, or forming the first training dataset includes: oversampling the labeled input dataset to produce a first supplementary dataset; and combining the labeled input dataset and the first supplementary dataset to form the first training dataset; wherein the second training dataset is the labeled input dataset, or forming the second training dataset includes: oversampling the labeled input dataset to produce a first supplementary dataset; and combining the labeled input dataset and the first supplementary dataset to form the second training dataset.

2. The method according to claim 1, wherein The oversampling of the labeled input dataset includes using the Synthetic Minority Over-sampling Technique.

3. The method according to claim 1, wherein, The oversampling of the labeled input dataset includes using the Adaptive Synthetic Over-sampling Technique.

4. The method according to claim 1, wherein, The fourth training dataset is the same as the third training dataset.

5. The method according to claim 1, wherein, Forming the fourth training dataset includes: combining a first portion of the labeled input dataset and the third training dataset to form the fourth training dataset.

6. The method according to claim 1, wherein Forming the fourth training dataset includes: combining a first portion of the labeled input dataset, the first supplementary dataset, and the third training dataset to form the fourth training dataset.

7. The method according to claim 6, further comprising: Validating the second classifier using a second portion of the labeled input dataset that is different from the first portion of the labeled input dataset.

8. The method according to claim 1, wherein The labeled input dataset includes: majority class data including a first number of data elements and minority class data including a second number of data elements, wherein the first number exceeds the second number by at least five times.

9. The method according to claim 8, wherein, The first number exceeds the second number by at least fifteen times.

10. A system for classifying manufactured parts as good or defective, comprising: a processing circuit configured to: form a first training dataset and a second training dataset from a labeled input dataset; train a first classifier using the first training dataset; train a variational autoencoder using the second training dataset, the variational autoencoder including an encoder and a decoder; generate a third dataset by feeding a pseudo-random vector into the decoder; label the third dataset using the first classifier to form a third training dataset; form a fourth training dataset based on the third dataset; and train a second classifier using the fourth training dataset, Wherein, the first training data set is the labeled input data set, or the processing circuit is configured to: oversample the labeled input data set to generate a first supplementary data set; and combine the labeled input data set and the first supplementary data set to form the first training data set; Wherein, the second training data set is the labeled input data set, or the processing circuit is configured to: oversample the labeled input data set to generate a first supplementary data set; and combine the labeled input data set and the first supplementary data set to form the second training data set.

11. The system according to claim 10, wherein, The processing circuit is configured to oversample the labeled input data set using the synthetic minority over-sampling technique.

12. The system according to claim 10, wherein, The processing circuit is configured to oversample the labeled input data set using the adaptive synthetic over-sampling technique.

13. A system for classifying manufactured parts as good or defective, the system comprising: A data collection circuit; And A processing circuit, The processing circuit is configured to: Form a first training data set and a second training data set from the labeled input data set; Train a first classifier using the first training data set; Train a variational autoencoder using the second training data set, the variational autoencoder comprising an encoder and a decoder; Generate a third data set by feeding a pseudo-random vector into the decoder; Label the third data set using the first classifier to form a third training data set; Form a fourth training data set based on the third data set; And Train a second classifier using the fourth training data set, Wherein, the first training data set is the labeled input data set, or the processing circuit is configured to: oversample the labeled input data set to generate a first supplementary data set; and combine the labeled input data set and the first supplementary data set to form the first training data set; Wherein, the second training data set is the labeled input data set, or the processing circuit is configured to: oversample the labeled input data set to generate a first supplementary data set; and combine the labeled input data set and the first supplementary data set to form the second training data set.

Images