Anomaly detection system using multi-layer support vector machines and method thereof

Abstract

The classifier network has at least two different sets of refinement data, where the first two sets of refinement data are sets of numbers representing feature value data received from sensors or manufactured parts. Anomaly detection is performed on the manufactured parts in a first tier using at least two different types of support vector machines, independently for each classifier using a relevant feature selection process. Then, using stored data, including refinement data of at least two different types of data transformations, and through at least two different types of support vector machines in a second tier, a relevant feature selection process is independently performed for each classifier. At least four different composite classifier types are formed using the stored data or coefficients for anomaly detection on the parts. The set of comparisons of the second tier support vector machine outputs determines whether an anomaly exists.

Description

[0001] U.S. government interest statement

[0002] This invention was completed with government support under Contract No. W58RGZ-13-D-0245 granted by the U.S. Army. The U.S. government holds certain rights to this invention. Technical Field

[0003] This invention relates to anomaly analysis apparatus, methods, and non-transitory computer-readable storage media thereof. More specifically, this invention relates to anomaly analysis apparatus, methods, and non-transitory computer-readable storage media related to machine learning and support vector machines. Background Technology

[0004] With the rapid development of science and technology, many manufacturing technologies are now available. Parts manufactured using these new technologies can pass standard quality control checks, but operational abnormalities still occur due to various factors.

[0005] Several techniques are available for detecting anomalies or defects in manufactured parts using machine learning models. For example, some traditional techniques require a user to determine which parameters are more important based on their experience, and then use these parameters to train a machine learning model to detect anomalies. However, different manufactured parts and anomalies are affected by different factors, so the determinations made by a user for one manufactured part are often not applicable to another. Furthermore, some other techniques only analyze certain parts or features of a manufactured part, rather than the entire part, thus the model obtained through training may not be suitable for other manufactured parts. Summary of the Invention

[0006] Therefore, there is a need in the art to provide a technique capable of objectively selecting more important features of manufactured parts, whether these features are unprocessed input data, processed input datasets using domain-appropriate filtering and / or analysis functions, or feature space projections generated by classifier outputs within the system. This technique combines optimization of different types of classifier sets, operates on selected features as different input sets for each member of different classifier sets, and each different feature subset is the result of a joint classifier / feature set optimization process for each classifier, which operates independently for each classifier in the ensemble, for detecting and analyzing anomalies at various steps in the manufacturing, calibration, and / or certification processes.

[0007] In one embodiment, a missing step in the calibration process (during which the calibration image data is acquired as a fundamental component of the data necessary for image sensor compensation) manifests as various types of patterns imprinted on the calibration data. These patterns exhibit a high degree of variation associated with their representation. The combination of different patterns and statistical differences within each pattern type makes it difficult even for subject matter experts to confidently detect the absence of a missing step in the calibration process. Due to the nature of potential gaps in the manufacturing process, only a very small number of anomalous examples are available for classifier training, exacerbating the detection problem and limiting the architectural options for manufacturing part anomaly detection systems.

[0008] An anomaly analysis apparatus in one exemplary embodiment includes a storage unit and a processor electrically connected to the storage unit. The storage unit stores coefficients obtained from training a Support Vector Machine (SVM), which define the SVM for new data. If a nearest-neighbor-based classifier (an alternative to adding diversity) is used, the storage unit stores selected feature values ​​for each training example, or simply select representatives of the training dataset. Furthermore, the storage unit can store multiple part data sets, each including multiple part feature values. The processor is configured to perform dimensionality reduction on each part data set to generate a discriminative projection through a process of linear projection onto an optimized discriminative axis, where the projection is the result of a classification model optimization process that minimizes the training error (classifying the training data into multiple first normal data sets and multiple first anomalous data sets according to a classification optimization technique).

[0009] According to the bootstrap cross-validation process, the processor can also be configured to select a separate subset of training data not included in the randomly selected training dataset as multiple test data sets, randomly sample this separate subset over a predetermined number of iterations, and test the classification model by using a second separate subset of training data as a "holdout" set during the bootstrap cross-validation process, thereby deriving the accuracy. It is worth noting that this can be applied to the entire network to estimate the overall error rate, but doing so across the entire network may be impractical in terms of the required CPU time.

[0010] The present invention also includes a method for manufacturing part anomaly analysis, suitable for electronic computing devices. In one example embodiment, the electronic computing device stores multiple part data, wherein each part data includes multiple manufacturing part feature values. The manufacturing part anomaly analysis method includes: (a) preprocessing the raw part feature values ​​using complementary filters and a set of analysis techniques to extract as much information as possible, which may be used to detect parts that need to be marked as substandard for inclusion in the final system. Preprocessing with complementary filters and analytical techniques does not preclude the use of original part feature values ​​if they are also determined by the classifier to be usable for discrimination; (b) based on a marginal maximization-based linear discriminant optimization technique, each part data is iteratively reduced to an optimized linear discriminant axis by analyzing the manufacturing part feature values ​​contained in the part data, in parallel with an algorithm that ranks features by measuring classification utility, for incremental removal from the active feature set during iterative linear discriminant optimization through each incremental ranking and removal of feature sets; (c) the optimized classifier output set (ensemble) is processed sequentially in a feedforward, layer-by-layer manner as input to subsequent sets of different types of classifiers, using the outputs of the preceding classifiers as internal feature values ​​input to subsequent layers; (d) based on techniques selected from the known and validated range of techniques in the prior art, the outputs of different optimized classifier sets are aggregated at the output layer, thereby utilizing multiple outputs of the classifier set to generate more accurate and robust classification decisions, thereby classifying manufactured parts as acceptable or unacceptable quality.

[0011] The present invention also includes a non-transitory computer-readable storage medium storing a computer program. After the computer program is loaded into an electronic computing device, the electronic computing device executes the code of the computer program, thereby performing the manufacturing part anomaly analysis method described above.

[0012] The anomaly analysis technique (including its apparatus, method, and non-transitory computer-readable storage medium) disclosed herein employs machine learning-related techniques to train a classification model for detecting anomalies in manufactured parts. Typically, the manufactured part anomaly analysis technique provided by this invention analyzes the feature values ​​of the manufactured parts contained in the collected part data using dimensionality reduction techniques, thereby generating a set of optimized linear discriminant axes. The input feature vector data is projected onto these optimized linear discriminant axes, and then an optimized feature space is created for the classification of manufactured parts based on large-margin linear discriminant optimization. In parallel with the large-margin linear discriminant optimization, the feature set used as input to the large-margin classifier is also optimized using a recursive feature selection method, automatically eliminating the least important features that reduce classifier performance, and selecting the optimal feature set for accuracy measurement using a bootstrap cross-validation technique with all available training data. Since the available training data is relatively scarce in cases where defects or skipped steps occur during part manufacturing or calibration, all available training samples are randomly divided into training and test groups. Replacement sampling (bootstrap) is used to select instances of given training data, while unselected training samples are retained as cross-validation test set data. The random selection (bootstrapping) process of training data is repeated multiple times, thereby estimating the classifier performance for each recursively defined subset of features based on bootstrapping-based statistical estimation theory. No separate sample validation set is used due to defect size limitations and / or skipped manufacturing or calibration steps. The bootstrapping-based classifier accuracy estimation process utilizes the limited available training data more effectively than a single pre-selected retention set. The final classifier performance estimate using a single pre-defined retention set (validation set) may be more a function of a specific set of selected samples and does not represent the mean and variance of the classifier performance expected in use in the new part data domain; therefore, the anomaly detection system of this invention does not reduce the available size of the training sample set to define a validation test set in the classifier training system.

[0013] Since the manufacturing part anomaly analysis technique provided by this invention begins by analyzing the manufacturing part feature values ​​contained in all collected part data, it is suitable for various manufacturing part environments. Furthermore, the manufacturing part anomaly analysis technique provided by this invention utilizes a linear discriminant marginal maximization optimization technique to train the classification model, thus minimizing overfitting caused by less important manufacturing part feature values ​​during training based on the large marginal classifier theory. Therefore, the accuracy of classification of manufacturing part anomalies can be improved, resulting in more accurate detection of manufacturing part anomalies.

[0014] To enable those skilled in the art to better understand the features of the claimed invention, detailed technical methods and preferred embodiments of the invention will be described in the following paragraphs in conjunction with the accompanying drawings.

[0015] In one embodiment, exemplary embodiments of the present invention can provide testing for anomalies in manufactured parts, wherein the testing includes: receiving at least one set of feature value data from at least one sensor or the manufactured part, wherein the feature value data includes feature value data from the sensor or the manufactured part; extracting at least two sets of refined data, each set based on different, domain-appropriate, and partially appropriate feature extraction operators, wherein at least the first two sets of refined data are sets of numbers representing meaningful transformations or analyses of the feature value data received from the sensor or the manufactured part; storing the refined dataset as a stored dataset; performing this through at least two first-layer support vector machines of different types, using a feature selection process independently associated with each support vector machine, further forming a pooled feature set including selecting those features for input to each classifier in the first layer; processing the manufactured part using the stored dataset to project the refined data onto an internal feature axis having properties that enhance anomaly detection; and performing anomaly detection on the manufactured part using the outputs of the first-layer support vector machines and the outputs of the pooled feature set, performed through at least two second-layer support vector machines of different types, using a feature selection process independently associated with each support vector machine, determining the presence of anomalies by comparing or combining the results of the second-layer support vector machine outputs.

[0016] In another embodiment, exemplary embodiments of the present invention may provide: a storage unit for storing multiple feature value data in a feature database and / or storing support vector machine coefficients in a separate database, wherein each feature value data includes multiple feature values; a microcomputer including instructions that, when executed by the microcomputer, configure the microcomputer to: receive feature value data of a manufactured part, wherein the feature value data includes raw feature value data from scans, images, tests, status, or outputs of the manufactured part; extract at least two sets of refined data, wherein each set of at least two sets of refined data extracted using different, domain-appropriate, and partially appropriate feature extraction operators is a set of numbers representing a transformation or analysis of feature value data received from scans, images, tests, status, or outputs of the manufactured part; and process the at least two sets of refined data into specific mechanical properties, chemical properties, etc., using at least two different types of first-layer support vector machines, using a feature selection process independently associated with each support vector machine. The system comprises: a feature database, coupled to a microcomputer, configured to store two sets of refined data based on the processing of the two sets of refined data by the first-layer support vector machine, thereby projecting the refined data onto an optimized internal feature axis to enhance anomaly detection, and storing the processed data as stored data; a feature database, coupled to a microcomputer, configured to store two sets of processed refined data as the output of the first-layer support vector machine based on the processing of the two sets of refined data by the first-layer support vector machine, wherein the microcomputer is further configured to perform anomaly detection on the part based on at least one set of first-layer outputs and pooled feature sets of at least two support vector machines, using the stored, processed, refined data, using at least one second layer of at least two different types of support vector machines, in this way connected to the output of the first-layer support vector machine and the pooled feature set, thereby forming a set of at least six different composite type classifiers.

[0017] In another embodiment, exemplary embodiments of the present invention may provide a classifier system and a cloud-based anomaly detection device operatively connected to sensors and a manufactured part, the manufactured part comprising: a microcomputer including instructions that, when executed by the microcomputer, configure the microcomputer to: store multiple feature value data in a feature database and / or store support vector machine coefficients in a separate database, wherein each feature value data includes multiple feature values; and a microcomputer including instructions that, when executed by the microcomputer, configure the microcomputer to: receive feature value data from the manufactured part, wherein the feature value data includes raw feature value data from scans, images, tests, status, or outputs of the manufactured part; extract at least two sets of refined data, wherein each set of at least two sets of refined data extracted using different, domain-appropriate, and partially appropriate feature extraction operators is a set of numbers representing the transformation or analysis of feature value data received from scans, images, tests, status, or outputs of the manufactured part; and use at least two different types of first-layer support vector machines, using independently associated... A feature selection process processes at least two sets of refined data into representations of specific mechanical properties, chemical properties, outputs of manufactured parts, or other similar properties of manufactured parts. Each pair of SVM types processes the outputs of at least two sets of refined data, and also forms a pooled feature set including those features selected by the inputs of each classifier in the first layer. This projects the refined data onto optimized internal feature axes to enhance anomaly detection, and stores the processed data as stored data. A feature database, coupled to a microcomputer, is configured to store the two sets of processed refined data as the outputs of the first-layer support vector machine based on the processing of the two sets of refined data by the first-layer support vector machine. The microcomputer is further configured to perform anomaly detection on parts using at least one set of first-layer outputs and pooled feature sets based on at least two different types of support vector machines, using the stored, processed, refined data, using at least one second layer of at least two different types of support vector machines, and using a feature selection process independently associated with each support vector machine. This is connected to the outputs and pooled feature sets of the first-layer support vector machines in this manner, thereby forming a set of at least six different composite type classifiers.

[0018] In another embodiment, exemplary embodiments of the present invention may provide a system comprising: at least two feature types, generating a transformation that converts raw data into an array or matrix of feature data for each type; at least two different types of trainable classifiers adapted to have real-valued outputs representing statistical confidence in the classification; feature selection logic associated with each trainable classifier adapted to filter a feature dataset from each view, the feature dataset being used as input to each classifier that processes data from that view during system operation; a first layer of at least two different types of classifiers, operating on features output from each feature type, isolated from processing data from other feature types; and a feature pooling function that pools features from each... The selected feature sets from the feature types are combined into a pooled feature set, where feature pooling uses all features selected for the classifier in the first layer; at least two different classifier types operate on the pooled feature set; a second-layer classifier receives input from the outputs of at least two different types of classifiers in the first layer, which are associated with all feature types and adapted to form a composite classifier type, wherein the composite classifier includes a classifier that processes the outputs from another type of classifier or feature set, thereby providing an array of arbitrary combinations of single and composite classifier types in the second layer; and an output function for parallel combination of the outputs from the classifier layer operating on the pooled feature set and the second-layer classifier. This exemplary embodiment or another exemplary embodiment may further provide that one of the at least two different types of classifiers in the first layer operating on the feature outputs is an SVM. This exemplary embodiment or another exemplary embodiment may also provide multiple linear SVMs for recursive feature elimination (RFE) for feature selection, adapted to control the feature selection process. This exemplary embodiment or another exemplary embodiment may also provide an RBF SVM, where multiple linear SVMs are used as sidecars to combine the RBF SVM to control RFE. This exemplary embodiment or another exemplary embodiment may further provide a grid search classifier hyperparameter optimization wrapper around the RFE, adapted to optimize any hyperparameters available for each type of classifier during training. This exemplary embodiment or another exemplary embodiment may further provide a bootstrap cross-validation error rate estimate added to the RFE, wherein the bootstrap cross-validation error rate estimate is obtained through each iteration of the RFE. This exemplary embodiment or another exemplary embodiment may further provide a marginal area measurement for the bootstrap cross-validation error rate estimate, adapted to establish an optimal operating point regarding which feature set should be used for system operation for each classifier across the set of iterations where features are removed. This exemplary embodiment or another exemplary embodiment may further provide a sigmoid function adapted to control the dynamic range of the output for each classifier type and normalize those outputs across classifiers.This exemplary embodiment or another exemplary embodiment may further provide multiple additional layers of multiple classifier types, these additional layers being adapted to form additional composite classifier types to operate as a collection of statistical combinations of the associated adaptive sigmoid function and the outputs across multiple additional layers as the final output of the network. This exemplary embodiment or another exemplary embodiment may further provide an additional layer including a single classifier adapted to serve as an output node of the network, which is trained to provide an optimal decision boundary in the feature space comprised of the outputs of the previous layer classifiers.

[0019] In another embodiment, an exemplary embodiment of the present invention may provide a method comprising: testing for anomalies in a manufactured part, wherein the testing comprises: receiving at least one set of feature value data from at least one sensor or the manufactured part; extracting at least two sets of refined data, wherein the two sets of refined data are digital sets representing the feature value data received from the sensor or the manufactured part; classifying the at least two sets of refined data into representations of characteristics or outputs of the part, and storing the at least two sets of refined data as stored data; performing anomaly detection on the manufactured part using the stored data via at least a first support vector machine, and performing anomaly detection on the manufactured part using the stored data via at least a second support vector machine; and using at least The second layer includes: at least a third support vector machine, which is a support vector machine of the same type as the first support vector machine that receives the output from the first layer; and at least a fourth support vector machine, which is a support vector machine of the same type as the second support vector machine that receives the output from the first layer; and at least a fifth support vector machine, which is a support vector machine of the same type as the first support vector machine that receives a pooled feature set as input; and at least a sixth support vector machine, which is a support vector machine of the same type as the second support vector machine that receives a pooled feature set as input, concatenating them to provide a parallel set of at least six composite classifier types, comparing or combining their outputs to determine the presence of an anomaly. This exemplary embodiment or another exemplary embodiment may further provide that the feature value data represents data from scanning, imaging, testing, or output from a sensor or manufactured part. This exemplary embodiment or another exemplary embodiment may further provide that the method is cloud-based. This exemplary embodiment or another exemplary embodiment may further provide that the support vector machine is a radial basis function (RBF) or linear. This exemplary embodiment or another exemplary embodiment may also provide that, based on the minimum specification of a support vector machine network, performing anomaly detection on a part using stored data includes: constructing an anomaly classifier based on a decision-oriented system; and calculating the presence of an anomaly based on the stored data, wherein the presence of an anomaly includes at least the difference in the occurrence of the anomaly.

[0020] In another embodiment, an exemplary embodiment of the present invention may provide an apparatus comprising: a storage unit for storing at least one of a plurality of feature value data or a plurality of coefficient data in a feature or coefficient database, wherein each feature value data includes a plurality of feature values, and each coefficient value data includes a plurality of coefficient values; and a non-transitory computer-readable storage medium including instructions that, when executed by a processor, configure operations to: receive feature value data of a manufactured part, wherein the feature value data includes raw feature value data from scans, images, tests, or outputs of the manufactured part; extract at least two sets of refined data, wherein the two sets of refined data are numbers representing feature value data received from scans, images, tests, or outputs of the manufactured part; and divide the at least two sets of refined data into... The system includes a class representing specific mechanical properties, chemical properties, output, or other similar characteristics of manufactured parts, and storing the refined data as stored data; and a feature database of SVM coefficients or coefficients associated with any similar classifier coupled to the processor, which is coupled to the processor and configured to classify based on two sets of refined data, storing the two sets of refined data as classified raw feature value data, wherein the processor is further configured to perform anomaly detection of manufactured parts using stored data based on at least ten support vector machines of at least two different types, in an arrangement forming at least two layers of support vector machines of at least two different types and a set of composite classifier types formed by combinations of different types of classifiers between the first and second layers that process at least two sets of refined data. This exemplary embodiment or another exemplary embodiment may be further provided, wherein the support vector machines are RBF or linear. This exemplary embodiment or another exemplary embodiment may also provide that the anomaly detection device further includes a central database coupled to a microcomputer and a feature database, wherein the central database is configured to: parse feature value data to obtain parsed feature value data; classify the parsed raw sensor data into classified parsed raw sensor data as representations of specific mechanical, chemical, or other similar properties of a part, and store the classified raw parsed sensor data after classifying the raw parsed sensor data, wherein the classified raw parsed sensor data corresponds to at least two sets of refined data, wherein extracting at least two sets of refined data from the raw sensor data includes extracting the classified raw parsed sensor data from the raw parsed sensor data. Attached Figure Description

[0021] Exemplary embodiments of the invention are set forth in the following description, illustrated in the accompanying drawings, and particularly explicitly pointed out and set forth in the appended claims. The accompanying drawings, which are wholly incorporated herein by reference and constitute a part of the specification, illustrate various examples, methods, and other exemplary embodiments of various aspects of the invention. It should be understood that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the figures represent one example of a boundary. Those skilled in the art will understand that in some examples, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some examples, an element shown as an inner component of another element may be implemented as an outer component, and vice versa. Furthermore, elements may not be drawn to scale.

[0022] Figure 1 ( Figure 1 This is a schematic diagram of the operation of a system processing and classifier architecture according to one embodiment.

[0023] Figure 2 ( Figure 2 ( ) is a schematic diagram of the operation of a first type of SVM in the first layer of a currently disclosed system according to an embodiment.

[0024] Figure 3 ( Figure 3 This is a schematic diagram of the operation of a second type of SVM in the first layer of a currently disclosed system according to an embodiment.

[0025] Figure 4A ( Figure 4A ( ) is a schematic diagram of the operation of the second SVM layer of a currently disclosed system according to an embodiment.

[0026] Figure 4B ( Figure 4B () is a schematic diagram of another exemplary operation of the second SVM layer.

[0027] Figure 5 ( Figure 5 ( ) is a flowchart of an exemplary process according to an embodiment of the present invention.

[0028] Similar reference numerals throughout the figure refer to similar parts. Detailed Implementation

[0029] As part of the routine production process, some manufactured parts undergo quality assurance or calibration checks. Missing steps in the calibration process (during which calibration image data is acquired as a fundamental component of the data required for image sensor compensation) can manifest as various types of patterns imprinted on the calibration data. These patterns can also exhibit high variability related to their performance. The combination of different patterns and statistical differences within each pattern type can make it difficult even for experts to confidently detect missing steps in the calibration process. Due to the nature of potential gaps in the production process, typically only a very small number of anomalous examples are available for classifier training, exacerbating the detection problem and limiting the choice of classifier network architecture.

[0030] In machine learning, Support Vector Machines (SVMs, also known as Support Vector Networks) are supervised learning models with related learning techniques used to analyze data for classification and regression analysis. Given a set of training examples, each labeled as belonging to one or the other of two categories, SVM training techniques build a model that projects new examples onto a discriminative axis optimized during training, thereby increasing the separability of data from each category using the well-known principle of marginal maximization in statistical learning. The data values ​​projected onto the optimized discriminative axis can be used for binary classification, in conjunction with the corresponding SVM-optimized threshold, or as a measure of membership or membership probability (e.g., using methods such as Platt scaling), using the real-valued results of the projection. An SVM model represents examples as points in space, mapping them onto a new synthetic axis orthogonal to the hyperplane, adjusted during training to optimize the separation (marginal width) of the data in the input space, such that examples of individual categories are separated by the widest possible distinct gap. The gap, often referred to as the margin, implements a hyperplane decision boundary with a non-zero thickness, such that the boundary of the gap is a hyperplane equidistant from the hyperplane defining the decision boundary. Maximizing the width of the margin through an optimization process on the training examples theoretically guarantees optimal performance when generalizing to new data not seen during training. New examples are then mapped onto the optimized discriminative axis and can be used as input to optimize for classification discriminancy, thus serving as subsequent inputs for SVMs or other types of classifiers. The values ​​of the new examples mapped onto the optimized discriminative axis can also be used to classify the data in a binary manner based on the data's polarity, since zeros on the discriminative axis represent the center of the margin in the input feature space.

[0031] In addition to performing linear classification, SVM can also effectively perform nonlinear classification using the so-called "kernel trick," implicitly mapping its input to a high-dimensional feature space. Then, the marginal maximization principle is used to project the data in the nonlinear inner feature space onto the linear discriminant axis. This allows SVM to use nonlinear transformations of the data, increasing the probability of finding an optimal discriminant axis. After nonlinearly transforming the data to a higher-dimensional feature space, linear separation of the data is permitted.

[0032] When data is unlabeled, supervised learning is impossible, necessitating an unsupervised learning method that attempts to find natural clusters of data into groups and then maps new data into these formed groups. Support vector clustering, which uses the statistics of support vectors developed from support vector machines to classify unlabeled data, is one of the most widely used clustering techniques in industrial applications.

[0033] To address the inherent variability of this data and the extremely limited set of examples available for training, a supervised classifier with a shallow, multi-layered architecture was designed to detect multiple, highly variable manifestations of low-level anomalies. This was achieved by using a multi-view feature set processed by a cascaded array of linear and radial basis function (RBF) SVMs trained layer-by-layer with a robust feature selection process for deep embeddings. The use of a large feature database, along with the large marginality of feature selection, SVMs, and the layer-by-layer training paradigm, provides a highly accurate yet compact system for classifying manufactured parts. Compared to the typical requirements for training dataset size using classic multilayer perceptrons (MLPs) and recently developed deep neural networks (CNNs or convolutional neural networks) trained with backpropagation, avoiding backpropagation allows for the development of efficient discriminators with relatively small training datasets. The random initialization of weights, commonly used in standard training of classifiers based on multilayer perceptron architectures, typically requires long training times because optimization via gradient descent is performed simultaneously on all hidden unit coefficients. Therefore, the optimization search space has the dimension of the total number of coefficients in the entire network, compared to Layer-by-Layer Hidden Unit (SVM) optimization, where the optimization search space is simply the dimension of the number of coefficients required for each SVM. For example, if all hidden units (SVMs) use the same number of features as the input (d), and there are n SVMs in the network, the CPU load for optimization will be proportional to n*f(d+1), where f() is the SVM optimization / search algorithm time. In contrast, in the MLP formulation, replacing n SVMs with the sum of input, hidden, and output units (total n), the CPU load for optimization will be proportional to h(n*(d+1)), where h() represents the MLP backpropagation optimization algorithm time, because all coefficients associated with each feature (note the bias term added to the number of features) are optimized simultaneously, rather than individually in a smaller group associated with the optimization of each hidden unit. It should be clear that as the network size increases, the computational burden of optimizing MLP-type networks represented as h() increases dramatically, because the number of possible combinations of coefficient values ​​(where the coefficients are quantized into a single value q) increases to (1 / q). n(d+1) Compared to networks optimized individually for each hidden unit (where the number of possible system combinations only increases to n(1 / q)). (d+1)While no single form of optimization algorithm used for SVM or MLP can obviously evaluate all possible combinations of quantized coefficients, this can still be understood using n copies of a space of size (d+1) or a single space of size n(d+1). Furthermore, it is well known in the literature that the convergence of backpropagation is affected by the fact that the error of the output layer needs to be backpropagated literally to optimize earlier layers; because the output layer cannot be effectively optimized until the first few layers have made a distinguishable and meaningful transformation of the input data, and the training of the randomly initialized earlier layers is driven by the error propagating backward from the randomly initialized later layers, effectively suppressing the backpropagation error signal, a significant amount of training time is simply used for the earlier layers to begin effective training with the propagated backpropagation error signal, and then the later layers are trained more effectively after the earlier layers have effectively developed meaningful feature extraction capabilities. Therefore, it should be clear that feedforward, layer-by-layer training methods are often more efficient in terms of training time than backpropagation techniques used to train the entire network simultaneously, in terms of the size of the coefficient search space, incorporating the error signal to drive the optimization of both architectures. In backpropagation, multiple local minima exist when training a network using gradient descent, along with convex optimization techniques used to train each hidden unit in the SVM network. This is because convex optimization is most commonly used in SVM training to guarantee a unique local minimum for each hidden unit, and consequently, a unique global minimum for the network of hidden units within the SVM. Note that other forms of classifiers, such as the k-nearest neighbor classifier, can also be used in this architecture; its basic form does not require training and therefore also lacks local minima. It can then be seen that the layer-by-layer, hidden-unit-by-hidden-unit training paradigm is efficient in terms of CPU resources, and perhaps more importantly, avoids the problem of getting stuck in suboptimal solutions due to local minima during gradient descent. Current hidden-unit discriminators in MLP or CNN architectures are also not trained with large marginal features as the target. Large-margin discriminators trained using Support Vector Machines (SVMs) theoretically have the best generalization performance by maximizing the separation between the two classes along the discriminant axis (defining the hyperplane decision boundary direction). Therefore, by using SVMs as hidden units, marginal maximization within the hidden units will also contribute to generalization accuracy. This is because a successive sequence of operations improves the separation between classes along the discriminative axis layer by layer, ultimately improving the separation between classes in the classifier's output layer. To prevent the errors introduced by hard decisions using "hard-margin" SVMs, soft-margin implementations allow robust handling of nonlinearly separable training data and linear or nonlinear combinations of parallel SVMs in an integrated manner.

[0034] Therefore, building a manufacturing parts classifier system layer by layer using a feedforward approach to construct an SVM is one way to develop a classifier system composed of hidden units, all exhibiting large / soft margin characteristics. The technique of training the classifier layer by layer for the manufacturing parts classifier system has proven to be an effective classifier training technique, capable of producing competitive results with current state-of-the-art deep neural networks trained using backpropagation. Layer-by-layer training during the incremental construction process of the manufacturing parts classifier allows the complexity of the manufacturing parts classifier to be appropriately matched to the size and / or difficulty of the dataset in the classification problem. Similar to pinpointing the optimal number of training iterations in terms of cross-validation performance, layer-by-layer training allows determining the complexity of the manufacturing parts classifier, albeit at a level quantified by layers (although it can add hidden units to each layer in more complex implementations of this paradigm), which provides optimal cross-validation performance.

[0035] Because SVMs, and by virtue of the theory of large-margin classifiers, are not subject to the "curse of dimensionality," a manufacturing part anomaly detection system built with SVMs as hidden units will possess the same property at the manufacturing part classifier system level. This can be proven by demonstrating this property directly above the output layer 150, and indeed, recursively fed by layers of SVM hidden units that are insensitive to dimensionality. While in practice, each SVM in a manufacturing part classification system deviates from this theoretical property, the relative insensitivity of each SVM to the curse of dimensionality is greater than that of any other type of hidden unit. Therefore, suppressing the sensitivity to the influence of a relatively large number of features on the size of the training dataset is a fundamental property of the manufacturing part classifier system at every level, in a multiplicative manner. Considering that each SVM suppresses the variance sensitivity caused by the curse of dimensionality, each subsequent layer further suppresses this sensitivity, a property referred to in the literature as "asymptotically linear separability."

[0036] One exemplary system design concept in this invention is the use of various feature extraction operators (or feature types, or "views") designed to capture unusual phenomena and exhibit multiple distinct representations. This approach has been termed a "multi-view" architecture. Multi-view learning, or learning with multiple distinct feature sets, is a rapidly evolving direction in machine learning, with a strong theoretical foundation and significant practical success. The most common concept of a multi-view feature set is features extracted from different forms of input, which are the primary perspectives of the "view," such as image data, audio, text, web links, etc. However, it is also recognized that applying different feature extraction operators to the same input data is also an effective application of this paradigm, as in the case of each sensor receiving only one image as input in this invention. It has been shown that, in practice, multi-view learning can often work effectively to generate multiple views from data from a single view. Current explorations of view-building techniques found in the literature include random segmentation, random subspace methods, feature clustering, pseudo-multi-view co-training, feature subset selection based on genetic algorithms, and using multi-kernel-based similarity metrics to influence different views through variations in the spatial relationships between samples. The present invention employs a method that uses the same raw image data from the sensor as input, defining each view as a set of feature vectors output from appropriate feature extraction operators for each domain.

[0037] Figure 1 The preprocessing and classifier architecture of the system is described. The raw data for system processing is input into the system in box 100. The raw data for processing can be processed by automatically or manually selected feature extraction operators, transforms, or sets of statistical measures, which select what is expected to be anomalies in the mechanical, chemical, or other similar properties of manufactured parts. The data is first processed by a function that provides segmentation and extraction of the data of interest, 105, and then by geometric transformations and geometric dimensionality reduction techniques, 110. A set of statistical analyses, 115, uses the extracted data of interest to provide a statistical feature array for further processing, while a manually defined set of feature extraction operators, filters, or transforms, including other parallel feature extraction operators, each outputs a feature array for data processing, such as... Figure 1 As shown in 116, 117, and 118, but including other parallel feature extraction operators in the current system. All these feature sets will operate independently in a "wrapper" arrangement through feature selection algorithms or processes, using SVM to evaluate, compare, and select the best-generated feature set. Finally, the optimized feature set is used to better discriminate anomaly representations. For a specific application, the selected features can be any features determined to be necessary, whether independently or in conjunction with other selected features. Each SVM always uses feature selection.

[0038] The feature arrays 115, 116, 117, and 118 used in the anomaly detection system consist of a selection of basic feature extraction operators designed to generate features capable of classifying data using functions that meaningfully match the data. According to one embodiment, the most basic feature set comprises a set of statistical features that operate on the segmented raw image data. A boundary segmentation function defines the region of interest (ROI) for the statistical set of the raw image data. A nonlinear geometric transformation, starting from the circular field of view defined by boundary detection, provides a rectangular array defined by radial and roll angular axes for further processing of feature extraction. The remaining feature types are functions that operate on this rectangular array. These include a set of multi-scale directional 2D Gabor filters, including 1D (on the marginal projection) and 2D FFT (on the rectangular array), wavelet scale maps operating along both radial and roll marginal projection axes, and a set of geometric features operating on a single level of contour analysis. The selection of the individual feature extraction operators is not unique and has been (and should be) selected in an application-specific manner. The fundamental requirement of a set of feature extraction operators in the multi-view paradigm is a set of complementary functions that provide different views of the data, so that each feature type provides some information gain relative to the whole set, with the goal of optimizing multiple performances for detecting potential problems.

[0039] Then, at least two different, independent Support Vector Machines (SVMs), also known as SVM techniques, are trained on features 115, 116, 117, and 118. Each SVM independently searches for anomalies. SVMs 120 and 125 can be of the same kernel type, trained with different parameters or independently sampled training data examples, or they can be of different kernel types for each feature 115, 116, 117, and 118. The SVM can be a linear, radial basis function (RBF), multinomial, or any other type of nonlinear kernel SVM, combined with a specified optimization algorithm (one of a set of hyperparameter options for defining the training mechanism for each SVM) to be compatible with the anomaly detection system. Note that the same type of SVM, such as an RBF SVM, can be trained using different optimization algorithms or different hyperparameters (e.g., setting the cost parameter or choosing a specific loss function). SVMs using the same kernel type but trained with different hyperparameters become different SVM types in practice, effectively extending the library of SVM types that can be applied in this architecture. At a minimum, different SVMs can be generated using the same set of hyperparameters, with training example sets generated using a bootstrapping method that randomizes the bootstrapping sampling with replacements. Since each set of random training examples is unique, an SVM trained using bootstrapping (although using the same hyperparameters) will still have different properties than an SVM trained in the same way.

[0040] Each SVM 120, 125 in the first layer 126 can be viewed simultaneously as a separate classifier used in the stacked ensemble architecture, or as an optimized projection on a single axis, thereby passing optimized, refined discriminative information to the next layer. Therefore, the first layer 126 is both a collection of classifiers and a hidden layer (Figure 4), projecting feature-selected data into a new feature space whose axis is the output of each SVM in the first layer.

[0041] For each different type of SVM, the system can be trained on a single feature set selected using wrapper techniques combined with bootstrap cross-validation training for each uniquely defined feature set during the iterative process of feature subset selection, thereby optimizing generalization performance. During the "recursive feature selection" process, the feature with the lowest estimated utility is removed, the SVM is retrained, and the features of the reduced-size feature set are rearranged using an estimate of the accuracy obtained from the bootstrap cross-validation process. This process is repeated until only one feature remains. In this system, the ability to prune the feature set to a single feature if it is indeed optimal at a particular node in the network demonstrates the highest level of flexibility in adapting to the network architecture regarding the problem domain and available data. The cross-validation accuracy of each unique feature set is estimated during the recursive feature elimination process, and for a particular SVM, the feature set selected, for a particular feature extraction operator type, is the one with the highest estimated cross-validation accuracy, obtained by averaging the bootstrap resampling of a predetermined number of datasets used for training and cross-validation.

[0042] The outputs 120 and 125 of the first layer 126 of the Support Vector Machine (SVM) are used as inputs to the second layer 128 of SVMs 130, 135, 140, and 145. Similar to the first layer 126 of the SVM, the composition of the second layer 128 SVM can be modified based on the system designer's judgment during the development of a classifier for a specific manufactured part or sensor. The number of views, feature extraction operator type, SVM kernel type, and hyperparameter options are predefined architectural features of the overall classifier and are not learned automatically. However, they can be modified by the system designer as experience is gained in specific manufacturing applications to optimize overall system performance.

[0043] Providing a different set of features for the input defines the input form of the classifier, and choosing a classifier architecture designed using front-end feature extraction is also a key factor in achieving effective discriminative performance. In one embodiment, the first layer 126 of the classifier includes two types of SVMs: linear 120 and radial basis functions (RBF) 125, each of which is trained to optimize classification performance on relevant views in the multi-view architecture. Each SVM hidden or output unit is further optimized along with SVM optimization techniques using an embedded feature selection technique that operates in parallel with hyperparameter grid search. In two different implementations of this classifier developed for two different types of image data, six or eight different feature types are used to generate the feature sets input to the SVM.

[0044] Since the number of informational features in each feature set can (and does) vary greatly when used alone and with other feature sets, the architecture aims to normalize the weights of the feature sets to some extent using three complementary approaches.

[0045] The first approach is a feature selection technique that heavily utilizes bootstrap cross-validation and incorporates a unique metric to indicate the effective margin of each point in the feature selection process. Feature selection not only effectively optimizes classifier performance in practice, but in our application, it also helps balance the number of features in each view's output, thereby reducing the difference in the "weights" of each feature set, which are proportional in a weighted sense to the number of (normalized) features within each set. Feature selection is combined with hyperparameter selection for each SVM in the wrapper configuration, using a "sidecar" linear SVM for feature ranking in Recursive Feature Elimination (RFE). RFE is a technique that uses the magnitude of the SVM feature coefficients to indicate the importance of each feature after SVM training. After each iteration of SVM training on different feature sets, the entire feature set is ranked by magnitude, and the feature with the lowest absolute magnitude value is removed before the next iteration. The result of each iteration is not only a ranked list of features but also a bootstrap cross-validation estimate of the error rate, thus generating a curve of error rate versus feature set size (and composition). Then, for each SVM in the manufacturing part anomaly detection system, a feature set and associated hyperparameters are selected based on the minimum error during the RFE process. The trained SVM associated with the feature set linked to the minimum error point is then saved as an SVM, which will be used as a hidden unit or node in the manufacturing part classifier to process new sensor data during classifier deployment.

[0046] A second approach to cross-view weight balancing utilizes each of the 126 SVMs in the first layer as a means of projecting feature data from each selected feature set of each view onto the corresponding set of discriminative axes. In one exemplary implementation, each view has two discriminative axis outputs, one from each of the two types of SVMs (linear and RBF) associated with each view. Regardless of size, refining the dimensions of each view onto a pair of projections onto the discriminative axes makes the outputs of each view exactly equal in terms of the number of features passed to the second layer 128. But the main result of this process is that it provides a highly discriminative feature space relative to each “view feature space” as input to the SVMs of the second layer 128.

[0047] In a third approach to view weight balancing, a selected subset of a finite number of selected features from each view is submitted to a pooling feature set 170. The feature selection process associated with each linear SVM 165 then doubles as a method of providing a controlled number of features from each type to the pooling feature set for balancing the feature type or view. In one embodiment, these feature sets are directly fed into SVMs 160 (linear) and 165 (RBF) in the second layer 128 via “skip connections” (deep learning terminology). In one embodiment, only linear SVMs are used to submit the selected feature subset to the pooling feature set. This does not preclude submitting a finite number of features to the pooling feature set from RBF SVMs or any other SVM type that might be used. Beyond simply being another approach to weight balancing, the generation of the pooling feature set allows the SVMs in the second layer 128 to leverage the co-correlation between features across different feature sets, thereby providing additional discriminative information at the second layer 128.

[0048] The second layer of the SVM (128) then presents two complementary feature spaces. The first is a set of optimized projections within each feature type or view, namely the linear and RBF SVM projections for each view. The second is the feature space defined by pooling the most discriminative subset of features selected from each feature type. Therefore, layers 130, 145, 140, and 145 of the second-layer 128 SVM use projection methods to view a refined version of the total input feature space, while layers 160 and 165 of the SVM use pooling techniques to view the refined version.

[0049] The SVM in the second layer 128, which receives view projection, is also of two types (linear and RBF), trained in the same way, using a wrapper method for feature and hyperparameter selection, and a “sidecar” linear SVM for recursive feature elimination. The main difference in the second layer 128 is that it now obtains two different types of outputs from two types of feature space refinement processes (pooling and projection).

[0050] These two basic types of SVMs were chosen because they have complementary properties. Linear SVMs are optimized to reveal linear separations between classes in the feature space. Clearly, not all (or even most) problems are suited to such a simple solution as a single linear discriminant unless the input feature space is very rich with available transformations. In such cases, SVM training also effectively achieves a weighted form of feature selection by performing a large marginal objective. RBF SVMs act more like weighted nearest neighbor classifiers, discriminating data from each class even with rich feature sets by more flexibly adjusting the default decision boundary, which remains in a more clustered permutation than is typical of standard nearest neighbor classifiers. While for nontrivial discrimination problems, any given single SVM at hidden layer 126 is unlikely to provide perfect discrimination, as evaluated by cross-validation, SVMs taking the hidden layer output as input are more likely to have the ability to utilize complementary information in the hidden layer 126 feature space and provide a higher level of accuracy at the second layer 128. From the following discussion of different sets of classifier types defined by the SVM type sequence, layer by layer, it becomes clear that this architecture can be scaled to any number of layers to achieve the desired performance level, given an appropriate level of manufacturing part classifier complexity relative to the size of the training dataset.

[0051] In one embodiment, a manufacturing parts classifier can be viewed as a collection of different classifiers within a limited range of diversity. It is generally accepted that an effective ensemble learning system should consist of individuals that are not only accurate but also diverse; that is, a good balance should be maintained between diversity and individual performance. In this embodiment, six classifiers are effectively synthesized in the second layer 128, constituting the entire classifier system: “Pooling->Linear SVM”, “Pooling->RBFSVM”, “Linear SVM->Linear SVM”, “Linear SVM->RBF SVM”, “RBF SVM->Linear SVM”, and “RBF SVM->RBFSVM”. The output layer 150 is the collection of all six types of sub-classifiers in the second layer 128 (although the two SVMs that take input from the pooling feature set can be viewed as second-layer nodes with skip connections from the original feature set bypassing the first layer 126, or in the first layer, skip connections to the output layer 150). The output of this manufacturing parts classifier system is currently expressed as the average or minimum of the sub-classifier outputs, where the sub-classifiers are the six second-layer 128 SVM types. It is possible to use a third layer in a stacked structure to train weights for a more optimized combination of sub-classifiers. If a trainable classifier is added to the third layer as a single output unit, the network transforms from a set of classifiers into a more traditional network where the output unit (classifier) ​​generates a single classification confidence metric. While the additional layer increases the network's capacity and may compromise generalization accuracy, it can also optimize the relative weights of the second-layer classifier set, thus improving accuracy. Therefore, the addition of this optional output unit must be evaluated against cross-validation accuracy to determine its incremental impact on performance. Note that for a single output unit, due to the ability to select features, it is possible to choose only one second-layer classifier as the active input to the output node. In the same way, the feature selection capability extends backward to the first layer, so theoretically, after training, the entire network can select only one feature across all views and choose one classifier type across the entire network. An extreme example might lead to feature #12 from view #2 being used as input to a composite classifier of the form "linear SVM->RBF SVM->linear SVM" (if the output node is a linear SVM). This example serves to illustrate the extreme flexibility of the resulting network architecture after training—any and all feature sets from a single view and pooled feature sets, as well as any combination of the first and second layer classifiers, can be selected as "active" as a result of feature selection during training, since all inputs to each layer are treated as features. Also note that, as will be discussed in further detail in this paper, additional classifier layers can be added incrementally as needed.For example, the third layer, instead of consisting of a single classifier as an output node, can be specified as another layer of different classifiers that take their inputs from the second layer, thus forming a set of classifiers before the statistical combination of their outputs, or as the input to a fourth layer of single classifiers as output nodes, or as the input to another layer of different classifiers.

[0052] One embodiment of the system can be used to screen calibration data. A second embodiment of the system can be used with calibration image data in a secondary form and can be modified for any specific case to meet the requirement of discriminating between any two categories of real-valued data of any form obtained from the manufacturing process, whether in image format or other matrix format, and can be meaningfully processed like an image.

[0053] When the hidden unit is a nonlinear SVM or other nonlinear classifier, a linear SVM is used as a "sidecar" as an efficient method for ranking features for elimination in recursive feature elimination techniques. Wrapper methods for feature selection use the classifier's performance to evaluate the utility of each unique feature set. Recursive feature elimination is an ideal technique because it allows for consideration of all interactions between features before removing features from the set. This contrasts with forward feature selection search, where features are incrementally added to the set, a process that never fully utilizes the potential synergies of a set of features (i.e., features considered independently or in small subsets that may not show promise but could actually be useful in a larger set of features).

[0054] Figure 2 An exemplary linear SVM used in the system is depicted. Raw data 200 is fed to a set of independent feature extraction operators 205, which transform the raw data into independent views of the raw data based on different, domain-appropriate analyses, filters, and transformations. The feature outputs from each view are then fed to a set of linear SVMs 210, 215, and 220. The outputs of these SVMs are collected and sent to the next SVM layer 230. In one embodiment, the linear SVMs 210, 215, and 220 also provide the pooled dataset 240 with a high-ranking subset of the selected feature data, which is then independently sent to a second SVM lawyer 245. Note that other embodiments may use a subset of features selected from other types of classifiers (e.g., RBF SVMs) at this layer, as the feature selection ranking in the generalized wrapper formula depends on the classifier used.

[0055] The standard recursive feature elimination process starts with a full set of features and removes the features estimated to contribute the least to classification performance in each iteration. The absolute magnitude of the feature coefficients in the linear discriminant is used as a proxy for the application of the features in the discrimination. Since the size of the linear SVM coefficients after training can be interpreted as a feature importance factor, this technique can be specifically applied to linear SVMs; it is only suitable for linear SVMs. Assuming that some reasonable type of normalization is performed on each individual feature, the coefficients of the linear discriminant (after training using the SVM technique) are an indicator of the weight associated with each feature in the decision process. The features with the lowest weights after training may be noise or lack useful discriminative information for the current problem (they may vary, but are not related to the classes in the training data). The features with the lowest coefficient magnitudes then become the primary candidates for elimination in each step.

[0056] For other types of classifiers, such as RBF SVM or nearest neighbor-based (k-NN) classifiers, there is no comparable metric applicable to this process—the result of training with a given feature set. In the case of non-linear classifiers, such as RBF SVM or k-NN classifiers, the effect of removing each feature from the current set can be directly measured, but this requires retraining the classifier for the number of features in the current set, which in typical applications can add several orders of magnitude to the CPU time requirement (meaning a typical feature set size). For example, with a feature set size of 4000, near some of the feature sets we use in this application, instead of the 3999 iterations required for a full evaluation using RFE, evaluating the removal of each feature at each point in each iteration would require 8,002,000 iterations (before removing the feature that contributes the least to reducing the error rate or the most to it). This would be a practically untenable computational level for a single SVM, let alone a manufacturing parts classification system consisting of multiple layers of SVMs.

[0057] Figure 3 This is an example of an RBF SVM. The raw data 200 is fed into a set of independent feature extraction operators 205, which transform the raw data into independent views of the raw data based on different, domain-appropriate analyses, filters, and transformations. These feature values ​​are then fed into a set of RBF SVMs 310, 315, and 320, and the output of the RBF SVMs is sent to the next layer.

[0058] Figure 2 and Figure 3The fact that the first layer of the network consists of a single type of classifier is explained only to enhance the network's flexible architecture, which is the result of training via a feature selection process associated with each classifier. It is well known in the field that different types of classifiers have different properties in developing decision boundaries, and therefore each type of classifier has unique advantages when applied to a specific type of data and problem domain. If, in a specific manufacturing application, a network consisting only of linear SVMs is determined as the optimal network configuration for that specific application as a training result, then it is also easy to see in different applications that the optimal network configuration might be a configuration consisting only of RBF SVMs. A particular advantage of this system is that any configuration can be automatically generated as a result of training on data from each application, or actually any combination of individual classifier types and their sequences in the network that results in a composite classifier type.

[0059] The "sidecar" technique developed for this classifier is used for recursive feature elimination, using a linear SVM in parallel with any other type of classifier. During this process, the "target classifier" (the type of classifier used with the feature set after selecting a subset of the full feature set) and the linear SVM are trained in parallel. At each iteration, the error rate (or any other comparable classification performance metric) of the target classifier is recorded. Hyperparameter selection can also be performed around the target classifier at each iteration; in this case, the hyperparameters associated with the lowest error rate metric of the target classifier are also recorded. After training the target classifier and the linear SVM on the current feature set, an array of coefficient magnitudes obtained from the "sidecar" SVM is used to eliminate one or more features from the current set. The sidecar method then provides a simple and efficient ranking technique to select which features to eliminate in each step, thus avoiding the computationally intensive process of evaluating the target classifier's performance, where each feature is removed individually (or worse, within a unique group). The trained target classifier and its associated optimal hyperparameters are then stored in a data structure. At the end of the recursive feature elimination process, the target classifier with the lowest error rate metric is retrieved for use in the operating system. To improve the efficiency of our implementation, a fixed, predetermined percentage of features to be eliminated in each step are the parameter inputs to the training process, resulting in an effective, exponential reduction in the size of the feature set; this often balances the time spent optimizing feature sets of different sizes and types, including feature sets with thousands of features each.

[0060] The most comparable measure to classifier accuracy, and the one first revealed here for marginal region measurement, used in our implementation for feature set evaluation, is AUC, which is the area under the ROC curve. In SVM terminology, the AUC measure is insensitive to the width and shape of the marginal region, in terms of how "clean" the marginal region is. This can be seen through a simple comparison of the AUC between two different types of distributions. If the distribution for each class is Gaussian with finite support (truncated Gaussian tails), and the data from the two classes are just separated and have no overlap, then the AUC will reach its maximum, the ROC curve will be ideal, and it cannot be improved further. Further separating the two Gaussian distributions will not produce any gain in AUC, and the ROC curve will remain in its ideal form. If we consider a uniform ("boxcar") distribution whose area and location of the mean are exactly the same as the truncated Gaussian distribution, then the area of ​​the "valley" between the two distributions will increase further, and the marginal region (in SVM terminology) will become cleaner. This transformation still does not produce a change in the ROC curve, and therefore does not increase the AUC. It is clear that a larger separation between truncated Gaussians may yield better performance in this domain, as demonstrated by theoretical analysis of the marginal maximization optimization of SVM and the case described by a more compact, more uniform distribution.

[0061] To leverage the aforementioned differences, this invention modifies the standard AUC by initiating the computation of the "marginal region" metric by generating a normalized cumulative distribution for each of the two classes during SVM cross-validation testing. Then, as the SVM output threshold changes, the cross-validation error rate for each class is recorded from a large number of bootstrapping cross-validation training / testing cycles. The change in the output threshold corresponds to shifting the hyperplane along the discriminant axis toward either class distribution. The threshold varies at predefined steps, from values ​​far beyond one side of the marginal region (SVM marginality is defined as + / -1.0) to far beyond the other side (we use a constraint like -2.0 -> +2.0). Because the SVM output threshold varies from a lower to an upper bound, the average cross-validation error rate for each of the two classes ranges from low to high, albeit in opposite directions. This allows visualization of the marginal region during system development / debugging, as the marginal region measurement accumulates within the threshold range. Since these curves are monotonically increasing or decreasing, depending on the category, the combination of these two curves results in a maximum error rate at each sampling point, producing a curve that looks like a smooth valley between two plateaus of roughly equal height (with values ​​close to 1.0 at both ends). The marginal region developed for this classifier is based on the region above this curve, below a predefined threshold for the average error rate. Using the error rate threshold is to focus on the lower error rate regions of the "marginal valley," preventing the shape of the valley above the (available) error rate limit from influencing the comparison and selection process between feature sets in iterative features. The squared depth below the error rate threshold, integrated over the data below the threshold, is also used to further reinforce the bias towards low average error rates in the marginal valley regions of interest. In the development and testing of this classifier, visualizations of the two average error rate curves over feature set iterations clearly show which combinations of SVM types and feature types exhibit wide and deep marginal valley sizes across a range of feature sets, compared to feature types exhibiting narrow and / or shallow valleys. These visualizations can then also serve as a guide for selecting classifier types, hyperparameters, and feature types when developing more general classifier applications.

[0062] Many alternatives to multi-view system architectures have emerged in the literature, but they do not offer the specific advantages in robustness and accuracy demonstrated by our particular architecture. One alternative approach is for the system to use filter methods instead of wrapper methods for feature selection, where a measure of the discriminative utility of each feature is used as a feedforward of predictions to rank features in each set or view, rather than using a classifier to measure accuracy (as in wrapper methods). However, there are two main drawbacks to using filter methods. First, a measure of the potential classification utility of each feature, such as a Fisher score, is obtained separately for each feature; therefore, the potential synergistic effect of any features working together is not brought into the feature selection process. Second, there is a significant difference in performance actually seen using relatively simple metrics (such as Fisher scores or correlation-based metrics) compared to using classifiers with complex optimization algorithms (such as SVMs). Due to the nonlinear nature of the decision boundary developed by the k-NN classifier (or other nonlinear classifiers), even a simple implementation like the k-NN classifier can provide significantly better classification results than those estimated by simple metrics such as Fisher scores, if the data exhibits sub-clusters—even in one dimension, as when estimating the utility of each feature individually. Therefore, the combination of methods described in this document for feature selection via a wrapper approach—including the synergistic use of a sidecar SVM control that extensively leverages bootstrap cross-validation, a recursive feature elimination process, and a modified margin width estimation—provides a sophisticated feature selection process that results in a powerful, high-accuracy classifier system.

[0063] In other common multi-view architectures described in the literature, the output of the manufacturing part classifier can be a simple majority vote, average, or robust statistical combination of an ensemble of SVMs, or any other non-linear method, predefined or learned training, where data from each view is processed independently until merged at the output. In the most common representation of the multi-view paradigm, independent classifiers are used to process feature types / views separately, followed by a simple ensemble combination of classifiers for individual feature sets. Thus, there is either no ability to learn potential co-correlations between individual features across a set of views, or only a weighted combination of the learning from sub-classifiers from each view is used at the system output. In our classifier, our approach offers the benefit of independent and joint processing of different feature sets forming the multi-view front end. Because feature selection is used at each node at every level of the system, the flexibility of this approach allows the resulting classifier to represent (in terms of processing feature data from the view set) a set of intermediate results derived from each view that are completely isolated from all other views, a set using weighted feature sets across all views, or some intermediate combination of these two extremes—independent or joint use of information across views. Of particular note is the node-specific feature selection capability. This allows, if a classifier trained as a single output unit is used in the final layer, all possible sets of features selected from the pooling feature set, and all possible combinations thereof, to be refined and projected onto the view after training, serving as active features in the operational network. Complementarily, all possible configurations of composite classifiers are also possible, ranging from a single composite classifier type to all composite classifier types involved in processing the raw input data for manufacturing part anomaly detection.

[0064] Figure 4 is a schematic diagram of the second layer 128 of the SVM system. As shown in Figure 4, the second layer has three different types of inputs. Input box 400 is the output of the linear SVM from the first layer. Input box 405 is the output of the pooled feature data from the first layer, and input box 410 is the output of the RBF SVM from the first SVM layer. Multiple layers of SVMs can be used as needed, so each layer can provide input to the next layer. This architecture is also easily extended to other types of trainable classifiers, running in parallel with linear and RBFSVMs, such as SVMs with other kernel forms (multinomial, sigmoid, etc.), k-NN classifiers, Bayesian classifiers, or any other type of classifier that can provide a real-valued output representing classification confidence. Clearly, apart from the linear SVM used in the sidecar RFE pattern, system designers can decide for themselves to use any combination of different types of classifiers with real-valued outputs in parallel.

[0065] In one particular embodiment, as follows: Figure 4AThe classifier ensemble architecture shown in Figure 4 is as follows: box 400 represents a linear SVM in the first layer, and box 405 represents an RBF SVM in the first layer. Box 410 in Figure 4 represents a linear SVM in the second layer, and box 415 represents an RBF SVM in the second layer. This particular figure illustrates the composite classifier types “linear SVM->linear SVM,” “linear SVM->RBF SVM,” “RBF SVM->linear SVM,” and “RBF SVM->RBF SVM.” The system can have more SVMs in the first, second, or any layer, thus creating additional composite classifier types. The SVMs are not limited to RBF and linear types but can be any type of SVM as needed. The outputs of the second-layer SVMs 415 and 420 are then robustly normalized based on the distribution of the output data from these nodes during training, resulting in a dynamic range that balances the relative influence of each node's output across the output layer classifiers. These normalized outputs are then processed by sigmoid functions that asymptotically constrain and balance the output range at the output layer. The combined function of normalization and sigmoid transitivity is called adaptive sigmoid, and in Figure 4A This is shown as box 420. Then, in this embodiment of the ensemble classifier, the output of the adaptive sigmoid is processed by a statistical function, thereby generating a single output value representing a robust combination of the classifiers in the second layer after processing by the adaptive sigmoid function. Besides this set of composite classifiers in the input and output layers, there are two other types of composite classifiers, "pooling -> linear SVM" and "pooling -> RBF SVM", as shown below. Figure 5 As shown, they are also input to the output layer via a similar adaptive sigmoid function. Figure 4B In the diagram, box 408 represents the pooled set of selected features collected from the feature selection process that occurs during training of the first layer. Therefore, the output layer takes input from all six types of composite classifiers, robustly combines the results of these six classifier types, and adaptively configures itself based on the training data discussed earlier.

[0066] In this architecture, at the 128 hidden layers, there are two distinct and complementary approaches to generating the feature space used as input to the second layer of the SVM. The first approach uses the first layer of the SVM not as the final classifier used in the ensemble, but as a way to generate an optimized projection into a more discriminative feature space, which is a 2*n dimensional space, where n is the number of our independent views (feature types). Note that if more SVMs or other classifier types are used, the discriminative feature space size will be p*n, where p is the number of SVMs and / or other different classifier types. In the classifier described herein, the second layer consists of two basic types of SVMs, each operating on the output array of the first layer, and can then be operated on either as a set of classifiers or as a second-level further discriminative projection into a further optimized second feature space. This aspect of the architecture provides a method for independently extracting the most discriminative information (optimized SVM projections) from each view (feature type), and then optimally combining these projections to fully leverage the advantages of multi-view processing. The invention then describes a complementary approach that uses feature data selected across multiple views in conjunction to increase the space for optimized projections.

[0067] The second approach involves pooling the features selected by each SVM during training (the number of features allowed for pooling in each feature set is limited). The combination of these two basic techniques achieves two main goals: the first is to provide diversity in the feature space of the hidden layers. This set of 2*n projections allows for optimization within each feature type, isolating features from other types. Maintaining the integrity of the multi-view paradigm in this way is important, allowing for the detection of multiple manifestations of potential anomalies with some degree of independence. Pooling techniques provide a complementary set of optimized pooled features (optimized through the feature selection process), so that training an SVM using the pooled feature set in the next layer can leverage the synergistic interactions of features from different feature sets, if such synergy exists. They might do so because all features extracted from the same image will obviously have some degree of correlation with each other, and the selection of a set of feature extraction operators (feature types) is somewhat arbitrary and can never be completely comprehensive (due to the problem of sampling from a distribution that can never be truly known). The second is to provide weight balancing across feature types in terms of feature set size. It is well known that while a small subset of highly discriminative features may exist within a large set, their effectiveness can be overwhelmed by the larger set of noise and / or non-discriminative features, especially when the amount of training data is relatively small. These two sub-methods—view projection and cross-view pooling—both provide a certain balance of relative weights for feature types throughout the classification process in different ways.

[0068] The first sub-method (projection method) ensures that each feature set presented to the second layer has the same size because there are only two outputs, namely the SVM projections of each of the two SVM types, which become the component SVM operations in the intermediate feature space of the second layer. Note that the four types of second-layer composite SVMs (linear->linear, linear->RBF, RBF->linear, RBF->RBF) operate only in this feature space.

[0069] The second sub-method for feature set size balancing is pooling. This method has less balancing capability because a variable number of selected features are submitted to the pool from each feature set. Since the best feature set in any large feature set may still consist of a subset of the highest-ranked features in that group, the number of (ranked) features that can be submitted to each group is limited. The pooling can be adjusted manually or automatically to balance the potential discriminative information provided by the features associated with each feature extraction operator (view), as well as the need to balance the relative weights of each view. The pooled selected features are then used as a separate feature space for the second level and fed into two basic SVM types for training. Two additional types of composite classifiers (pooling -> linear, pooling -> RBF) are associated with the pooled feature set and provide the last two output nodes from the second layer.

[0070] While there has been recent work in the literature on stacked classifiers of traditional hidden units and SVMs, these papers do not report the use of hidden unit-specific feature selection techniques. Feature selection techniques used in classical MLPs and recent deep learning-based classifiers are not applied at the hidden unit level; instead, they are evaluated using methods such as “best brain injury” and dropout-based techniques, which are classifier-wide approaches. By applying feature selection independently on each hidden unit, the complexity of the classifier can be controlled, as applying feature selection (or hidden unit selection) across the entire classifier adds additional complexity, expanding the possible configurations a manufacturing part classifier might reach post-training, potentially negating the benefits of feature selection. Furthermore, the system of this invention uses extensive cross-validation at each hidden unit (SVM) to evaluate each candidate feature set; if bootstrap cross-validation were used as a wrapper across the entire classifier, this would easily become an unreasonable additional burden on CPU resources, as each SVM / node is trained using a wrapper technique with embedded bootstrap cross-validation. Due to this burden on CPU resources and training time, evaluating deep neural network classifiers (DNNs), or even classical multilayer perceptrons (MLPs), in a wrapper formula for feature selection is essentially impractical.

[0071] Using SVM to construct hidden layers provides an optimized feature space at each hidden layer level, especially when combined with feature selection, particularly when the size of the available training dataset is relatively small. The theoretical guarantee of the generalization performance of large-margin discriminators is enhanced in practice when feature set selection reduces the complexity of the classifier, thus better matching small training dataset sizes. The classifier system described in this paper optimizes generalization performance by incorporating feature selection as an integral part of training each SVM node in a manufactured parts classifier. The utility of each candidate feature set is evaluated using a wrapper method (retraining the SVM on each feature set iteration), which estimates the relative quality of the feature set through extensive use of bootstrapping cross-validation, thus providing the most accurate utility estimate of the aforementioned marginal area measure.

[0072] The architecture of using a set of classifiers with different architectures increases the robustness of the entire system. In this exemplary classifier, a unique sequence of classifier types is formed layer by layer to create a composite classifier. This invention uses two basic, common types of SVMs as the basis for diversity.

[0073] In this classifier architecture, the example system starts with two basic types of SVMs and builds a set of six unique types in the second layer: linear SVM array -> linear SVM, linear SVM array -> RBF SVM, RBF SVM array -> linear SVM, RBFSVM array -> RBF SVM, pooling -> linear SVM, and pooling -> RBF SVM. Since the distributions in the feature spaces formed by the two SVM types have different properties given the same training data, so too do the distributions of the outputs of the two SVM types. They operate on the different distributions of the two “types” of input features in the data space (refining projected and pooled features). It is clear that each of the four classifier types presents a different distribution in its output, thus each classifier type is not only unique in form but, more importantly, in its output distribution. Assuming that the error rate of each type is approximately equal, it is the variation in the output distribution that adds robustness to the output of the classifier set. Clearly, using composite SVMs of multiple types increases the diversity of classifiers for the diversity of the multi-view feature extraction process.

[0074] Since this architecture has no inherent limitation on the number of layers, there are no restrictions on adding a third layer except for the size of the training data pool. For this analysis, a third layer consisting of two SVM types can take all input combinations of the outputs of the six second-layer SVMs for a new set of SVMs. In this case, the exemplary system of the present invention can be formed, restricting itself to using at least two third-layer SVMs, with a unique set of classifiers that can be further expanded and used in a holistic manner at the output of the third layer. The proposed classifiers for the third layer will take the form of "all classifier types -> linear", "all classifier types -> RBF", etc. Then, in the example of adding two different SVM types to the third layer, a total of 12 different sub-classifier types will be defined. Also note that the system is not limited to using SVM types, because with the sidecar approach for feature selection, additional classifier diversity can be added by including other classifier types, such as weighted k-NN classifiers or traditional Bayesian classifiers, as long as they have the ability to output practical value. In the case of the k-NN type, we can use a distance metric to the most recently trained labels to indicate the class and confidence (a weighted version of k-NN), just like the output of the SVM. Adding classifier types increases our choice of classifier diversity, theoretically improving robustness, provided the overall classifier capacity remains within acceptable limits in terms of the VC (Vapnik-Chervonekis) dimension, considering the amount of available training data. While using bootstrapping cross-validation accuracy estimation for the entire classifier system (as a whole) is impractical from a CPU resource perspective, the expected accuracy of each layer can be estimated based on the average of the classifiers in each chosen operation point, since each layer can be considered as a set of classifiers, for example, using the average of the classifier set for each layer. Note that this applies to layers three and above, as the first two layers are predefined, based on a technique that learns the optimal combination of features across views through training using pooling and projection. The expected generalization performance can then be monitored as layers are added; for layers three and above, the optimal number of layers is determined by the average cross-validation accuracy per layer, stopping when the classifier capacity begins to decline (in the VC dimension sense) beyond the allowable limits of the available training data. Similarly, as mentioned earlier, diversity can be added to each layer by adding new types of SVMs or classifiers, starting from the first layer of the SVM. Reference Figure 1This is equivalent to adding hidden units 126, 127, 128, etc., in parallel with hidden unit 125, each of which is, for example, an SVM (126), a k-NN classifier (127), a Bayesian classifier (128) with a multinomial kernel, and continuing with other nonlinear classifiers with real-valued outputs. While diversity generally increases the robustness of the ensemble, this is contingent on the added classifiers typically being within the same accuracy range. Classifiers that perform worse than the standard for a given domain will degrade the performance of the ensemble. On the other hand, the potential synergistic effects between suboptimal feature sets (where the hidden units become in the next layer) should not be ignored. It is clear that the way to evaluate incremental performance gains by adding SVMs or other nonlinear classifiers to a given layer is by monitoring the average accuracy of the next layer. This is computationally feasible because only retraining subsequent layers is needed to estimate the incremental gains of the additional classifiers added to the previous layer. As before, this estimate is simply the average accuracy of the next layer's classifier set after retraining with each additional classifier. The computational burden of retraining subsequent layers is significant because each new type of classifier is added to the previous layer, and the number of features transferred to the next layer at these levels is very small. This contrasts sharply with the extremely high computational cost of retraining the entire network using bootstrap cross-validation every time a new hidden unit is added. This high computational cost arises because bootstrap samples from each training data set are selected as one of the training datasets for the entire network, and each hidden unit is then trained using its own bootstrap cross-validation process, thus affecting the bootstrap cross-validation within the hierarchical bootstrap cross-validation, leading to high computational costs. Therefore, it is clear that by using bootstrap cross-validation at each hidden unit, our proposed method here allows for the addition and evaluation of novel SVMs or other nonlinear classifiers at each layer, as well as new layers added to the entire network, with an efficient method for evaluating the impact on overall classifier performance. This inherently includes the statistical use of cross-validation results during the bootstrap process, without requiring the application of bootstrap cross-validation to the entire network every time a hidden unit is added to a layer, or an entire layer is added.

[0075] Figure 5This is a flowchart at 500 generally depicting an exemplary method according to an exemplary embodiment of the present invention. Method 500 includes testing for anomalies in a manufactured part using a two-layer arrangement discussed herein, generally shown at 502, where test 502 includes receiving at least one set of feature value data from at least one sensor or the manufactured part, generally shown at 504. Test 502 includes extracting at least two sets of refined data, wherein the at least two sets of refined data are numerical sets representing feature value data extracted from the sensor or the manufactured part using feature extraction operators, generally shown at 506. Test 502 includes classifying the at least two sets of refined data into a representation of a characteristic or output of the part and storing the at least two sets of refined data as stored data, generally shown at 508. Test 502 includes performing anomaly detection on the manufactured part using the stored data through a first layer of at least two support vector machines 120 and 125, and performing anomaly detection through a second layer of at least two support vector machines of different types, generally shown at 510. Method 500 uses the output of the second layer of support vector machines to compare or combine the results to determine the presence of anomalies, generally shown at 512. Method 500 may further provide characteristic data representations of scanned, imaged, tested, or output data from sensors or manufactured parts.

[0076] Method 500 may further include determining the severity level of an anomaly, wherein anomaly detection indicates the presence of an anomaly, and wherein the severity level is indicated by the real-valued output of the network. According to a particular embodiment, the method uses only binary decision-making in the system, although internally all outputs are real-valued. It is noteworthy that an anomaly may not assign a bin to the classification output in the system. In one embodiment, this is done only after expert review of the results of the classifier defined on the training data, combined with considerations such as cost, human factors, etc. In some cases, method 500 is cloud-based. In other instances of method 500, at least the first and second support vector machines are different. If there are two SVMs of the same type, using the same input, they must be trained to differentiate them somewhat. While bootstrapping training will provide some variation, it is likely that different hyperparameters will be used for each SVM of the same type, resulting in each SVM having different characteristics. In method 500, the support vector machine is typically RBF or linear, but it could also be other types of SVMs determined by the system designer, or even other forms of nonlinear classifiers. In addition, to compare, contrast, or analyze the integrated output of the second-layer support vector machine, statistical functions or output support vector machines operating on the second-layer support vector machine are used to provide abnormal function detection and indicate the severity of manufactured parts.

[0077] Additionally, method 500 may include anomaly detection of parts using stored data based on a two-layer support vector machine technique, comprising: constructing an anomaly classifier based on a decision-oriented system; and calculating the presence of anomalies based on the stored data, which determines the output from the trained support vector machine (or other classifier type), wherein the presence of an anomaly is indicated by the output of the entire system across a threshold determined during the training process. It is noteworthy that, in the case of an SVM, the stored data consists of coefficients. If any nearest-neighbor based classifier is used, the stored data will select examples (or all examples) from the training data.

[0078] A system implementing method 500 or another exemplary method of the present invention includes at least two "views" or feature types, generating arrays or matrices that transform raw data into feature data of each type. The system also provides at least two different types of classifiers. In this system, these can be any trainable classifier (including k-NN, which is actually trained in the sense of storing samples or prototypes, or can be trained with hyperparameters to obtain a more flexible version) that has or is configured to have real-valued outputs representing some form of classification statistical confidence. For this system, a feature selection method is associated with each classifier. This can be any form of feature selection method (there are many, such as filters, wrappers, forward search, backward search, hybrid methods, etc.) used to filter the feature dataset from each view to be used as input to each classifier during system operation.

[0079] In this exemplary system, the first layer operates on the features output by each view using at least two different types of classifiers (therefore, in the simplest case, 2 views * 2 classifier types = 4 classifiers in the first layer). The system also includes a feature pooling function that combines the feature sets selected from each view into a "pooled feature set." This occurs after a feature set is selected from each view. The selected features are pooled only from one classifier type (currently linear SVM). Pooling can or will be done using all the features selected for the first-layer classifiers.

[0080] The system includes a set of different classifier types (at least two) that operate on a pooled feature set. This is a set of auxiliary classifiers for either the first or second layer. There is an output function for combining the set of outputs from the second-layer classifiers. This function can be a statistical combination of all the second-layer classifier outputs (mean, robust mean, minimum, maximum, etc.) or a final classifier trained to effectively weight the combination of second-layer classifier outputs. Both approaches have advantages—statistical combination influences the ensemble of classifiers, while adding another trainable classifier to a single output unit might optimize the combination of second-layer classifier outputs but lose the ensemble property and can be overtrained (this is part of the reason for using ensembles—for robustness).

[0081] These elements, combined, provide the system's fundamental functionality: the ability to train a classifier focused entirely on a single view, allowing a low (#features) / (#training flags) ratio for improved generalization performance, isolating noisy features to obtain a better feature set, and optimizing the detection of different anomalies by centrally using different feature extraction operators. Furthermore, these elements provide classifiers that explore the potential for collaborative feature coordination across feature sets (cross views). If such cross-view synergy exists, this complements the use of independent classifier-based processing for each view.

[0082] In this system, feature selection provides several important functions. First, it optimizes the generalization performance of each classifier in the network by removing noisy or irrelevant features and reducing the ratio of (#features) / (#training labels) that is critical to domain performance, especially considering the relatively small amount of training data available to the target. Second, it provides a method for pooling features from different views, thus allowing some control over the difference in the number of features sent to the pool from different views.

[0083] Because feature selection is used at the input of each classifier, it allows for highly flexible network training, isolating features from a single view or a selected set of views, if optimal, whether they are in the ensemble or represent a refined item from each view of the first-layer classifier. It also supports the development of network architectures that can represent a wide range of composite classifier types (composite types being such as "linear -> RBF" or "pooling -> linear"). This is especially true if a final classifier / feature selection combination is used at the output—in which case, if we push the feature selection method to its limits, we can go from all the composite classifier types used to just one type. A feature in the final output classifier (if we use a classifier as the final output, rather than an ensemble); however, we would generally expect the feature selection of the output unit to select a mixture of classifier types. It also allows the feature selection function to dynamically adjust the overall network architecture, and, in terms of the composite classifier types used after training, it can be extended to even more kinds of composite classifier types if additional layers are added before the output unit or ensemble, as discussed in the section on the ability to scale architecture layer by layer, if it is found to be beneficial to performance. Furthermore, it allows for the construction of multiple composite classifier types for diversity. While diversity is known to have beneficial effects on robustness and accuracy, in general, for the same reason, it may be beneficial to use parallel, distinct, composite classifier types to process data flowing through a network with a single output unit as it does when it is in a merged form.

[0084] Furthermore, the system may include the use of SVMs due to their theoretical properties of generalization performance based on the theory of large-margin classifiers. The system can additionally add an efficient method for recursive feature elimination for feature selection. This is achieved by combining a linear SVM to control the feature selection process. In our current system, the linear SVM does this itself; we use a “sidecar” linear SVM and an RBF SVM to control the RFE process because an efficient RFE process, using the size of the trained SVM coefficients as a metric for feature utility, is incompatible with RBF SVMs and most other forms of nonlinear classifiers. The system can implement a grid search / classifier hyperparameter optimization wrapper around the recursive feature elimination process, thereby optimizing any available hyperparameters for training each classifier. The system can implement a bootstrap cross-validation error rate estimation process into the recursive feature elimination process, where a bootstrap cross-validation estimate of the error rate is obtained through each iteration of the feature removal process. The system can additionally include a “marginal region measurement” to estimate the optimal operating point regarding which feature set on the iteration set at which features are removed should be the feature set used in the system operation for each classifier. The system can also include a sigmoid function to control the dynamic range of the output for each classifier type, and to normalize the output across classifiers. As long as the output is real-valued, the system can include additional classifiers at each layer (which is also the layer processing the pooling feature set), representing a statistical measure of classification confidence. The system can also include an option to incrementally add additional layers layer by layer based on the average of the estimated cross-validation accuracy for each new layer (stopping when this starts to decline). The system can further include options for the final output layer, either as a collection of different classifiers implemented using some statistical combination of classifier outputs, or using the final classifier, with feature selection, to achieve a network with a single trained classifier as its final output, utilizing a weighted combination of classifiers selected through feature selection in the penultimate layer.

[0085] Various inventive concepts can be embodied in one or more methods, examples of which have been provided. Actions performed as part of a method can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than those illustrated can be constructed, which may include performing several actions simultaneously, even if they are shown as sequential actions in the illustrative embodiments.

[0086] Although various inventive embodiments have been described and illustrated herein, those skilled in the art will readily conceive of various other means and / or structures for performing the functions and / or obtaining the results and / or one or more advantages described herein, and each such variation and / or modification is considered to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials, and configurations described herein are exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on the specific application or application using the teachings of the invention. Those skilled in the art will recognize or be able to determine many equivalents of the particular inventive embodiments described herein using only conventional experimentation. Therefore, it should be understood that the foregoing embodiments are presented as examples only, and embodiments of the invention may be practiced in ways different from those specifically described and claimed within the scope of the appended claims and their equivalents. Inventive embodiments of the invention are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods, provided that such features, systems, articles, materials, kits, and / or methods do not contradict each other, is included within the scope of the invention.

[0087] The above embodiments can be implemented in any of a variety of ways. For example, embodiments of the technology disclosed herein can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code or instructions can execute on any suitable processor or set of processors, whether provided in a single computer or distributed across multiple computers. Furthermore, the instructions or software code can be stored in at least one non-transitory computer-readable storage medium.

[0088] Furthermore, a computer or smartphone used to execute software code or instructions via its processor may have one or more input and output devices. Among other things, these devices can be used to present a user interface. Examples of output devices that can be used to provide a user interface include printers or displays for visual presentation of output and speakers or other sound-generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computer may receive input information via speech recognition or other audible formats.

[0089] Such computers or smartphones can be interconnected via one or more networks in any suitable form, including local area networks (LANs) or wide area networks (WANs), such as corporate networks, as well as intelligent networks (INs) or the Internet. Such networks can be based on any suitable technology and can operate according to any suitable protocol, and can include wireless networks, wired networks, or fiber optic networks.

[0090] The various methods or processes outlined in this article can be encoded as software / instructions executable on one or more processors, employing any of a variety of operating systems or platforms. A particular advantage of multi-view architectures is their ease of adaptation to the use of parallel CPUs. Most computational workloads typically involve extracting feature data for each view. Furthermore, such software can be written using a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine.

[0091] In this regard, various inventive concepts can be embodied in a computer-readable storage medium (or multiple computer-readable storage media) (e.g., a computer memory, one or more floppy disks, optical disks, magnetic tapes, flash memory, USB flash drives, SD cards, field-programmable gate arrays or other semiconductor devices, or other non-transient media or tangible computer storage media) encoding one or more programs that, when executed on one or more computers or other processors, perform the methods implementing the various embodiments disclosed above. One or more computer-readable media may be transportable, such that one or more programs stored thereon can be loaded onto one or more different computers or other processors to implement the various aspects of the invention as described above.

[0092] The terms “program”, “software”, or “instructions” are used herein in a general sense to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to implement the various aspects described above. Furthermore, it should be understood that, according to one aspect, when executed, one or more computer programs performing the methods of the invention do not need to reside on a single computer or processor, but can be distributed in a modular manner across multiple different computers or processors to implement the various aspects of the invention.

[0093] Computer-executable instructions can take many forms, such as program modules, and can be executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. The functionality of program modules can usually be combined or distributed as needed in various embodiments.

[0094] Furthermore, data structures can be stored in any suitable form on a computer-readable medium. For simplicity of illustration, a data structure can be shown as having fields related by their position within the data structure. Such relationships can also be implemented by allocating storage to the fields, which has a location in the computer-readable medium that conveys the relationships between the fields. However, any suitable mechanism can be used to establish relationships between the information in the fields of a data structure, including by using pointers, labels, or other mechanisms that establish relationships between data elements.

[0095] All definitions defined and used herein should be understood as control dictionary definitions, definitions incorporated by reference in other files, and / or the general meaning of the defined terms.

[0096] As used herein, “logic” includes, but is not limited to, hardware, firmware, software, and / or combinations of each to perform one or more functions or one or more actions, and / or from another logic, method, and / or system. For example, depending on the desired application or need, logic may include software-controlled microprocessors, discrete logic such as processors (e.g., microprocessors), application-specific integrated circuits (ASICs), programmable logic devices, instruction-containing storage devices, electronic devices with memory, etc. Logic may include one or more gates, combinations of gates, or other circuit components. Logic may also be entirely embodied in software. In the case of describing multiple logics, the multiple logics may be combined into a single physical logic. Similarly, in the case of describing a single logic, that single logic may be distributed among multiple physical logics.

[0097] Furthermore, the logic of the various methods proposed herein for implementing this system can be adapted to improve existing computer-centric or internet-centric technologies that may not have prior simulated versions. The logic can provide specific functions directly related to the structure to solve and resolve some of the problems identified herein. By providing exemplary inventive concepts as specific logical structures and consistent functions for methods and systems, the logic can also offer significant additional advantages in solving these problems. Furthermore, the logic can provide specific computer implementation rules to improve prior art processes. The logic provided herein is not merely about collecting data, analyzing information, and displaying results. Moreover, part or all of the invention may rely on fundamental equations derived from specific arrangements of devices or components as described herein. Therefore, the parts of the invention relating to specific arrangements of components do not involve abstract concepts. Furthermore, the teachings of the invention and the appended claims are not limited to performing activities previously known, conventional, and routine in the industry. In some methods or processes of the invention that may incorporate certain aspects of natural phenomena, process or method steps are new and useful additional features.

[0098] The articles “a” and “an” used in the specification and claims, unless explicitly stated otherwise, shall be understood as “at least one”. The phrase “and / or” used in the specification and claims (if any) shall be understood as meaning “one or two” of the elements so combined, i.e., elements present in combination in some cases and separately in others. Multiple elements listed with “and / or” shall be interpreted in the same way, i.e., “one or more” of the elements so combined. In addition to the elements specifically identified by the “and / or” clause, other elements may optionally be present, whether related to or unrelated to those specifically identified elements. Thus, as a non-limiting example, when used in conjunction with open-ended language such as “comprising,” a reference to “A and / or B” may in one embodiment refer only to A (optionally including elements other than B); in another embodiment, it may refer only to B (optionally including elements other than A); and in yet another embodiment, both A and B (optionally including other elements); as used herein in the specification and claims, “or” shall be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” should be interpreted as inclusive, meaning it includes at least one, but also multiple elements or lists of elements, and (optionally) other unlisted items. Only when the opposite term is explicitly stated, such as “only one” or “exactly one,” or when used in a claim, will “consist of” refer to containing a quantity or a list of elements. In general, the term “or” as used herein should only be interpreted as indicating an exclusive alternative (i.e., “one or the other but not both”), if preceded by an exclusive term such as “any,” “one of,” “only one,” or “exactly one.” The phrase “consist of substantially” as used in a claim should have the general meaning used in the field of patent law.

[0099] As used herein in the specification and claims, the phrase “at least one” when referring to a list of one or more elements should be understood to mean at least one element selected from any one or more of the following: a list of elements, but not necessarily including at least one of each element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements referred to by the phrase “at least one,” whether or not they are related to those specifically identified elements. Thus, as a non-limiting example, “at least one of A and B” (or equivalently, “at least one of A or B”, or equivalently, “at least one of A and / or B”) could mean, in one embodiment, at least one, optionally including more than one A, with no B (and optionally including elements other than B); in another embodiment, at least one, optionally including more than one B, with no A (and optionally including elements other than A); in yet another embodiment, up to at least one, optionally including more than one A, and at least one, optionally including more than one B (and optionally including other elements); and so on.

[0100] When a feature or element is referred to herein as being "on" another feature or element, it may be directly on the other feature or element, or there may be intermediate features and / or elements. Conversely, when a feature or element is referred to as being "directly on" another feature or element, there are no intermediate features or elements. It should also be understood that when a feature or element is referred to as being "connected," "attached," or "coupled" to another feature or element, it may be directly connected, attached, or coupled to the other feature or element, or there may be intermediate features or elements. Conversely, when a feature or element is referred to as being "directly connected," "directly linked," or "directly coupled" to another feature or element, there are no intermediate features or elements. Although described or shown with respect to one embodiment, the features and elements thus described or shown can be applied to other embodiments. Those skilled in the art will also understand that a reference to a structure or feature configured as being "adjacent" to another feature may have portions that overlap with or lie below the adjacent feature.

[0101] For convenience, this document uses spatially related terms such as “below,” “below,” “under,” “above,” “over,” “behind,” “rear,” and “front” to describe the relationship between one element or feature and another, as shown in the figures. It should be understood that, in addition to the directions depicted in the figures, the spatially related terms are intended to cover different orientations of the device in use or operation. For example, if the device in the figure is inverted, an element described as “below” or “under” other elements or features would be oriented as “above” other elements or features. Thus, the exemplary term “below” can include both above and below orientations. The device may be oriented in other ways (rotated 90 degrees or in other orientations) and the spatially related descriptors used herein will be interpreted accordingly. Similarly, unless explicitly stated otherwise, the terms “up,” “down,” “vertical,” “horizontal,” “lateral,” “reverse,” “longitudinal,” etc., are used herein solely for illustrative purposes.

[0102] Although the terms “first” and “second” may be used herein to describe various features / elements, these features / elements should not be limited by these terms unless the context otherwise requires. These terms are used to distinguish one feature / element from another. Thus, the first feature / element discussed herein may be referred to as the second feature / element, and similarly, the second feature / element discussed herein may be referred to as the first feature / element, without departing from the teachings of the invention.

[0103] Examples are implementations or embodiments of the present invention. References to "example," "one example," "some examples," "a specific example," or "other examples," etc., in the specification indicate specific features, structures, or characteristics described. These examples are included in at least some examples of the present invention, but not necessarily in all examples. The various appearances of "example," "one example," "some examples," "a specific example," or "other examples," etc., do not necessarily refer to the same example.

[0104] If this specification states that it "can," "may," or "may" include a component, feature, structure, or characteristic, then it is not necessary to include that particular component, feature, structure, or characteristic. If the specification or claims refer to the element "a" or "an," this does not mean that there is only one element. If the specification or claims refer to the element "additional," this does not exclude the existence of multiple additional elements.

[0105] As used herein in the specification and claims, including in the embodiments, and unless otherwise expressly stated, all figures may be understood as if beginning with the words “about” or “approximately”, even if the term is not explicitly stated. When describing magnitude and / or location, the phrase “about” or “approximately” may be used to indicate that the described value and / or location is within a reasonably expected range of value and / or location. For example, a numerical value may have + / - 0.1% of a specified value (or range), + / - 1% of a specified value (or range), or + / - 2% of a specified value (or range), + / - 5% of a specified value (or range), + / - 10% of a specified value (or range), etc. Any numerical range referenced herein is intended to include all subranges contained therein.

[0106] Furthermore, any method of the present invention may be performed in a different order than those described herein. Therefore, unless explicitly stated otherwise, the order of the method should not be considered a limitation. It will be appreciated that performing some steps of the method in a different order can yield similar results.

[0107] In the claims and the foregoing description, all transitional phrases, such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “consisting of,” etc., should be understood as open-ended, meaning including but not limited to. As described in the U.S. Patent Examination Procedure Manual, only the transitional phrases “consisting of” and “substantially consisting of” should be closed or semi-closed transitional phrases, respectively.

[0108] In the preceding description, certain terms have been used for the purpose of brevity, clarity, and understanding. No unnecessary limitations should be implied from these terms beyond the requirements of the prior art, as they are used for descriptive purposes and are intended to be broadly interpreted.

[0109] Furthermore, the descriptions and illustrations of various embodiments of the present invention are examples, and the present invention is not limited to the exact details shown or described.

Claims

1. A method for testing anomalies in the manufacture of parts, comprising: Testing for abnormal conditions in manufactured parts, including: Receive at least one set of characteristic value data from at least one sensor or manufactured part; Extract at least two sets of refined data, wherein each set is extracted using different, domain-appropriate and partial feature extraction operators. The at least two sets of refined data are digital sets representing the transformation or analysis of feature value data received from images of manufactured parts. The at least two sets of detailed data are classified as representations of part characteristics or outputs, and the at least two sets of detailed data are stored as stored data; The first layer uses at least two different types of classifiers, which operates on the features output from each feature type, and is isolated from the processing of data from other feature types. Anomaly detection of the manufactured part is performed using the stored data via at least a first support vector machine, and anomaly detection of the manufactured part is performed using the stored data via at least a second support vector machine; and At least a second layer is used, comprising: at least a third support vector machine, which is a support vector machine of the same type as the first support vector machine that receives the output from the first layer; and at least a fourth support vector machine, which is a support vector machine of the same type as the second support vector machine that receives the output from the first layer, connected to provide a parallel set of at least four composite classifier types, whose outputs are compared or combined to determine the presence of anomalies. The feature value data represents data from images of the sensor or manufactured parts. The use of at least a second layer further includes: At least a fifth support vector machine, which is of the same type as the first support vector machine that receives a pooled feature set as input; and at least a sixth support vector machine, which is of the same type as the second support vector machine that receives a pooled feature set as input, are connected to provide a parallel set of at least six composite classifier types, whose outputs are compared or combined to determine the presence of anomalies, the pooled feature set being composed of a feature set selected from each feature type.

2. The method for testing abnormal conditions in manufactured parts according to claim 1, wherein, This method is cloud-based.

3. The method for testing abnormal conditions in manufactured parts according to claim 1, wherein, The support vector machine is either radial basis function (RBF) or linear.

4. An apparatus for testing abnormal conditions in manufactured parts, comprising: A storage unit for storing at least one of multiple feature value data or multiple coefficient data in a feature or coefficient database, wherein each feature value data includes multiple feature values ​​and each coefficient value data includes multiple coefficient values; A non-transitory computer-readable storage medium includes instructions that, when executed by a processor, cause the following operation: Receive feature value data of a manufactured part, wherein the feature value data includes raw feature value data from an image of the manufactured part; Extract at least two sets of refined data, wherein each set is extracted using different, domain-appropriate and partial feature extraction operators. The at least two sets of refined data are digital sets representing the transformation or analysis of feature value data received from images of manufactured parts. The at least two sets of refined data are categorized into mechanical properties, chemical properties, output of the manufactured part, or other similar characteristics of the manufactured part, and the refined data is stored as stored data; and A feature database of SVM coefficients, or coefficients associated with any similar classifier after training, is coupled to a processor and configured to perform anomaly detection on the manufactured part using the stored data, in an arrangement forming at least two different types of at least two layers of support vector machines and a set of composite classifier types formed by combinations of different types of classifiers between the first and second layers that process at least two sets of refined data. The anomaly detection device also includes a central database coupled to the microcomputer and the feature database, wherein the central database is configured as follows: The feature value data is analyzed to obtain the analyzed feature value data; The parsed raw sensor data is classified into categorized parsed raw sensor data as a representation of the mechanical, chemical, or other similar properties of the part, and the categorized raw parsed sensor data is stored after classification, wherein the categorized raw parsed sensor data corresponds to the at least two sets of refined data, and wherein extracting the at least two sets of refined data from the raw sensor data includes extracting the categorized raw parsed sensor data from the raw parsed sensor data. The first layer of at least two different types of classifiers operates on features output from each feature type, isolated from the processing of data from other feature types. This first layer includes: at least a first support vector machine that uses the stored data to perform anomaly detection on the manufactured part; and at least a second support vector machine that uses the stored data to perform anomaly detection on the manufactured part; and The second layer of support vector machines includes: at least a third support vector machine, which is of the same type as the first support vector machine that receives the output from the first layer; at least a fourth support vector machine, which is of the same type as the second support vector machine that receives the output from the first layer, connected to provide a parallel set of at least four composite classifier types, whose outputs are compared or combined to determine the presence of anomalies; at least a fifth support vector machine, which is of the same type as the first support vector machine that receives a pooled feature set as input; and at least a sixth support vector machine, which is of the same type as the second support vector machine that receives a pooled feature set as input, connected to provide a parallel set of at least six composite classifier types, whose outputs are compared or combined to determine the presence of anomalies, wherein the pooled feature set is composed of a feature set selected from each feature type.

5. The apparatus for testing abnormalities in manufactured parts according to claim 4, wherein, The support vector machine is either radial basis function (RBF) or linear.

Images