SYSTEM AND METHOD FOR THE JOINT DETECTION, LOCALIZATION, SEGMENTATION AND CLASSIFICATION OF ANOMALIES IN IMAGES

Description

STATEMENT REGARDING FEDERALLY SPONSORED DEVELOPMENT

[0001] Development for this invention was supported in part by Technology Investment Agreement number W15QKN-19-3-0003 awarded by US Army Contracting Command-Picatinny, that operates under the U.S. Department of Defense. Accordingly, the United States Government may have certain rights in this invention.TECHNICAL FIELD

[0002] The present disclosure relates to applying artificial intelligence to visual inspection systems. In particular, disclosed embodiments relate to a deep learning-based solution for joint detection, localization, segmentation and classification of anomalies in images within industrial settings with scarce labeled data.BACKGROUND

[0003] Deep learning models for computer vision typically require a large amount of labeled data for training. However, in industrial applications, such as in visual quality inspection of manufactured parts, acquiring sufficient amount of labeled data to train a model is often difficult and expensive.

[0004] Defects are typically rare in manufacturing. To collect enough data to train a deep learning model (e.g., ~1000 images of defective parts on a production line), the vision system may have to be in data collection mode for several months or even years. Furthermore, even with enough training data, the generalization capabilities of the model may still rely on the specific parts and defects that it is trained on. Inevitable parts and defects variability make it difficult to automatically inspect and generate repair strategies, or perform direct decision making, using traditional deep learning models.

[0005] Typically, there are large number of manufacturing process parameters that affect the quality of a manufactured part, controlling which could reduce the observed part variability (i.e., occurrence of defects). Understanding defects and their causes may represent the first step to maintaining quality of a manufactured part within a desired specification. Identifying various categories of defects may provide more insight about the source of error in the manufacturing process. However, the above task is challenging due to the scarcity of labeled defect data in manufacturing. Jake Snell ET AL: "Prototypical Networks for Few-shot Learning" teaches a few shot classification using a neural network learning a metric space in which classification can be performed by computing distances to prototype representations of each class. In the training phase of the neural network, class prototypes are computed from support examples, and a classification loss is computed.SUMMARY

[0006] Aspects of the present disclosure can be used to develop advanced deep neural networks to perform visual inspection of anomalies that is trainable on coarsely labeled limited defect data or unlabeled data. This flexibility helps to overcome the challenges of limited defect data in industrial applications, such as in manufacturing. The developed deep learning-based framework is configured to perform detailed anomaly analysis, including anomaly detection, anomaly localization, anomaly segmentation, and anomaly classification with little to no supervision.

[0007] According to a first aspect, a computer-implemented method is provided for training a deep learning-based vision system for joint detection, localization, segmentation and classification of anomalies in images. The method comprises inputting images from a training dataset to train an autoencoder by backpropagation of an autoencoder loss that includes an attention loss. The attention loss encourages an encoder network of the autoencoder to generate attention maps maximizing focus on normal regions of images, whereby, given an anomalous input image, the encoder network is configured to generate an attention map that spatially localizes an anomaly. The method further comprises training an image segmentation model. This includes extracting feature representations using the encoder network from images in the training dataset, and alternatively optimizing cluster labels of pixels in a forward process and optimizing the feature representations by backpropagation of a segmentation error. The method further comprises tuning the encoder network in a few-shot learning process for classifying anomalies. This includes extracting feature representations using the encoder network from anomalous images in respective support and query sets created from the training dataset, using support set feature representations to learn embeddings of anomaly classes, and updating the encoder network based on a classification error computed using query set feature representations.

[0008] Other aspects of the disclosure implement features of the above-described methodology in computer program products and computing systems for joint detection, localization, segmentation and classification of anomalies in images.

[0009] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced. FIG. 1 schematically illustrates a learning framework for anomaly detection and localization extended to unsupervised anomaly segmentation, according to an example embodiment. FIG. 2 illustrates a trained vision system for joint detection, localization, segmentation and classification of anomalies in images according to an example embodiment. FIG. 3 illustrates an example of (i) an input image, (ii) an attention map spatially localizing an anomalous region in the input image and (ii) segmentation of a defective object area in the input image based on binary clustering. FIG. 4 illustrates examples of anomaly classes predicted from query images. DETAILED DESCRIPTION

[0011] An anomaly refers to an event or occurrence which differs from the norm. In machine learning, anomaly detection is usually based on the assumption that anomalies occur rarely in the dataset and that their features differ from the normal instances significantly. Deep learning methods can be used to train neural networks to perform visual quality inspection for manufacturing processes, wherein a defect on a manufactured part can be identified based on detection of an anomaly in an image of the part. Anomaly detection based on deep learning methods can largely reduce or obviate the need for anomalous data (e.g., images with defects) in the training process, as the model training can be substantially implemented using normal data (e.g., defect-free images), which is abundantly available. Deep learning methods can also be used for localizing an anomaly in an image by generation of attention maps. While attention maps provide general spatial information of an anomaly, they do not provide enough information for finer tasks such as segmentation of the anomaly or classification of the anomaly (e.g., defect classes), which may be desirable for detailed analysis of defects, such as determining a root cause and implementing corrective actions.

[0012] The disclosed methodology provides a unified deep learning framework for training a vision system to perform anomaly detection and localization, unsupervised anomaly segmentation and anomaly classification. Anomaly detection and localization is performed using an autoencoder. An autoencoder architecture comprises an encoder network having a number of convolutional layers that can generate a latent variable from an input image, and a decoder network that can reconstruct an output image from the latent variable. In accordance with the disclosed methodology, the autoencoder is trained on images in a training dataset based on an autoencoder loss, which includes an attention loss, typically in addition to other losses (e.g., reconstruction loss). The attention loss encourages the encoder network to generate attention maps maximizing focus on normal regions of images. Thus, given an anomalous input image, the encoder network is configured to generate an attention map that spatially localizes an anomaly. One example of autoencoder architecture for anomaly detection and localization that can be suitably employed herein is described in the Internal Application Publication WO2021062133A1.

[0013] For anomaly segmentation, the disclosed methodology uses the convolution layers of the same encoder network (used for anomaly detection and localization) as a feature extractor for an image segmentation model in an unsupervised learning process. Training the image segmentation model includes alternately executing forward and backward processes as follows. The forward process includes extracting feature representations from training images using the encoder network and optimizing cluster labels of pixels. The backward process includes optimizing the feature representations by backpropagation of a segmentation error, whereby parameters of the encoder network and the image segmentation model are simultaneously updated.

[0014] It has been observed that joint training of the autoencoder and the image segmentation model provides faster training and greatly improves the accuracy of the respective models, which inform each other in the learning process. In one embodiment, the autoencoder and the image segmentation model may be jointly trained by updating their respective parameters based on the autoencoder loss and the segmentation error in alternate iterations. The joint training also pre-trains the encoder network for anomaly classification using few-shot learning. In some embodiments, the autoencoder may be trained based on the autoencoder loss for a specified number of training epochs to attain a desired degree of model stability, prior to commencement of the joint training of the autoencoder and the image segmentation model.

[0015] According to the disclosed methodology, the encoder network, which has been trained for anomaly detection, localization and segmentation, is subsequently tuned in a few-shot learning process for anomaly classification tasks. The few-shot learning process is based on extracting feature representations using the encoder network from anomalous images in respective support and query sets created from the training dataset. The support set feature representations are used to learn embeddings of anomaly classes in the support set. The query set feature representations are used to compute a classification error, which may be backpropagated to update the encoder network.

[0016] The disclosed methodology is particularly suitable for visual quality inspection in a manufacturing process for segmenting defect areas and classifying defects of manufactured parts. The model predictions can be useful for determining causes of defects as well as for implementing corrective actions, such as adjusting manufacturing process parameters, so that similar defects are prevented in future.

[0017] FIG. 1 illustrates a deep learning framework 100 for anomaly detection and localization extended to unsupervised anomaly segmentation, according to an example embodiment. The framework 100, including the autoencoder 110 and the image segmentation model 140, and components thereof, may be implemented by a computing system in various ways, for example, as hardware and software modules. The programming for the software modules may take the form of processor-executable instructions stored on non-transitory machine-readable storage mediums and the hardware may include processors to execute those instructions. The processing capability of the systems, devices, and modules described herein, may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements.

[0018] The autoencoder 110 is trained on images from a training dataset 112. Consistent with a disclosed embodiment, the autoencoder 110 may be trained in a weakly supervised learning process with coarsely annotated images. For example, for a visual quality inspection use-case, the training dataset 112 may include image-level labels for "normal" images representing nominal manufactured parts and "anomalous" images representing defective manufactured parts. In other embodiments, the autoencoder may be trained in an unsupervised learning process.

[0019] As shown in FIG. 1, input image x is passed through encoder network 114 comprising a number of convolutional layers (e.g., including a residual network) that generates a latent variable z as output. To preserve spatial relationship between the latent variable z and the input image x, the latent variable z may include a convolutional variable. The latent variable z is used by decoder network 116 (e.g., also including a residual network) to generate a reconstructed image x̂, which is a reconstruction of original input image x. The loss function L for training an autoencoder typically includes a reconstruction loss L R , which is a measure of an error (e.g., a mean-squared error) between the input images x and the reconstructed images x̂ in a training batch. The shown example depicts a variational autoencoder, where the loss function L also includes Kullback-Liebler divergence (KL) to calculate the statistical distance between the true distribution and an approximating distribution in the latent variable z. To obviate a blurry reconstruction, the shown example uses a discriminator 118 to improve the stability of the training and generate sharper reconstructed images x̂ using adversarial learning based on an adversarial loss L adv .

[0020] To train the autoencoder 110 to spatially localize anomalies, attention maps are generated utilizing the convolutional layers of the encoder network 114. The attention maps may be computed from a feature representation of the latent variable z generated by the encoder network 114 from normal images x in the training dataset 112. The attention maps may be computed using class activation mapping (CAM) methods, such as Grad-CAM, among others. Since attention maps obtained from feature representations (or "feature maps") illustrate the regions of the image responsible for specific activation of neurons in the feature maps, an attention loss L attn is incorporated into the overall loss used for training the autoencoder 110. The attention loss L attn encourages the encoder network 114 to generate attention maps maximizing focus on normal regions of images. During testing, given an anomalous input image, the encoder network 114 generates an attention map that can be used to spatially localize an anomaly from the areas of the image that the attention map does not focus on.

[0021] In a weakly supervised setting, a few anomalous images, typically (but not necessarily) in the range of 5-20% of the training dataset 112, may be used. The training dataset 112 may be labeled with image-level labels ("normal", "anomalous") without pixel-wise annotation. In this setting, a binary classifier 120 may be connected at the output of the feature representation of the latent variable z. Given an image x and its ground truth label y, the prediction p of the classifier 120 can be defined as p ∈ {ca, cn}, where ca and cn are anomalous and normal classes, respectively. The binary classifier 120 may be trained using a binary classification loss, for example, a binary cross-entropy loss L bce , based on the predicted class label p and the ground truth label y. As shown in FIG. 1, the convolutional latent variable z may be cloned into a new tensor and flattened to form a fully connected layer z fc , to which, a 2-node output layer may be added to form binary classifier 120. Variables z and z fc share parameters. Flattening z fc may enable a higher magnitude of gradient backpropagation from prediction p for computing the attention maps.

[0022] In the shown example, the attention maps include an attention map for the normal class A x cn and an attention map for the anomalous class A x ca , which are generated by gradient backpropagation operation 122 where gradients from the prediction p are backpropagated via the binary classifier 120 to the last convolutional layer of the encoder network 114. Here, normal and anomalous attention represent respectively the areas affecting the normal and anomalous prediction p of the binary classifier 120. The attention loss L attn may be designed to minimize an area of the image covered by the attention map of the anomalous class A x ca and simultaneously enforce the attention map of the normal class A x cn to cover the entire image. In this way, the encoder network 114 may be encouraged to generate an attention map that focuses on all normal (i.e., non-anomalous) regions of the image to reduce the need for large amount of anomalous training data. Based on this training regimen, when the network is given an image classified as anomalous by the binary classifier 120, the underlying anomalous attention map will focus on the image regions considered as abnormal by the solution.

[0023] Since an attention map is computed by backpropagating the gradients from prediction p, any incorrect prediction p would generate an undesired attention map. This would lead to the encoder network 114 learning to focus on erroneous areas of the image during training, which is to be avoided using the attention loss L attn . The attention loss L attn may therefore be computed only for the normal images correctly classified by the binary classifier 120 (i.e., if p = y = cn). The attention loss L attn,1 for each image may be defined as follows: L attn , 1 = 1 p = y = cn A x cn ∑ i , j 1 − A x cn i , j + A x ca i , j

[0024] In equation (1), 1(·) is an indicator function. The attention loss L attn is the average of loss L attn,1 over all images in a training batch. The final autoencoder loss L final may then be defined as follows: L final = w 1 L + w 2 L adv + w 3 L bce + w 4 L attn where w 1 , w 2 , w 3 , w 4 are empirically chosen weights.

[0025] Incorporating the binary classifier loss L bce in the final autoencoder loss L final enables training the binary classifier 120 to be used during testing to predict whether an input image is normal or anomalous. An attention map may be generated for a test image classified as anomalous, which can be used to spatially locate an anomaly in the test image.

[0026] An attention map can indicate regions of the input image that the encoder network focusses on to detect an anomaly. However, for detailed analysis of a defect, it is often helpful to gain an insight on the geometry of the defect. Anomaly segmentation provides a finer form of anomaly localization by assigning labels to image pixels for separating pixels that belong to a defect area, which can better describe the geometry of a defect. Traditional supervised deep learning-based segmentation algorithms need detailed labeled data that shows the segmented defects, which can be costly to acquire in industrial settings. The disclosed methodology is based on an unsupervised image segmentation algorithm that does not depend on detailed ground truth segmentation map data.

[0027] An example of unsupervised image segmentation is described in the publication: A. Kanezaki, "Unsupervised Image Segmentation by Backpropagation," 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 2018, pp. 1543-1547.

[0028] Continuing with reference to FIG. 1, the described deep learning framework for anomaly detection and localization can be extended to train an image segmentation model 140 in an unsupervised learning process. Consistent with a disclosed embodiment, the image segmentation model 140 may be trained for optimizing cluster labels of pixels based on a binary clustering, such that the image segmentation model 140 is configured to segment a defect area from a background in an anomalous image. In the unsupervised learning process, given an input image x from the training dataset 112, a feature representation {x n } is extracted by passing the image x through convolutional layers of the encoder network 114. A joint optimization of the cluster labels of pixels together with feature representations {x n } is carried out to update parameters of the encoder network 114 and image segmentation model 140. Alternate execution of pixel label prediction in a forward process and network parameter learning in a backward process may be carried out to meet three main constraints: (a) pixels of similar features will be assigned the same label, (b) spatially continuous pixels will be assigned the same label, and (c) the number of unique labels is expected to be large.

[0029] According to the disclosed embodiment, cluster labels of pixels may be optimized in the forward process as follows. From an input image x, a feature representation x n ∈ ℝ d n = 1 N may be extracted using the encoder network 114 based on 2D convolutions, where each pixel n is represented by a d dimensional feature vector x n , and N is the total number of pixels in the input image x. A classification model may be applied to the extracted feature vectors x n to generate a pixel-wise response map. The classification model can include any model that can be used to classify pixels having similar feature vectors into the same label, forcing the constraint (a). In the shown embodiment, the classification model includes a linear classifier 142. The response map {y n } may be generated by the linear classifier 142 using 1D convolution as y n = W c x n + b c n = 1 N , where y n ∈ ℝ q , q being the number of clusters, and where W c and b c are parameters of the linear classifier 142 such that W c ∈ ℝ q × d and B c ∈ ℝ q .

[0030] To force constraint (c) and avoid under-segmentation, the original response map {y n } may be normalized by a batch normalization process 144 to generate a normalized response map y ′ n n = 1 N , such that {y' n } has zero mean and unit variance, where y ′ n ∈ ℝ q . This gives an even chance to each y' n,i (i = 1, ... q) to be the maximum value of {y' n } across axes. The cluster label l n for each pixel n may be computed by determining the dimension that has the maximum value in y' n , using argmax classification operation146. The argmax classification operation 146 thus yields cluster labels {l n }, which result in a segmented image 148 based on a clustering of the feature vectors into q clusters.

[0031] To achieve spatial continuity of pixels as per constraint (b), neighboring pixels may be grouped into super-pixels via a super-pixel refinement operation 150, such that each pixel in a super-pixel is forced to have the same cluster label. The cluster label for all pixels in a super-pixel may be assigned the most frequent label in the cluster. In this manner, refined cluster labels {l' n } may be generated, resulting in a segmented image 152 having reduced number of cluster labels. The cluster labels, as well as the number of clusters, may thus be optimized in the forward process. In the shown example, the maximum number of cluster labels after super-pixel refinement is set as 2, whereby a defective object area can be separated from the rest of the image.

[0032] In a backward process, the parameters of the image segmentation model 140 and the encoder network 114 are updated based on computation of a segmentation error 154 using the cluster labels and the pixel-wise response map. In the shown example, the segmentation error 154 is computed as a SoftMax loss between the normalized response map {y' n } and the refined cluster labels {l' n }. The segmentation error 154 may be backpropagated (as shown by dashed lines 154) based on gradient descent to update parameters of the convolutional filters of the encoder network 114 and the parameters of the linear classifier 142.

[0033] In one embodiment, the forward and the backward processes of unsupervised learning for image segmentation may be executed after each training iteration of the autoencoder for anomaly detection and localization (described above), to obtain the final prediction of cluster labels {l' n }. In this manner, the autoencoder 110 and the image segmentation model 140 may be jointly trained while learning from each other, to provide faster training and produce a high degree of prediction accuracy.

[0034] Once an anomaly is detected and localized, it is desirable to categorize it at a more granular level to gain a better understanding of the root cause of a defect. However, the challenge is that for many of the anomaly classes, sufficient data is not available for training traditional deep learning-based classification algorithms. For example, in many manufacturing applications, available data may include less than 10 samples per class of defect. Moreover, a traditional neural network (such as a convolutional neural network) usually inputs a batch of samples each time in training until the entire training dataset is covered, and it is thus not robust to cope with imbalanced data.

[0035] The disclosed methodology uses Few-shot learning (FSL), also known as low-shot learning (LSL), to train the encoder network 114 for anomaly classification tasks. FSL is a type of meta-learning where a learner is trained on various related tasks during the meta-training phase to generalize well to unseen tasks with just a few instances during the meta-testing phase. The encoder network 114, which has been already trained on anomaly detection, localization and segmentation as described in FIG. 1, can thus function as a pre-trained feature extractor that may be subsequently fine-tuned using FSL. Using this approach, depending on data availability, 1-10 samples per class could be sufficient for training. This approach can also handle the issue of imbalanced data distribution over the number of class samples.

[0036] According to the disclosed methodology, the training dataset 112 is split into a support set and a query set based on a M-way K-shot paradigm, where M is the number of anomaly classes and K is the number of samples per class. The pre-trained encoder network 114 is used for feature extraction for the support set as well as the query set. The support set and query set comprise anomalous images from the training dataset 112, which have been assigned a ground truth anomaly class label. The support set feature representations are used to learn embeddings of anomaly classes. The query set feature representations are used to compute a classification loss for updating the encoder network 114.

[0037] An example approach to FSL suitable for the disclosed methodology involves employing a prototypical network, although other FSL approaches may be also applicable. A prototypical network learns a metric space where classification can be executed by computing distances to prototype representations of each class. In the present application, the encoder network 114 may function as the prototypical network. In a meta-training phase, the support set feature representations extracted using the encoder network 114 may be used to compute a prototype of each anomaly class in the support set as a mean vector representation of samples in that anomaly class in an embedding space. Samples in the query set may be classified based on computing distances to the prototype of each anomaly class in the embedding space, and therefrom computing the classification error between the predicted classification and the ground truth label. The classification error may be backpropagated to update or tune parameters of the encoder network 114. In a meta-testing phase, the well-learned embedding feature representations extracted using the encoder network 114 may be used to compute the distances among the query images and the class prototypes generated from the support set. The final anomaly classification may be performed by calculating the posterior probability for the query instance. In some embodiments, class activation mapping may be used to visualize and interpret the discriminative regions of interest most relevant to specific anomaly classes.

[0038] FIG. 2 illustrates a deep learning-based vision system 200 that has been trained for joint detection, localization, segmentation and classification of anomalies in images according to the embodiments disclosed herein. The vision system 200 can be suitably deployed for visual quality inspection of parts produced by a manufacturing process. Visual quality inspection may be carried out by acquiring an image 202 of a manufactured part to be inspected. The image 202 may be acquired, for example, via a camera at an inspection station of a production line. The acquired image 202 may be inputted to the encoder network 114 with trained convolutional filters to extract a feature representation 204 of the acquired image 202. The feature representation 204 may represent an output of a final convolutional layer of the encoder network 114. The extracted feature representation 204 can be used for joint anomaly detection (classifying image 202 as normal or anomalous), localization and segmentation, and if anomalous, predicting an anomaly class, as described below.

[0039] For anomaly detection, the extracted feature representation 204 of the image 202 may be passed through the trained binary classifier 120 to predict an image class label 206 ("normal or "anomalous"). If the predicted image class label 206 is "anomalous", an anomaly is said to be detected in the image 202, which is indicative of a defective part. In some embodiments, the gradients of the binary classifier 120 prediction may be backpropagated to generate an attention map that spatially localizes an anomaly in the image 202 based on the areas that affect the "anomalous" prediction by the binary classifier 120. An output image 208 may be generated that depicts the defective part with localization of the detected anomaly based on the attention map.

[0040] For anomaly segmentation, the extracted feature representation 204 of the image 202 may be fed to the image segmentation model 140 with trained parameters (e.g., linear classifier parameters) to predict cluster labels for pixels in the image. Consistent with the embodiment described above, a binary clustering of image pixels may be executed using the image segmentation model 140, to generate an output image 210 with refined pixel labels separating a defective object area from the background.

[0041] An example of anomaly localization and segmentation is shown in FIG. 3. Here, the input image 202 shows portion of a manufactured part that has a glue dot 302 deposited on a part surface by a robotic arm holding a deposition nozzle. The nominal or "normal" shape of the glue dot 302 is approximately circular. However, in the shown example, the glue dot has an elongated portion, which differs from expected shape and therefore results in the image 202 being classified as "anomalous". The image 208 illustrates an output of anomaly localization using an attention map, which highlights only the specific area 304 of image that contributed to the "anomalous" classification. The image 210 illustrates an output of anomaly segmentation that segments the defective object area from the rest of the image based on binary clustering of image pixels. Note that the defective object in this case is the glue dot and the segmented area 306 includes the entire defective glue dot, which is descriptive of the precise geometry of the defect to be used for further defect quantification.

[0042] Continuing with reference to FIG. 2, in the event that an anomaly has been detected in the image 202 (represented by decision block 212), an FSL framework 220 may be triggered to predict an anomaly class. The extracted feature representation 204 from the encoder network 114 may be fed to the FSL framework 220 as a query feature vector 222. The FSL framework 220 may be based on embeddings learned using the encoder network 114. Consistent with the disclosed embodiment, the encoder network 114 may function as a prototypical network, which may be used to extract feature representations 204 to define feature vectors for a support set 224. The support set may comprise a set of anomalous images with ground truth class labels for a number of anomaly classes with one or more samples per class. The anomaly classes in the support set may be similar to or different from the anomaly classes that were used during meta-training. The feature vectors for the support set 224 may be used in prototypical learning 226, which involves computing a class prototype 228 of each anomaly class in the support set as a mean vector representation of samples in that anomaly class in an embedding space. Anomaly classification may be performed by computing a distance 230 (e.g., a Euclidean distance) between the query feature vector 222 and each of the class prototypes 228 and computing a class posterior 232 by identifying the class protype 228 having the minimum distance to the query feature vector 222. The anomaly class associated with the identified class prototype 228 with the minimum distance to the query feature vector 222 may be output as the predicted anomaly class 234 of the image 202.

[0043] An example of anomaly classification is illustrated in FIG. 4, which depicts an output 402 of the above-described prototypical network-based anomaly classifier in each case for a set of test images 202. The test images 202 each depict a manufactured part with a glue dot deposited on a part surface, similar to the example shown in FIG. 3. In the present example, anomaly classification is done on two attributes, namely, size and shape. For the first attribute, shape, the anomaly classes for prototypical learning included "elongated", "tail", "sparse", "sparse with tail" and "random". For the second attribute, size, the anomaly classes for prototypical learning included "large", "small" and "normal". The illustrated output 402 compares the predicted anomaly class label with a ground truth (GT) anomaly class label.

[0044] Anomaly classification, often in combination with anomaly segmentation, can be useful in analyzing defects in manufactured parts to determine corrective actions. Continuing with the example illustrated in FIG. 4, an anomaly classified as "elongated" or "tail" may indicate that the robotic arm holding the deposition nozzle was not moving at an optimum speed, which may have caused it to move from the deposition site before completing the deposition process, or start the deposition process before arriving at the deposition site. An anomaly classified as "sparse" may indicate bubble formation in the deposition nozzle, which may have caused the glue dot to have a different texture. The anomaly segmentation output may be used to additionally gain quantitative insight into the defects. For example, depending on the location and extent of the "tail" portion in the segmented defect, it may be determined how much faster or slower than the optimum speed the robotic arm was moving. The outputs of anomaly classification and / or anomaly segmentation may be utilized to adjust an upstream manufacturing process parameter, to prevent future occurrences of similar defects.

[0045] The embodiments of the present disclosure may be implemented with any combination of hardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non-transitory computer-readable storage medium. The computer readable storage medium has embodied therein, for instance, computer readable program instructions for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0046] The computer readable storage medium can include a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network.

[0047] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the disclosure to accomplish the same objectives. Although this disclosure has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the appended claims.

Claims

1. A computer-implemented method for training a deep learning-based vision system (200) for joint detection, localization, unsupervised segmentation and classification of anomalies in images, comprising: inputting images from a training dataset (112) to train an autoencoder (110) by backpropagation of an autoencoder loss (Lfinal) that includes an attention loss, which encourages an encoder network (114) of the autoencoder (110) to generate attention maps maximizing focus on normal regions of images, whereby, given an anomalous input image, the encoder network (114) is configured to generate an attention map that spatially localizes an anomaly, characterized in comprising unsupervised training an image segmentation model (140) by extracting feature representations (204) using the encoder network (114) from images in the training dataset (112), and alternatively optimizing cluster labels of pixels in a forward process and optimizing the feature representations (204) by backpropagation of a segmentation error (154), and tuning the encoder network (114) in a few-shot learning process for classifying anomalies by extracting feature representations (204) using the encoder network (114) from anomalous images in respective support and query sets created from the training dataset (112), using support set feature representations (204) to learn embeddings of anomaly classes, and updating the encoder network (114) based on a classification error computed using query set feature representations (204).

2. The method according to claim 1, wherein the autoencoder (110) and the image segmentation model (140) are jointly trained by updating respective parameters based on the autoencoder loss (Lfinal) and the segmentation error (154) in alternate iterations, followed by the tuning of the encoder network (114) in the few-shot learning process for classifying anomalies.

3. The method according to claim 2, comprising training the autoencoder (110) based on the autoencoder loss (Lfinal) for a number of training epochs prior to commencement of the joint training of the autoencoder (110) and the image segmentation model (140).

4. The method according to any of claims 1 to 3, wherein the training dataset (112) comprises image-level labels for normal images representing nominal manufactured parts and anomalous images representing defective manufactured parts.

5. The method according to claim 4, wherein, for training the autoencoder (110), the attention maps are computed from a feature representation (204) of a latent variable (z) generated by the encoder network (114) from normal images in the training dataset (112).

6. The method according to claim 5, wherein the autoencoder (110) comprises a binary classifier (120) connected at the output of the latent representation, wherein computing the attention maps comprises computing attention maps for normal and anomalous classes by backpropagation of gradients from a prediction of the binary classifier (120) from the normal images, and wherein the attention loss is designed to minimize an area covered by the attention map of the anomalous class and simultaneously enforce the attention map of the normal class to cover an entire image.

7. The method according to claim 6, wherein the autoencoder loss (Lfinal) further includes a binary classification loss (Lbce) computed from a prediction of the binary classifier (120) for images in the training dataset (112), to configure the binary classifier (120) for classifying an input image as normal or anomalous.

8. The method according to any of claims 4 to 7, wherein the image segmentation model (140) is trained for optimizing cluster labels of pixels based on a binary clustering, such that the image segmentation model (140) is configured to segment a defective object area from a background in an anomalous image.

9. The method according to any of claims 1 to 8, wherein training the image segmentation model (140) comprises: optimizing the cluster labels of pixels in the forward process by: extracting a feature representation (204) from an input image using the encoder network (114) where each pixel of the input image is represented by a feature vector (xn), applying a classification model to the extracted feature vectors (xn) to generate a pixel-wise response map, computing a cluster label for each pixel based on pixel-wise response map, and utilizing the cluster labels and the pixel-wise response map to compute the segmentation error (154), and backpropagating the segmentation error (154) to update parameters of the encoder network (114) and the classification model.

10. The method according to any of claims 1 to 9, wherein the few-shot learning process comprises: using the support set feature representations (204) extracted using the encoder network (114) to compute a prototype of each anomaly class in the support set as a mean vector representation of samples in that anomaly class in an embedding space, classifying samples in the query set based on computing distances to the prototype of each anomaly class in the embedding space, and therefrom computing the classification error, and backpropagating the classification error to update parameters of the encoder network (114).

11. A computer-implemented method for performing visual quality inspection of parts produced by a manufacturing process, comprising: training a deep learning-based vision system (200) by a method according to any of claims 1 to 10, acquiring an image of a manufactured part to be inspected, inputting the acquired image to the encoder network (114) to extract a feature representation (204) of the acquired image, classifying the acquired image as normal or anomalous based on the feature representation (204) of the acquired image, and if anomalous, determining an anomaly class by feeding the feature representation (204) of the acquired image as a query to a few-shot learning framework (220) that is based on embeddings learned using the encoder network (114).

12. The method according to claim 11, further comprising feeding the feature representation (204) of the acquired image to the image segmentation model (140) to segment a defective object area in the acquired image.

13. The method according to any of claims 11 and 12, further comprising adjusting a manufacturing process parameter based on the anomaly class and / or an output of the image segmentation model (140).

14. A non-transitory computer-readable storage medium encoded with instructions that, when processed by a computing system, configure the computing system to perform the method according to any of claims 1 to 10.

15. A computing system comprising: one or more processors, and non-transitory memory storing instructions executable by the one or more processors to carry out a method according to any of claims 1 to 10.