Self-supervised anomaly detection framework for visual quality inspection in manufacturing

Abstract

An AI-based method for visual inspection of a component manufactured in a workshop includes acquiring a set of real images of a nominal component manufactured in the workshop to create a training data set. A self-supervised pre-trainer module is used to pre-train the loss calculation neural network on a pre-task defined by real-world conditions related to the workshop using the first data set in a self-supervised learning process. The first data set is marked by automatically extracting pre-task related information from the image metadata. A primary anomaly trainer module is used to train the primary anomaly detection neural network to reconstruct a nominal component image from the input image of the manufacturing component using the second data set in an unsupervised learning process. The primary anomaly training measures perceptual loss between the input image and the reconstructed image by measuring differences between feature representations of the input image and the reconstructed image between one or more layers of the pre-trained loss calculation neural network.

Description

Technical Field

[0001] The present disclosure relates to a computer vision system for visual quality inspection of parts manufactured on a production line. Embodiments of the present disclosure specifically relate to an improved technique for detecting defects in manufactured parts on a production line based on anomaly detection implemented by artificial intelligence, in particular using a self-supervised anomaly detection framework. Background Art

[0002] Many modern production lines operate at high volumes, resulting in very short cycle times for human operators to effectively inspect manufactured parts for defects. Recent advances in the field of artificial intelligence (AI) have made it possible to employ deep learning models to address this challenge. Computer vision systems are commonly used in manufacturing environments in different industries such as automotive, aerospace, food and beverage, packaging, etc. These systems are typically designed and implemented as end-of-line stations, where the primary goal is to inspect and monitor the visual appearance of parts produced on the production line.

[0003] In particular, deep learning-based computer vision models, such as convolutional neural networks (CNNs), have been increasingly used in research to detect defects, such as for inspecting cemented surfaces, industrial products, etc. In particular, CNNs may be particularly suitable for detecting defects based on object detection, image segmentation, and classification. One of the reasons why deep learning-based models have made great progress in computer vision is that they can learn from large amounts of carefully labeled image data in a supervised learning process. This supervised learning paradigm has a good track record for training specialized models that can perform extremely well in the tasks they are trained to complete.

[0004] However, there are limitations on the suitability of deep learning based computer vision models for detecting defects in manufacturing environments using supervised learning alone.

[0005] First, in an industrial setting, defects may occur infrequently, such as once or twice during an entire shift. Furthermore, "defective" image data must be manually labeled (e.g., via bounding boxes or segmentation), typically by one or more domain experts on-site. Therefore, creating a sufficiently large data set consisting of images of defective parts to robustly train these models can involve an extremely long data collection period (e.g., about six months) and can also be labor intensive, which can generate additional workload and can change process flows that can be problematic for traditional manufacturers. Manual labeling is also subjective and may not be consistent over large data sets and long data collection periods.

[0006] Secondly, while supervised learning methods can capture known defects and defects located in similar areas on a part well, they cannot be used to accurately detect any unknown type of defect or known defects that appear in different areas of a part. This is again related to the variance of the training data mentioned above. In addition, in industrial vision applications, the intra-class (defect) variance is typically large as well. There may be multiple subclasses under one main defect category. For example, a defect class related to subassemblies of manufactured parts may have subclasses such as a) missing subassemblies, b) improper placement of subassemblies, and c) incorrect welding of subassemblies. In the absence of representative data during model training, the accuracy of these supervised models tends to be low.

[0007] Considering the current status of AI research and the complexity of industrial applications, there is a demand for a new and domain-agnostic solution. Summary of the invention

[0008] In short, aspects of the present disclosure provide an improved technique for detecting defects in manufactured parts on a shop floor using a self-supervised anomaly detection framework based on artificial intelligence-enabled anomaly detection.

[0009] A first aspect of the present disclosure provides a method for performing artificial intelligence-based visual quality inspection on a component manufactured in a workshop. The method includes acquiring a set of real images of a nominal component manufactured in a workshop. The method also includes executing a self-supervised pre-trainer module to pre-train a loss calculation neural network using a first data set created from the acquired set of real images in a self-supervised learning process. The loss calculation neural network is pre-trained on a pre-task (English: pretext) defined by real-world conditions related to the workshop, and the first data set is labeled by automatically extracting pre-task related information from image metadata. The method also includes executing a main anomaly trainer module to train a main anomaly detection neural network to reconstruct a nominal component image from an input manufactured component image using a second data set created from the acquired set of real images in an unsupervised learning process. The unsupervised learning process includes using a main anomaly detection neural network to process an input image from a second data set to output a corresponding reconstructed image, and measuring a reconstruction loss to be minimized from the corresponding reconstructed image. The reconstruction loss includes a perceptual loss measured by feeding each input image and the corresponding reconstructed image to a pre-trained loss computation neural network and computing a metric of the difference between feature representations of the input image and the corresponding reconstructed image at one or more layers of the pre-trained loss computation neural network.

[0010] Other aspects of the present disclosure implement the features of the above methods in the form of computer program products and computing systems.

[0011] Additional technical features and benefits can be achieved through the technology of the present disclosure. The embodiments and aspects of the present disclosure are described in detail herein and are considered to be part of the claimed subject matter. For a better understanding, please refer to the detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The foregoing and other aspects of the present disclosure will be best understood from the following detailed description when read in conjunction with the accompanying drawings. To facilitate identification of the discussion of any element or action, one or more of the most significant digits in the reference numerals refer to the figure number in which the element or action is first introduced. For clarity, certain images herein are schematically represented as line drawings.

[0013] Figure 1 is a schematic diagram illustrating a pipeline for training an anomaly detection neural network for visual quality inspection of manufactured parts on a shop floor, according to one embodiment of the present disclosure.

[0014] Figure 2 Schematically illustrates the extraction of image patches from images of manufactured parts to create data sets for pre-trainer modules and main trainer modules according to disclosed embodiments.

[0015] Figure 3 An exemplary embodiment of a system for visual quality inspection of manufactured parts on a shop floor using an anomaly detector module including a trained anomaly detection neural network is illustrated.

[0016] Figure 4 An example of generating a loss map from an acquired image by an anomaly detector module is schematically illustrated.

[0017] Figure 5 An exemplary visualization of the output of an anomaly detector module is schematically illustrated.

[0018] Figure 6 A computing system that can support visual quality inspection of manufactured parts on a shop floor in accordance with disclosed embodiments is illustrated. DETAILED DESCRIPTION

[0019] In data science, "anomaly detection" refers to the process of identifying unexpected items or events in a data set that differ from the norm. Anomaly detection is typically based on the assumption that anomalies rarely occur in the data and that their characteristics are significantly different from normal or "nominal" instances. The disclosed method can be used to train a neural network to detect defects in manufactured parts on a production line from acquired images of the manufactured parts by detecting anomalies. Such neural networks are referred to herein as anomaly detection neural networks. The disclosed method can avoid the need for "defective" data during training because model training can be achieved using only nominal (i.e., non-defective) data that is available in large quantities. Therefore, the trained neural network can detect any defects, such as cracks, cracks, scratches, missing components, improperly assembled components, etc., without having to be trained with knowledge of any specific type of defect.

[0020] From experiments with multiple industrial use cases, the inventors realized that anomaly detection models can be very sensitive to environmental and operational changes such as lighting changes, changes in orientation (rotation) of manufactured parts, adaptation to unseen parts, etc. The disclosed method can address the above challenges by using a self-supervised anomaly detection framework, thereby providing a robust, reliable, and generalizable solution for anomaly detection for industrial use cases.

[0021] According to the disclosed embodiment, the self-supervised anomaly detection framework pre-trains the loss calculation neural network to learn the feature representation of the image data based on the pre-task defined by the real-world conditions related to the workshop (e.g., environmental conditions, operating conditions, camera conditions, etc.) in the self-supervised process. Creating the pre-task based on the real-world conditions of the workshop can ensure that the learned representation is robust to the different conditions of the workshop. The self-supervised learning process can be performed based on the image labels generated by automatically extracting the information related to the pre-task from the image metadata. The pre-trained loss calculation neural network is used in the downstream task of training the main anomaly detection neural network for calculating the perceptual loss between the input image and the corresponding reconstructed image by measuring the difference between the feature representations of the input image and the corresponding reconstructed image at one or more layers of the loss calculation neural network. The perceptual loss can form part of the reconstruction loss function to be minimized through the process of training the main anomaly detection neural network.

[0022] Turning now to the disclosed embodiments, Figure 1An exemplary pipeline 100 for training an anomaly detection neural network for visual quality inspection of manufactured parts in a workshop is illustrated. The various modules described in the present invention, such as an image block generator module 104, a self-supervised data set generator module 108, a self-supervised pre-trainer module 112, an anomaly pre-trainer module 118, and a master anomaly trainer module 122, including their components, can be implemented by a computing system in various ways, such as hardware and programming. The programming for modules 104, 108, 112, 118, 122 can take the form of processor executable instructions stored on a non-temporary machine-readable storage medium, and the hardware for the engine can include a processor that executes these instructions. In an embodiment, the computing system can appropriately include a workstation with multiple graphics processing units (GPUs).

[0023] refer to Figure 1 , the disclosed method involves acquiring a set of real images 102 of actual parts manufactured in a workshop, for example, via a production line camera in the workshop. The images 102 may only be related to nominal (i.e., defect-free) manufactured parts. The acquired set of real images 102 can be used to create a training data set, for example, as shown below. The images 102 can suitably cover a wide range of real-world conditions in a workshop.

[0024] Many industrial applications require high-resolution cameras to capture small defects or flaws. For example, the camera resolution can be set at 3K for imaging die-cast parts, 16K for imaging stamped parts, 8K for imaging motherboards, etc. Most advanced AI-based vision models are not designed to take such high-resolution images as input. This is due to the receptive field and kernel filter size defined in the convolutional layers of the neural network. One solution is to downsize the image to a more easily processable resolution. However, downsizing may also cause the "defective" pixels to disappear.

[0025] According to the disclosed embodiment, the patch generator module 104 may be used to extract image patches from individual whole images in the acquired set of real images 102. The patch generator module 104 may segment the high-resolution images into smaller images or image patches (with a reduced number of pixels) that may be processed by a state-of-the-art neural network. According to the embodiment, the image patches extracted by the patch generator module 104 may be used to create a training data set for the pre-trainer modules 112, 118 and the main anomaly trainer module 122. In some embodiments, depending on the resolution of the acquired images 102 and / or the architecture of the neural network, the patch generator module 104 may not be required and the entire images 102 may be used to create the training data set.

[0026] The image patch generator module 104 can apply one or more different modalities for extracting image patches from an image. According to the exemplary embodiments disclosed herein, image patches can be extracted as ordered position embedding, overlapping position embedding, and random position embedding. In each case, the number and size of image patches can be defined based on user-specified parameters. Figure 2 The application of these modalities is illustrated on an image 102 of a part being manufactured in a shop. Figure 2 The images depicted in are merely illustrative (ie, not actual images) and are schematically represented as line graphs for the sake of clarity.

[0027] refer to Figure 2 According to the first modality (A), a set of ordered or non-overlapping image blocks can be extracted from the entire image 102 by dividing the image 102 based on a specified number of rows and columns (i.e., the number of image blocks along the length and width of the image). The extracted image blocks 202 are referred to as ordered position embeddings. According to the second modality (B), a set of overlapping image blocks can be extracted from the entire image 102 by defining a sliding window based on a specified image block size (pixel size) and a degree of overlap (e.g., a percentage or number of pixels). The extracted image blocks 204 are referred to as overlapping position embeddings. The overlap between adjacent image blocks 204 is named 206. The degree of overlap can be specified along two directions (X-axis and Y-axis). According to the third modality (C), a set of random image blocks can be extracted from the entire image 102 based on a specified image block size (pixel size) and a number of image blocks. Image blocks 208, referred to as random position embeddings, can be randomly extracted with or without overlap with other image blocks.

[0028] Continue to refer Figure 1, the image patch generator module 104 can therefore be used to extract one or more groups of image patches corresponding to one or more modalities (e.g., as shown above) for each image 102. The extracted image patches can be provided as input to the pre-trainer module and the main trainer module. For example, ordered position embedding and overlapping position embedding can be used as images in the corresponding data sets of the self-supervised pre-trainer module 112 and the main anomaly trainer module 122. Using both ordered image patches and overlapping image patches in the data set for the self-supervised pre-trainer module 112 and the main anomaly trainer module 122 can provide an increase in the robustness of the trained model. In some embodiments, only ordered position embedding can be used as images in the data set for the self-supervised pre-trainer module 112 and the main anomaly trainer module 122, in which case it may not be necessary to extract the modality of the overlapping position embedding. Random position embedding can be used as images in the data set of the anomaly pre-trainer module 118. In an embodiment where the anomaly pre-trainer module 118 is not used, it may not be necessary to extract the modality of the random position embedding in the image patch generator module 104. The image patches in each of the data groups of the self-supervised pre-trainer module 112, the anomaly pre-trainer module 118, and the master anomaly trainer module 122 may be derived from real images of different subgroups in the acquired set of real images 102. The overlap between these subgroups may be appropriately determined to avoid overfitting of the model.

[0029] Consistent with the disclosed embodiments, the self-supervised dataset generator module 108 can create a self-supervised dataset 110 using the ordered and overlapping position embeddings 106 extracted from the first subset of the acquired set of real images 102. To create the dataset 110, the self-supervised dataset generator module 108 can automatically generate image labels by extracting information from image metadata based on a prior task associated with self-supervised pre-training.

[0030] In short, self-supervised learning is a machine learning method that can involve using self-supervision in a pre-training process to learn useful representations of data from an unlabeled data pool, with the goal of subsequently fine-tuning these representations in downstream tasks that can be supervised or unsupervised. In the present application, the downstream tasks include unsupervised anomaly detection tasks. In self-supervised learning, the tasks used for pre-training are called "pre-tasks". An important aspect of self-supervised learning is the establishment of pre-tasks. In current research, the most common methods for establishing pre-tasks include augmenting the unlabeled data pool using methods such as coloring, changing the position of image patches, repairing, adding damaged images, adding synthetic rotations to images, etc. However, it should be recognized that in industrial applications, and particularly in the context of anomaly detection, these augmentation techniques cannot provide useful optimization properties during model pre-training.

[0031] The disclosed method relates to an inventive technique for establishing pre-tasks defined by real-world conditions relevant to a plant floor for self-supervised pre-training. This is in contrast to the augmentation techniques mentioned above that are synthetically generated after data collection. Thus, a pre-training task may include inputting an image (in this case, an image block 106) to a neural network to infer the real-world conditions of the plant floor associated with the image using a self-supervised data set 110. In order to learn subtle differences, the objectives of the self-supervised training may be made more complex, for example, by selecting multiple "real-world" pre-tasks that may be relevant in the context of a specific industrial use case. The real-world conditions used to define the pre-tasks may include one or more environmental conditions, operating conditions, camera conditions, or preferably any combination thereof.

[0032] Since the nature of the end-of-line inspection station is dynamic, for example, involving changing machine conditions, ambient lighting conditions, different parts, different trigger positions, etc., such conditions can be exploited as pre-tasks for model pre-training. Taking the first exemplary use case of inspecting parts manufactured by stamping as an example, a table of pre-tasks can be visualized (see Table 1), including three real-world condition categories: (1) environmental conditions, (2) camera conditions, and (3) operating conditions. Each of these categories can include multiple pre-tasks with assigned difficulty scores. The number of subtasks indicates the number of possible outcomes associated with a given pre-task.

[0033] Table 1

[0034]

[0035]

[0036] It is important to note that the difficulty score may indicate the level of complexity of the defined pre-task. Here, a difficulty score of 5 indicates the highest complexity, while a difficulty score of 1 indicates the lowest complexity. For example, the pre-task “Ambient Lighting” is assigned a difficulty score of 5 because it involves a highly complex task, since the mechanical system may be designed to block / reduce any ambient lighting reflections. Therefore, it may be assumed that the variation from morning to afternoon should be minimal. Therefore, it would be a challenging task for the pre-trained model to capture the differences between samples from different shifts. On the other hand, the pre-task “Camera ID” is assigned a difficulty score of 1 because it involves a simple task, since each camera sees a different area of ​​the part with a certain degree of overlap (e.g., 20%).

[0037] Taking the second exemplary use case of inspecting a manufacturing motherboard as an example, a table of predecessor tasks can be visualized (see Table 2), including two real-world condition categories: (1) environmental conditions and (2) operating conditions. Each of these categories can similarly include multiple predecessor tasks with assigned difficulty scores. The number of subtasks indicates the number of possible outcomes associated with a given predecessor task.

[0038] Table 2

[0039]

[0040] In one embodiment, the objectives of the self-supervised pre-trainer module 112 can be generated using a combination of predecessor tasks, where the predecessor tasks can be selected from one or more real-world condition categories (e.g., environmental conditions and / or operating conditions and / or camera conditions related to the workshop). Referring to the use case shown in Table 1, an exemplary pre-training objective may involve tasks of inferring "ambient lighting", "exposure value", and "mold condition" associated with an input image or image block 106 (a total of 3x2x2=12 inference results). The combination of multiple predecessor tasks can cause the feature space in the pre-trained model to be more compact. The self-supervised data set generator module 108 can be configured to extract relevant predecessor task related information from the image metadata of the image / image block 106 to generate an image label. It should be noted that the image label for each image block 106 extracted from a single real image 102 will be the same because the metadata of a given image 102 is common to all image blocks 106 extracted from that image 102.

[0041] In another embodiment, the goal of the self-supervised pre-trainer module 112 can be generated (manually or automatically) by combining predecessor tasks based on a specified overall difficulty score. In an exemplary implementation, based on the overall difficulty score specified by the user, a graph-based algorithm can be employed to construct the self-supervised data set 110 by automatically selecting a combination of predecessor tasks that will satisfy the user input.

[0042] A self-supervised pre-trainer module 112 may be executed to pre-train a loss calculation neural network 114 using a self-supervised data set 110 including labels generated by previous task information. As described above, the goal of pre-training may be based on multiple previous tasks. For example, the pre-training goal may involve a task of inferring a combination of real-world conditions of a workshop associated with an input image or image patch. As shown, the loss calculation neural network 114 may appropriately include multiple convolutional layers, which, when pre-trained, may be configured to map multiple levels of feature representations of an input image or image patch by learning to encode perceptual and semantic information. The loss calculation neural network 114 may include, for example, a state-of-the-art architecture. In a non-limiting embodiment, the loss calculation neural network 114 may be implemented by a deep residual neural network architecture, such as Resnet50. The output of the self-supervised pre-trainer module 112 may include a pre-trained (and verified) loss calculation neural network 130 that may then be utilized by the main anomaly trainer module 122.

[0043] As an additional feature, it may be beneficial to introduce the ability of transfer learning into the anomaly detection framework. Transfer learning involves taking a pre-trained neural network and adapting it to a new task. However, due to the architecture of the anomaly detection neural network, there are no available pre-trained models that can be used directly in the main anomaly trainer module 122. One solution to the above problem may involve using an anomaly pre-trainer module 118 according to another embodiment of the disclosed method.

[0044] The anomaly pre-trainer module 118 may be executed to pre-train a candidate anomaly detection neural network 120 from scratch (i.e., without prior initialization of weights) using a data set comprising random image patches 116 (random position embeddings) extracted from a second subset of the acquired set of real images 102. The candidate anomaly detection neural network 120 may have the same architecture as the main anomaly detection neural network 124 to be trained subsequently.

[0045] In the disclosed embodiment, the candidate anomaly detection neural network 120 and the main anomaly detection neural network 124 have (the same) autoencoder architecture, which is particularly suitable for learning encodings of unlabeled data, such as in anomaly detection. The autoencoder architecture may include an encoder E and a decoder D, the former for encoding or compressing the input data into low-dimensional data (represented in a latent space LS), and the latter for mapping the low-dimensional data into a reconstruction of the input data. The encoder E and / or the decoder D may include convolutional layers. The architectures described herein are exemplary, and the neural networks 120, 124 may include other architectures, such as a visual transformer including a multi-layer perception, etc.

[0046] The anomaly pre-trainer module 118 may perform an unsupervised learning process that involves the following tasks: inputting an unlabeled random image patch 116 and outputting a reconstruction of the input random image patch 116 by minimizing a reconstruction loss that measures the difference between the input image and the reconstructed image. The candidate anomaly detection neural network 120 may be pre-trained using thousands of random image patches 116, whereby the model may learn some well-defined characteristics of the surface of a manufactured part, such as holes, edges, bend points, curvature, lighting variations, etc. After pre-training of the candidate anomaly detection neural network 120, the master anomaly detection neural network 124 may be initialized using the weights of the pre-trained candidate anomaly detection neural network 120 before executing the master anomaly trainer module 122.

[0047] Using the weights of the pre-trained candidate anomaly detection neural network to start the main anomaly training loop can provide a better view of the losses in the main anomaly training loop. The initialized weights can provide a better starting point for the main anomaly detection neural network 124, allowing it to perform tasks with higher accuracy without prior training. Since the candidate anomaly detection neural network 120 is trained on similar problems based on reconstructing random image patches extracted from multiple different parts, this approach can provide a higher learning rate during the main anomaly training loop. As another feature, due to the random image patch extraction, the main anomaly detection neural network 124 becomes more robust to changes in the production line (such as part rotation, part being moved to another location on the conveyor system, etc.).

[0048] The goal of the primary anomaly trainer module 122 is to train the primary anomaly detection neural network 124 to reconstruct a nominal part image from an image of an input manufactured part acquired from a shop floor. Consistent with the disclosed embodiments, the primary anomaly trainer module 122 may use a data set that includes ordered and overlapping position embeddings 126 extracted from a third subset of the acquired set of real images 102. Thus, the ordered and overlapping position embeddings 126 may define an input image in a primary anomaly training loop. In one embodiment, the primary anomaly detection neural network 124 may be initialized using the weights of the pre-trained candidate anomaly detection neural network 120, as described above.

[0049] The primary anomaly trainer module 122 may perform an unsupervised learning process involving the following tasks: using the primary anomaly detection neural network 124 for processing an input image 126 to output a corresponding reconstructed image 128, and measuring a reconstruction loss to be minimized from the corresponding reconstructed image, the reconstruction loss being an overall measure of the difference between the input image 126 and the reconstructed image 128. The reconstruction loss may include a perceptual loss that may be measured by feeding each input image 126 and the corresponding reconstructed image 128 to a pre-trained loss computation neural network 130, and computing a measure of the difference between feature representations of the input image 126 and the corresponding reconstructed image 128 at one or more layers of the pre-trained loss computation neural network 130.

[0050] The concept of perceptual loss has been studied in applications such as image restoration, image super-resolution, denoising, and colorization, where the input is a degraded image and the output is a high-quality color image. This includes an optimization technique that has been used for generating images, where the goal is perceptual, depending on high-level features extracted from convolutional neural networks. For example, Johnson et al. discuss the concept of perceptual loss in the following publication: Johnson, Justin, Alexandre Alahi, and Li Fei-Fei, "Perceptual losses for real-time style transfer and super-resolution", European Conference on Computer Vision, Springer, Cham, 2016.

[0051] Thus, the disclosed method provides an inventive technique that adapts the concept of perceptual loss to an anomaly detection framework for industrial applications to provide accurate reconstruction of images of nominal parts. It is assumed that the output of the self-supervised pre-trainer module 108 (i.e., the pre-trained loss calculation neural network 130) has learned to encode perceptual and semantic information that will be useful for measurement in the reconstruction loss function. Thus, the pre-trained loss calculation neural network 130 can be used to define a perceptual loss that measures the content difference between the input image 126 and the reconstructed image 128.

[0052] In one embodiment, the perceptual loss at a given depth (i.e., layer) of a pre-trained loss calculation neural network 130 may be calculated by accessing the activations of the layer corresponding to the input image 126 and the reconstructed image 128, and measuring the Euclidean distance between feature representations of the input image and the reconstructed image as defined by the respective activations produced by the input image 126 and the reconstructed image 128 at the layer (e.g., as described by Johnson et al. in the above-mentioned publication).

[0053] In one embodiment, the perceptual loss may be measured by combining contributions from multiple levels of feature representations at multiple convolutional layers of the pre-trained loss calculation neural network 130, respectively. This operation may cause the reconstructed image 128 to be perceptually similar to the input image 126, but does not force them to match exactly. For example, the contribution at each layer may be calculated as described above. In the disclosed embodiment, these contributions include a first contribution P1 associated with a low-level feature representation, a second contribution P2 associated with a medium-level feature representation, and a third contribution P3 associated with a high-level feature representation. In this example, the perceptual loss may be calculated as the sum of P1, P2, and P3. In various embodiments, contributions may be measured from additional or different layers and combined in different ways (e.g., as a sum, or a weighted sum) to calculate the perceptual loss.

[0054] To improve the accuracy of the reconstruction, the reconstruction loss may be measured by combining a perceptual loss with one or more per-pixel loss metrics between the reconstructed image 128 and the input image 126. The per-pixel loss metric may be appropriately determined to force the pixels of the reconstructed image 128 to exactly match the pixels of the input image 126. According to the disclosed embodiment, the per-pixel loss metric may include a combination (e.g., a sum) of an L1 loss metric, an L2 loss metric, and a structural similarity index metric (SSIM) of the primary anomaly detection neural network 124. The reconstruction loss may be determined as, for example, a weighted combination (e.g., a sum) of the perceptual loss and the per-pixel loss.

[0055] In one embodiment, the main anomaly training loop may involve dividing the training data set into multiple batches of images 126 and measuring the reconstruction loss for each batch of images 126. The main anomaly training loop may proceed by repeatedly adjusting the weights of the main anomaly detection neural network 124 after processing a batch of images 126 until the reconstruction loss is minimized (e.g., based on a gradient descent method). As described below with reference to Figures 3 to 5As shown, a trained (and validated) primary anomaly detection neural network can be deployed on a production line to detect defects on manufactured parts by identifying anomalies based on a comparison between an input image of the manufactured part (e.g., acquired by a production line camera) and a reconstructed nominal part image.

[0056] Figure 3 An exemplary embodiment of a system 300 for visual quality inspection of manufactured parts on a shop floor is shown. The various modules described herein, such as an image block generator module 308 and an anomaly detector module 310, including their components, can be implemented in various ways by a computing system (e.g., an industrial PC, an edge device, etc.), such as hardware and programming. The programming for modules 308, 310 can take the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware for the engine can include a processor that executes these instructions. In one suitable implementation, such a computing system can include an industrial PC, an edge device, or another type of computing device, which can be equipped with one or more processors, such as a GPU or a deep learning processor, such as a neural processing unit (NPU), so as to run the deployed anomaly detection neural network in a computationally efficient manner.

[0057] refer to Figure 3 , the system 300 may include a camera 304 positioned to acquire an image 306 of a manufactured component on a production line 302. The production line 302 may include a queue of manufactured components positioned on a conveyor system. In an example, the manufactured components may include die cast components, stamped metal components, motherboards, etc. During operation of the production line 302, the camera 304 may be used to acquire an image 306 for each individual manufactured component on the production line 302, which may be used to provide a real-time input image to an anomaly detector module 310 including a trained primary anomaly detection neural network. The real-time input image may be processed by the anomaly detector module 310 using the trained primary anomaly detection neural network to reconstruct a nominal component image. As described below, the anomaly detector module 310 may detect anomalies based on measuring differences between the real-time input image and the reconstructed nominal component image.

[0058] In the disclosed embodiment, the system 300 suitably includes an image patch generator module 308 that can be executed to extract image patches from each acquired image 306. The extracted image patches for the individual images 306 may include, for example, ordered position embeddings extracted based on a specified number of rows and columns, such as Figure 2As shown. Thus, in the disclosed embodiments, each real-time input image for the anomaly detector module 310 includes an image patch extracted from the acquired image 306, which can be processed by the trained master anomaly detection neural network to reconstruct a corresponding nominal component image corresponding to the image patch. In certain embodiments, depending on the resolution of the acquired image 306 and / or the structure of the master anomaly detection neural network, the real-time input image for the anomaly detector module 310 can include the entire image 306 acquired by the camera 304.

[0059] Figure 4 An example of an image block 402 extracted from an image 306 of a manufactured component (e.g., a die cast component) acquired by a camera 304 is shown. The image shown is merely illustrative (i.e., not a real image) and is schematically represented as a line graph for clarity. As shown, the extracted image block 402 includes defects 404a, 404b, and 404c. Since the main anomaly detection neural network is trained only on images of nominal (defect-free) manufactured components, any defects (such as 404a, 404b, and 404c) that appear on the extracted image block 402 will not be recognized by the main anomaly detection neural network. Therefore, the reconstructed nominal component image excludes such defects (because they are outside the normal distribution). Defects in the acquired image 306 can be revealed by the following steps: each extracted image block (e.g., image block 402) of the acquired image 306 is compared with the corresponding reconstructed nominal component image to detect anomalies.

[0060] In one embodiment, anomaly detector module 310 may determine pixel-level locations of defects based on a loss map defined by differences between extracted image patches of the acquired image and a reconstructed nominal part image. Figure 4 An example of a loss map 406 corresponding to the extracted image block 402 is shown. According to an exemplary technique, the loss map 406 can be generated by determining a pixel-level anomaly score based on a reconstruction error between the extracted image block 402 and a reconstructed nominal image corresponding to the image block 402, with a higher score indicating a greater likelihood of anomaly. Optionally, the residual map can be smoothed by filtering. Subsequently, binary segmentation can be performed to identify the location of the anomaly. Figure 4 In the example shown, anomalies 408a, 408b, and 408c in loss map 406 indicate the locations of defects 404a, 404b, and 404c, respectively.

[0061] The anomaly detector module 310 can determine the location of the identified defects on the actual acquired image 306 by superimposing each loss map onto a corresponding image patch of the image 306, stitching the extracted image patches to recreate the entire image, and drawing a bounding box around the determined defect location (e.g., based on the loss map) in the entire image. Figure 5 An exemplary visualization of the output 500 of the anomaly detector module 310 is shown. The image shown is illustrative, which is schematically represented as a line graph. In this document, defective areas as synthetically generated by the anomaly detector module 310 are located by bounding boxes 502. Figure 3 The output image 312 including the identified defect-related information (including the presence and location of the identified defect) can then be visualized (e.g., as Figure 5 ) and is displayed to an operator via a user interface 314 (such as an HMI device, etc.).

[0062] Thus, the disclosed embodiments provide a solution for automatically and reliably detecting defects in manufactured parts at an early stage of production. Based on the output of the anomaly detector module 310, appropriate control actions may be performed. Examples of control actions may include isolating defective parts from production, shutting down production to avoid catastrophic events (e.g., a defect is detected in a continuous stream of molded parts), and the like. In response to the output of the anomaly detector module 310, and in particular, a positive defect detection in one or more manufactured parts in the production line 302, control actions may be performed automatically, semi-automatically, or manually. For example, in one embodiment, as Figure 3 As shown, the system 300 may include a field controller 316 for automatically executing control actions based on the output of the anomaly detector module 310 to control the production line 302. In an embodiment, the field controller 316 may include or be integrated with an automation system device for executing control actions, such as a PLC (programmable logic controller), a drive, etc.

[0063] Figure 6 An example of a computing system 600 according to the disclosed embodiments is shown, which can support visual quality inspection of manufactured parts on a shop floor. In the example, the computing system 600 can be configured as a powerful multi-GPU workstation as well as other types of computing devices. The computing system 600 includes at least one processor 610, which can take the form of a single or multiple processors. The processor(s) 610 may include one or more CPUs, GPUs, microprocessors, or any hardware device suitable for executing instructions stored on a memory including a machine-readable medium. The computing system 600 also includes a machine-readable medium 620. As shown in FIG. Figure 6As shown, the machine-readable medium 620 may take the form of any non-transitory electronic, magnetic, optical or other physical storage device that stores executable instructions, such as image block generation instructions 622, self-supervised pre-training instructions 624, anomaly pre-training 626, and master anomaly training instructions 628. Therefore, the machine-readable medium 620 may be, for example, a random access memory (RAM) such as a dynamic RAM (DRAM), a flash memory, a spin transfer torque memory, an electrically erasable programmable read-only memory (EEPROM), a storage drive, an optical disk, etc.

[0064] The computing system 600 may execute instructions stored on the machine-readable medium 620 via the processor(s) 610. Executing the instructions (e.g., image patch generation instructions 622, self-supervised pre-training instructions 624, anomaly pre-training 626, and master anomaly training instructions 628) may cause the computing system 600 to perform any of the technical features described herein, including any of the features according to the image patch generator module 104, the self-supervised pre-trainer module 112, the anomaly pre-trainer module 118, and the master anomaly trainer module 122 described above.

[0065] The above-described systems, methods, devices, and logic, including the image block generator module 104, the self-supervised pre-trainer module 112, the anomaly pre-trainer module 118, and the master anomaly trainer module 122, can be implemented in many different ways in many different combinations of hardware, logic, circuits, and executable instructions stored on machine-readable media. For example, these engines may include circuits in a controller, a microprocessor, or an application-specific integrated circuit (ASIC), or may be implemented with a combination of discrete logic or components or other types of analog or digital circuits combined on a single integrated circuit or distributed between multiple integrated circuits. A product such as a computer program product may include a storage medium and machine-readable instructions stored on the medium, which, when executed in a terminal, a computer system, or other device, causes the device to perform operations according to any of the above descriptions, including performing operations according to any features of the image block generator module 104, the self-supervised pre-trainer module 112, the anomaly pre-trainer module 118, and the master anomaly trainer module 122. The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (eg, the Internet, a local area network, a wide area network, and / or a wireless network).

[0066] The processing capabilities of the systems, devices, and engines described herein, including the image patch generator module 104, the self-supervised pre-trainer module 112, the anomaly pre-trainer module 118, and the master anomaly trainer module 122, can be distributed among multiple system components, such as between multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements. Parameters, databases, and other data structures can be stored and managed separately, can be incorporated into a single memory or database, can be logically and physically organized in many different ways, and can be implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs can be parts of a single program (e.g., subroutines), separate programs, can be distributed across multiple memories and processors, or can be implemented in many different ways, such as in the form of libraries (e.g., shared libraries).

[0067] Although the present disclosure has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the current design may be implemented by those skilled in the art without departing from the scope of the patent claims.

Claims

1. A method for performing artificial intelligence-based visual quality inspection of a component manufactured in a workshop, comprising: Obtain a set of real images of nominal parts manufactured in said workshop, executing a self-supervised pre-trainer module to pre-train a loss computation neural network in a self-supervised learning process using a first data set created from the acquired set of real images, wherein the loss computation neural network is pre-trained on a predecessor task defined by real-world conditions associated with the plant, the first data set being labeled by automatically extracting predecessor task related information from image metadata, and executing a primary anomaly trainer module to train a primary anomaly detection neural network to reconstruct a nominal part image from an input image of a manufactured part using a second data set created from the acquired set of real images in an unsupervised learning process, wherein the unsupervised learning process comprises using the master anomaly detection neural network for processing input images from the second data set to output corresponding reconstructed images, and measuring a reconstruction loss to be minimized from the corresponding reconstructed images, and Wherein, the reconstruction loss comprises a perceptual loss measured by feeding each input image and the corresponding reconstructed image to a pre-trained loss calculation neural network and calculating a metric of the difference between feature representations of the input image and the corresponding reconstructed image at one or more layers of the pre-trained loss calculation neural network.

2. The method according to claim 1, wherein: The objectives of the self-supervised pre-trainer module are generated using a combination of predecessor tasks, which are defined by real-world conditions including environmental conditions and / or operating conditions and / or camera conditions related to the plant.

3. The method according to claim 2, wherein: The predecessor tasks are assigned respective difficulty scores, and the objective of the self-supervised pre-trainer module is generated by combining the predecessor tasks based on the assigned overall difficulty score.

4. The method according to any one of claims 1 to 3, wherein: The loss computation neural network comprises a plurality of convolutional layers, and wherein the perceptual loss is measured by combining contributions of feature representations from a plurality of levels respectively at the plurality of convolutional layers of the pre-trained loss computation neural network.

5. The method according to any one of claims 1 to 4, wherein: The master anomaly detection neural network includes an autoencoder architecture.

6. The method according to any one of claims 1 to 5, wherein: The reconstruction loss is measured by combining the perceptual loss with one or more per-pixel loss metrics between the reconstructed image and the input image.

7. The method according to claim 6, wherein: The one or more per-pixel loss metrics include a combination of an L1 loss metric, an L2 loss metric, and a structural similarity index metric (SSIM) of the primary anomaly detection neural network.

8. The method according to any one of claims 1 to 7, further comprising executing an image patch generator module to extract image patches from individual whole images in the acquired set of real images, in, The extracted image patches include ordered position embeddings extracted from a separate whole image based on a specified number of rows and columns into which the whole image is to be divided, and Wherein, the ordered position embeddings are used as images in corresponding data sets for pre-training the loss calculation neural network and training the main anomaly detection neural network.

9. The method according to claim 8, wherein: The extracted image patches also include overlapping position embeddings extracted from separate whole images based on specified image patch sizes and overlap levels, and wherein the ordered position embeddings and the overlapping position embeddings are used as images in the corresponding data set for pre-training the loss computation neural network and training the primary anomaly detection neural network.

10. The method according to any one of claims 8 and 9, wherein: The extracted image patches also include randomly located embeddings extracted from a separate whole image based on a specified image patch size and a specified number of image patches, wherein the method further comprises: executing an anomaly pre-trainer module to pre-train a candidate anomaly detection neural network having the same architecture as the master anomaly detection neural network using a third data set including the extracted random position embeddings in an unsupervised learning process, and Before executing the primary anomaly trainer module to train the primary anomaly detection neural network, the primary anomaly detection neural network is initialized using weights of pre-trained candidate anomaly detection neural networks.

11. The method according to any one of claims 1 to 10, comprising detecting defects in manufactured parts of the workshop by the following steps: acquiring an image of the manufactured part via a production line camera, providing a real-time input image to an anomaly detector module including a trained primary anomaly detection neural network based on the acquired image of the manufactured part, processing the real-time input image by the anomaly detector module using the trained primary anomaly detection neural network to reconstruct a nominal part image, and Anomalies are detected by the anomaly detector module based on measuring differences between the real-time input image and a reconstructed nominal part image.

12. The method of claim 11, comprising determining a pixel-level location of a defect based on a loss map between the real-time input image and the reconstructed nominal part image.

13. A non-transitory computer-readable storage medium comprising instructions which, when processed by a computing system, configure the computing system to perform the method according to any one of claims 1 to 12.

14. A computing system for supporting artificial intelligence-based visual quality inspection of parts manufactured in a shop floor, comprising: one or more processors, and A memory storing an algorithm module executable by the one or more processors, wherein the algorithm module comprises: a self-supervised pre-trainer module configured to pre-train a loss computation neural network in a self-supervised learning process using a first data set created from a set of acquired real images of nominal parts manufactured at the shop, wherein the loss computation neural network is pre-trained on a prior task defined by real-world conditions associated with the shop, the first data set being labeled by automatically extracting prior task related information from image metadata, and a primary anomaly trainer module configured to train a primary anomaly detection neural network to reconstruct a nominal part image from an input image of a manufactured part using a second data set created from a set of real images acquired of a nominal part manufactured at the shop in an unsupervised learning process, wherein the primary anomaly trainer module is configured to use the primary anomaly detection neural network for processing input images from the second data set to output corresponding reconstructed images, and to measure a reconstruction loss to be minimized from the corresponding reconstructed images, and Wherein, the reconstruction loss comprises a perceptual loss measured by feeding each input image and the corresponding reconstructed image to a pre-trained loss calculation neural network and calculating a metric of the difference between feature representations of the input image and the corresponding reconstructed image at one or more layers of the pre-trained loss calculation neural network.

15. The computing system of claim 14, wherein: The self-supervised pre-trainer module is configured to pre-train the loss calculation neural network based on targets generated using a combination of predecessor tasks, wherein the combination of predecessor tasks is defined by real-world conditions including environmental conditions and / or operating conditions and / or camera conditions related to the workshop.

16. The computing system according to any one of claims 14 and 15, wherein: The loss computation neural network comprises a convolutional layer, and wherein the primary anomaly trainer module is configured to measure the perceptual loss by combining contributions from multiple levels of feature representations at multiple convolutional layers, respectively, of the pre-trained loss computation neural network.

17. A computing system according to any one of claims 14 to 16, wherein: The primary anomaly trainer module is configured to measure the reconstruction loss by combining the perceptual loss with one or more per-pixel loss metrics between the reconstructed image and the input image.

18. The computing system according to any one of claims 14 to 17, further comprising an image patch generator module, the image patch generator module being configured to extract image patches from individual whole images in the acquired set of real images, in, The extracted image patches include ordered position embeddings extracted from a separate whole image based on a specified number of rows and columns into which the whole image is to be divided, and Wherein, the self-supervised pre-trainer module and the primary anomaly trainer module are configured to use the ordered position embeddings as images in the corresponding first data set and the corresponding second data set.

19. The computing system of claim 18, wherein: The extracted image patches also include overlapping position embeddings extracted from separate whole images based on a specified image patch size and degree of overlap, and wherein the self-supervised pre-trainer module and the primary anomaly trainer module are configured to use the ordered position embeddings and the overlapping position embeddings as images in the corresponding first data group and the corresponding second data group.

20. The computing system according to any one of claims 18 and 19, in, The extracted patches also include randomly located embeddings extracted from a separate whole image based on the specified patch size and number of patches. wherein the computing system further comprises an anomaly pre-trainer module configured to pre-train a candidate anomaly detection neural network using a third data set including the extracted random position embeddings in an unsupervised learning process, the candidate anomaly detection neural network having the same architecture as the primary anomaly detection neural network, and Before executing the main anomaly trainer module to train the main anomaly detection neural network, the main anomaly detection neural network is initialized using the weights of the pre-trained candidate anomaly detection neural network.

Images