Systems, methods, and computer devices for aggregate thresholding, adaptive cropping, and image classification for anomaly detection in machine vision applications

Abstract

A system and method for visual inspection and anomaly detection is disclosed. An inspection image is compared to a golden sample image to identify an anomaly map. Aggregate thresholding is performed on the anomaly map to identify anomalies. Adaptive cropping is performed on the identified anomalies to obtain cropped images of the anomalies. The cropped images are provided to an image classification model, which is a pseudo-one-class classifier. The image classification model classifies the anomalies.

Description

[Technical Field]

[0001] The following relates generally to machine learning-based visual inspection, and specifically to the visual inspection and improvement of images for anomaly detection. [Background technology]

[0002] Image analysis, anomaly detection, and similar procedures often require significant computational resources to thoroughly analyze each portion of an input image, which can be burdensome in both cost and time when not all of the input image is useful or reveals valuable information.

[0003] Traditional image analysis techniques may fail when an image containing multiple objects is presented to the classifier. Traditional image cropping techniques lose contextual information such as the size and dimensional ratio of the cropped anomaly. Traditional classifiers cannot detect new defects.

[0004] Traditional deep learning algorithms can require excessively large amounts of training data and may not perform reliably on tasks outside the scope of the training data.

[0005] Similarly, as anomaly detection and analysis evolves, regions of an input image and / or its background may be identified as a particular class of anomaly with a low confidence score. If these low confidence scores occur frequently, computer systems and devices performing such anomaly detection may waste further computational resources identifying detected anomalies as new and unnecessarily initiate further downstream operations triggered by the detection of new anomalies. This can be particularly problematic in visual inspection operations where time to inspect an object is limited, such as in manufacturing quality control applications.

[0006] Therefore, there is a need for systems, methods, and devices that overcome at least some of the shortcomings of existing systems and methods for visual inspection and anomaly detection. Summary of the Invention [Means for solving the problem]

[0007] Systems and methods are provided for anomaly detection in machine vision applications such as visual inspection. Novel techniques for aggregate thresholding, adaptive cropping, and image classification are also provided for use in anomaly detection and machine vision applications.

[0008] In one aspect, a machine vision anomaly detection method is provided. An inspection image (e.g., an image of an object or product under visual inspection) is compared to a golden sample image. An image subtraction operation is performed using the inspection image and the golden sample image to obtain a subtraction image. An image thresholding operation is performed on the subtraction image to identify anomalies corresponding to artifacts present in the inspection image but not in the golden sample image. The anomalies may be defined by bounding boxes. The anomalies identified by the thresholding operation are cropped using a cropping operation to obtain a cropped image containing the anomalies. The cropped image is provided to a trained image classifier. The image classifier classifies the cropped image (and any anomalies contained therein) into an anomaly class and assigns the anomalies a class label corresponding to the anomaly class. The output of the image classification process may be used to generate an annotated inspection image. For example, anomalies in the inspection image may be located using bounding boxes, and the bounding boxes may be labeled with the assigned anomaly class label. The annotated inspection image may be displayed in a user interface for review or used downstream in a comparison process where it is compared to the output of an object detection process performed on the inspection image. The location data of anomalies (e.g., bounding box coordinates, centroid values) in the annotated inspection image may be compared with the location data of objects (e.g., defects) detected using the object detection process. Comparing the outputs of the anomaly detection process and the object detection process may enable confirmation of the detected objects (e.g., defects) and improve the overall effectiveness of the machine vision system.

[0009] In one embodiment, the image thresholding operation is an aggregate thresholding process as described herein.

[0010] In one embodiment, the region cropping operation is an adaptive region cropping process as described herein.

[0011] In one embodiment, the image classifier is a pseudo-one-class classifier as described herein. The image classifier may be configured to classify anomalies as anomalous deviations, normal deviations, or novel deviations. Novel deviations may be considered true anomalies that the semi-supervised classification model has not seen in its training dataset.

[0012] In one embodiment, a system for inspecting an inspection image is provided, the system including a memory for receiving or storing the inspection image, a golden sample generator for generating a golden sample image from the inspection image, an image subtraction module for generating a subtraction image from the inspection image and the golden sample image, an aggregate thresholding module for generating an aggregate threshold image for identifying anomalies, an adaptive region cropping module for obtaining cropped images of the anomalies, and a cropped image classification module for classifying the cropped images of the anomalies with a pseudo one-class classifier.

[0013] The system may further include a camera configured to capture the inspection image.

[0014] The inspection image may show a part or object under inspection, or a portion of an area thereof.

[0015] The inspection image may be part of a video captured by a camera device.

[0016] The inspection image may be used to analyze the presence of defects.

[0017] The system may further include an adaptive ROI segmentation module for masking the test image and the golden sample image.

[0018] The system may further include a shape analysis and binarization module for receiving the subtracted image and generating a shape analyzed and binarized image.

[0019] The shape analysis and binarization module may perform binarization and binary image processing to remove defects and anomalies in the subtraction image that are smaller than a specified size and / or caused by minor surface texture variations.

[0020] Binary image processing may include erosion and dilation.

[0021] In one embodiment, a method for inspecting an inspection image is provided, the method including acquiring the inspection image, generating a golden sample image from the inspection image, performing an image subtraction operation on the inspection image and the golden sample image to obtain a subtraction image, performing aggregate thresholding on the subtraction image to generate an aggregate thresholded image for identifying anomalies, performing adaptive cropping on the aggregate thresholded image to obtain a cropped image of the anomalies, and classifying the cropped image of the anomalies with a pseudo one-class classifier.

[0022] The method may further include annotating the aggregate threshold map.

[0023] The method may further discard parts containing anomalies detected and / or confirmed in the aggregate threshold image.

[0024] The aggregate threshold map may include a bounding box surrounding each anomaly.

[0025] The method may further identify each region defined by each bounding box.

[0026] In one embodiment, a device for inspecting an inspection image is provided, the device including: a memory for receiving or storing the inspection image; a golden sample generator for generating a golden sample image from the inspection image; an image subtraction module for generating a subtraction image from the inspection image and the golden sample image; an aggregate thresholding module for generating an aggregate threshold image for identifying anomalies; an adaptive region cropping module for obtaining cropped images of the anomalies; and a cropped image classification module for classifying the cropped images of the anomalies with a pseudo one-class classifier.

[0027] The golden sample image generator may include a generative model.

[0028] The generative model may be an autoencoder.

[0029] An autoencoder may include an encoder component for compressing a test image to generate code components, and a decoder component for reconstructing the test image using the code components.

[0030] The golden sample image generator may obtain a suitable pre-stored golden sample image.

[0031] A golden sample image generator may receive the golden sample image.

[0032] In one embodiment, a system for aggregate thresholding is provided that includes an image subtraction module for generating an anomaly map from a comparison of an inspection image and a golden sample image, a primary threshold map generator for applying a first threshold value to the anomaly map to obtain a primary threshold map, a processed primary threshold map generator for processing the primary threshold map to obtain a processed primary threshold map, a secondary threshold map generator for applying a second threshold value to the anomaly map to obtain a secondary threshold map, and an aggregate threshold map generator for aggregating the primary and secondary threshold maps according to a set of one or more aggregation rules to obtain an aggregate threshold map.

[0033] The system may further include a camera configured to capture the inspection image.

[0034] The system may further include an adaptive ROI segmentation module for masking the test image and the golden sample image.

[0035] The system may further include a shape analysis and binarization module for receiving the subtraction image and performing binarization and binary image processing to generate a shape analysis and binarized image to remove defects and anomalies smaller than a specified size and / or caused by minor surface texture variations in the subtraction image. Binary image processing includes erosion and dilation.

[0036] The first threshold may be set by the user.

[0037] The second threshold may be set by the user.

[0038] The first threshold may be more conservative than the second threshold.

[0039] In one embodiment, a method for aggregate thresholding is provided that includes providing an anomaly map generated from a comparison of a test image and a golden sample image, performing a first image thresholding operation on the anomaly map using a first threshold to obtain a primary threshold map, processing the primary threshold map to obtain a processed primary threshold map, performing a second image thresholding operation on the anomaly map using a second threshold to obtain a secondary threshold map, and aggregating the primary and secondary threshold maps according to a set of one or more aggregation rules to obtain the aggregate threshold map.

[0040] The method may further discard parts containing anomalies detected and / or confirmed in the aggregate threshold image.

[0041] The aggregate threshold map may include a bounding box surrounding each anomaly.

[0042] The first threshold may be set by the user.

[0043] The second threshold may be set by the user.

[0044] The first threshold may be more conservative than the second threshold.

[0045] In one embodiment, a device for aggregate thresholding is provided that includes an image subtraction module for generating an anomaly map from a comparison of the inspection image and the golden sample image, a primary threshold map generator for applying a first threshold value to the anomaly map to obtain a primary threshold map, a processed primary threshold map generator for processing the primary threshold map to obtain a processed primary threshold map, a secondary threshold map generator for applying a second threshold value to the anomaly map to obtain a secondary threshold map, and an aggregate threshold map generator for aggregating the primary and secondary threshold maps according to a set of one or more aggregation rules to obtain the aggregate threshold map.

[0046] The golden sample image generator may include a generative model, which may include an autoencoder including an encoder component for compressing the inspection image to generate code components and a decoder component for reconstructing the inspection image using the code components.

[0047] The golden sample image generator may obtain a suitable pre-stored golden sample image.

[0048] A golden sample image generator may receive the golden sample image.

[0049] The first threshold may be set by the user.

[0050] The second threshold may be set by the user.

[0051] The first threshold may be more conservative than the second threshold.

[0052] In one embodiment, a system for adaptive region cropping is provided, the system including an adaptive cropping region module for receiving an aggregate threshold map based on a test image, the adaptive cropping region module implemented on a processor and memory having computer-executable instructions stored thereon that cause the processor to perform operations to: apply an expansion factor to each bounding box-defined region in the aggregate threshold map to obtain an expanded selection having an expanded size; if the input size of the image classification model is less than or equal to the expanded size, resize and downsize the expanded selection to the input size of the image classification model to provide a resized and / or downsized expanded selection; and if the input size of the image classification model is not less than or equal to the expanded size, zero-pad the expanded selection to the input size of the image classification model to provide a zero-padded expanded selection.

[0053] The expansion factor is expansion size = (16 (長さ-入力) / (0.88×入力)+ 1) × length, where "input" refers to the input size of the image classification model.

[0054] Resizing and downsizing may be done according to the formula crop size = min(input, dilation size), where "input" refers to the input size of the image classification model.

[0055] The system may further include a collision avoidance module to constrain the cropping coordinates to stay within the inspection image.

[0056] The system may further include a downsampling module for downsampling the aggregate threshold map when an abnormal blob indicated in the aggregate threshold map is larger than the input size of the image classification model.

[0057] The aggregate threshold map may include a bounding box surrounding each abnormal blob.

[0058] The inspection image may be captured by a camera.

[0059] In one embodiment, a method for adaptive region cropping is provided, the method including: providing an aggregate threshold map; for a region defined by each bounding box in the aggregate threshold map based on the test image, applying an expansion factor to obtain an expanded selection having an expanded size; if the input size of the image classification model is less than or equal to the expanded size, resizing and downsizing the expanded selection to the input size of the image classification model and providing the resized and / or downsized expanded selection; if the input size of the image classification model is not less than or equal to the expanded size, zero-padding the expanded selection to the input size of the image classification model and providing the zero-padded expanded selection.

[0060] The expansion factor is expansion size = (16 (長さ-入力) / (0.88×入力) + 1) × length, where "input" refers to the input size of the image classification model.

[0061] Resizing and downsizing may be done according to the formula crop size = min(input, dilation size), where "input" refers to the input size of the image classification model.

[0062] The method may further include providing collision avoidance by constraining the cropping coordinates to stay within the inspection image.

[0063] The method may further include downsampling the aggregate threshold map if the abnormal blob indicated in the aggregate threshold map is larger than the input size of the image classification model.

[0064] The aggregate threshold map may include a bounding box surrounding each abnormal blob.

[0065] The inspection image may be captured by a camera.

[0066] In one embodiment, a device for adaptive region cropping is provided, the device including an adaptive cropping region module for receiving an aggregate threshold map based on an inspection image, the adaptive cropping region module implemented on a processor and a memory with computer-executable instructions stored thereon to cause the processor to perform the following operations: for a region defined by each bounding box in the aggregate threshold map, apply an augmentation factor to obtain an augmented selection based on the augmented size, if the input size of the image classification model is less than or equal to the augmented size, resize and downsize the augmented selection to the input size of the image classification model and provide the resized and / or downsized augmented selection, and if the input size of the image classification model is not less than or equal to the augmented size, zero-pad the augmented selection to the input size of the image classification model and provide the zero-padded augmented selection.

[0067] The expansion factor is expansion size = (16 (長さ-入力) / (0.88×入力) + 1) × length, where "input" refers to the input size of the image classification model.

[0068] Resizing and downsizing may be done according to the formula crop size = min(input, dilation size), where "input" refers to the input size of the image classification model.

[0069] The device may further include a collision avoidance module to constrain the cropping coordinates to stay within the inspection image.

[0070] The device may further include a downsampling module for downsampling the aggregate threshold map when an abnormal blob indicated in the aggregate threshold map is larger than an input size of the image classification model.

[0071] The aggregate threshold map may include a bounding box surrounding each abnormal blob.

[0072] In one embodiment, a system for classifying an input image according to a pseudo-one-class classifier is provided, the system including a cropped image classification module including a classifier model, the cropped image classification module being implemented on a processor and memory with computer-executable instructions stored thereon that cause the processor to: receive the cropped image; determine a preliminary class label for the cropped image with the classifier model and a confidence level associated with the preliminary class label determination; if the preliminary class label is a first class, compare the confidence level of the preliminary class label determination to a first confidence threshold; if the confidence level meets the first confidence threshold, assign the cropped image a final class label indicating the first class; if the confidence level does not meet the first confidence threshold, assign the cropped image a final class label indicating a new class; if the preliminary class label is a second class, compare the confidence level of the preliminary class label determination to a second confidence threshold; if the confidence level meets the second confidence threshold, assign the cropped image a final class label indicating the second class; and if the confidence level does not meet the second confidence threshold, assign the cropped image a final class label indicating the new class.

[0073] The first trust level may be set by the user.

[0074] The second trust level may be set by the user.

[0075] The image classifier model can be a convolutional neural network.

[0076] The first class may represent normal deviations. The second class may represent abnormal deviations. The novel class may represent novel deviations.

[0077] Meeting a trust level may include equaling the trust level and may also include exceeding the trust level.

[0078] Meeting a trust level may include exceeding the trust level and may not include equaling the trust level.

[0079] In one embodiment, a method for classifying an input image according to a pseudo-one-class classifier is provided, the method includes providing a cropped image, determining a preliminary class label for the cropped image and a confidence level associated with the preliminary class label determination in a classifier model, if the preliminary class label is a first class, comparing the confidence level of the preliminary class label determination to a first confidence threshold, and assigning the cropped image a first final class label indicating the first class if the confidence level meets the first confidence threshold, and assigning the cropped image a second final class label indicating a novel class if the confidence level does not meet the first confidence threshold, if the preliminary class label is a second class, comparing the confidence level of the preliminary class label determination to a second confidence threshold, and assigning the cropped image a third final class label indicating the second class if the confidence level meets the second confidence threshold, and assigning the cropped image a second final class label indicating the novel class if the confidence level does not meet the second confidence threshold.

[0080] The first trust level may be set by the user.

[0081] The second trust level may be set by the user.

[0082] Meeting a trust level may include equaling the trust level and may also include exceeding the trust level.

[0083] Meeting a trust level may include exceeding the trust level and may not include equaling the trust level.

[0084] The method may further include generating a second annotated inspection image using the final class labels.

[0085] The method may further include flagging the part for review based on the final class label.

[0086] In one embodiment, a device is provided for classifying an input image according to a pseudo-one-class classifier, the device including a cropped image classification module including a classifier model, the cropped image classification module implemented on a processor and memory with computer-executable instructions stored thereon that cause the processor to perform the following operations: receive the cropped image; determine with the classifier model a preliminary class label for the cropped image and a confidence level associated with the preliminary class label determination; if the preliminary class label is a first class, compare the confidence level of the preliminary class label determination to a first confidence threshold, and if the confidence level meets the first confidence threshold, assign the cropped image a final class label indicating the first class; if the confidence level does not meet the first confidence threshold, assign the cropped image a final class label indicating a new class; if the preliminary class label is a second class, compare the confidence level of the preliminary class label determination to a second confidence threshold, and if the confidence level meets the second confidence threshold, assign the cropped image a final class label indicating the second class; and if the confidence level does not meet the second confidence threshold, assign the cropped image a final class label indicating the new class.

[0087] The first trust level may be set by the user.

[0088] The second trust level may be set by the user.

[0089] Meeting a trust level may include equaling the trust level and may also include exceeding the trust level.

[0090] Meeting a trust level may include exceeding the trust level and may not include equaling the trust level.

[0091] The image classifier model can be a convolutional neural network.

[0092] Other aspects and features may become apparent to those of ordinary skill in the art upon review of the following description of several embodiments. [Brief explanation of the drawings]

[0093] [Figure 1] FIG. 1 is a schematic diagram of a vision inspection system including anomaly detection in one embodiment. [Figure 2] FIG. 2 is a block diagram of a computing device in accordance with an embodiment of the present disclosure. [Figure 3] FIG. 3 is a block diagram of a computer system for vision inspection including anomaly detection in one embodiment. [Figure 4] FIG. 4 is a conceptual diagram of aggregate thresholding in one embodiment. [Figure 5] FIG. 5 is a conceptual diagram of an adaptive cropping process in one embodiment. [Figure 6A] FIG. 6A illustrates a set of cropped images classified by a pseudo one-class classifier in accordance with an embodiment of the present disclosure. [Figure 6B] FIG. 6B shows the cropped image set of FIG. 6A classified by a binary classifier. [Figure 6C]FIG. 6C shows the classification of the cropped image set of FIGS. 6A and 6B using a one-class classifier. [Figure 7] FIG. 7 is a block diagram of an anomaly detection pipeline of the present disclosure, in one embodiment, that may be executed by the computer system of FIG. [Figure 8] FIG. 8 is a flowchart of a visual inspection method including anomaly detection in one embodiment. [Figure 9] FIG. 9 is a flowchart of an aggregate thresholding method for use in machine vision anomaly detection in one embodiment. [Figure 10] FIG. 10 is a flowchart of an adaptive region cropping method for use in machine vision anomaly detection in one embodiment. [Figure 11] FIG. 11 is a flow diagram of a method for classifying anomalies in a cropped image using a pseudo one-class classifier for use in machine vision anomaly detection, according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0094] The drawings included herein are intended to illustrate various examples of the present disclosure.

[0095] Various devices or processes are described below to provide example embodiments. The embodiments described below do not limit any embodiment, and embodiments may include processes or devices different from those described below. Embodiments are not limited to devices or processes having all the features of any device or process described below, or to features common to multiple or all of the devices described below.

[0096] One or more systems described herein may be implemented as a computer program executing on a programmable computer, each including at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, without limitation, the programmable computer can be a programmable logic unit, a mainframe computer, a server, a personal computer, a cloud-based program or system, a laptop, a personal data assistant, a mobile phone, a smartphone, or a tablet device.

[0097] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In either case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage medium or device readable by a general or special-purpose programmable computer, and is used to configure and operate the computer when the storage medium or device is read by the computer.

[0098] The description of an embodiment in which multiple components communicate with each other does not imply that all components are required, but rather various optional components are described to illustrate the variety of possible embodiments of the present invention.

[0099] Additionally, although process steps, method steps, algorithms, or the like may be described (in the disclosure and / or claims) in a sequential order, such processes, methods, algorithms, or the like may be configured to operate in different orders. In other words, the order or sequence of steps that may be described does not necessarily indicate that the steps must be performed in that order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0100] When a single device or product is described herein, it will be readily understood that multiple devices / products (whether or not cooperating) can be used in place of the single device / product. Similarly, when multiple devices or products are described herein (whether or not cooperating), it will be readily understood that the single device / product can be used in place of the multiple devices or products.

[0101] The following relates generally to machine vision applications, and more specifically to anomaly detection techniques for use in machine vision applications such as automated visual inspection.

[0102] This disclosure provides new advances in deep learning for visual inspection.

[0103] Deep learning solutions continue to deliver high customer value and are being widely implemented in vision inspection applications. New advances in deep learning model architectures are creating better performing inspection software while significantly reducing the amount of data required to develop a solution.

[0104] To perform robust inspections, deep learning algorithms typically require large amounts of training data. This is especially true when supervised learning algorithms and models are used for inspection. While large amounts of defect data can yield reliable object detection, segmentation, and classification networks, many customers consider it impractical to wait for the accumulation of large datasets to train reliable AI models. Large datasets often extend project lead times and result in customer dissatisfaction. More importantly, there is no guarantee that a supervised deep learning model can accurately detect defects outside the scope of the training dataset.

[0105] Therefore, a new set of deep learning and AI algorithms are being created to enhance visual inspection software. Unsupervised algorithms, especially anomaly detection, are well equipped to deal with unknown and infrequent types of defects in production environments.

[0106] To obtain an accurate anomaly map while keeping false positives low, the present disclosure provides a two-stage thresholding method called adaptive or agile thresholding, which takes into account desired measurements (e.g., length, width, area) of the defect area and can report only anomalies that fall within the specified specifications.

[0119] The present disclosure also provides a method for adaptive cropping. The adaptive cropping technique of the present disclosure can improve the accuracy of the classification network used in the disclosed anomaly detection visual inspection system and method. The adaptive cropping method determines a dynamic window and zero-padding region for the image. As a result, only regions of the image that have the context information necessary for the next step (i.e., image classification) are preserved.

[0107] This disclosure also provides a new classifier (a pseudo-one-class classifier) ​​for use in anomaly detection visual inspection that can reliably separate normal surface variations on a product from abnormal occurrences (ranging from small defects to large debris and stains). This allows the anomaly detection system to accommodate complex parts with inconsistent visual appearances and maintain high recall and precision without relying on defect data.

[0108] While this disclosure describes systems and methods for anomaly detection and visual inspection of objects, the systems, methods, and devices provided herein have additional applications and different uses beyond those described herein, whether in the context of defect detection and visual inspection of objects. Computational devices configured for anomaly detection described herein can have functions other than anomaly detection. The input data may change in such cases, and the output data may also change, but elements of the present disclosure, such as aggregate thresholding, adaptive cropping, and image classification (via pseudo-one-class classifiers), can operate similarly.

[0109] While this disclosure provides novel systems and methods for (i) aggregate thresholding, (ii) image region cropping, and (iii) image classification in an anomaly detection system, it should be understood that this disclosure is intended to cover embodiments of an anomaly detection system that include any one or more of (i) through (iii). In embodiments that include fewer than three of (i) through (iii), it should be understood that for novel techniques not included, similar techniques may be used instead (e.g., other forms of image region cropping rather than adaptive region cropping as described herein).

[0110] 1, a system 10 for visual inspection and anomaly detection according to an embodiment is shown. The system 10 includes an anomaly detection visual inspection device 12 that communicates with a camera device 14, a user device 16, and a control device 18 via a network 20.

[0111] The devices 12, 14, 16, and 18 may be server computers, node computing devices (e.g., JETSON computing devices), embedded devices, desktop computers, notebook computers, tablets, PDAs, smartphones, or other computing devices. The devices 12, 14, 16, and 18 may include a connection to a network 20, such as a wired or wireless connection to the Internet. In some cases, the network 20 may include other types of computers or communications networks. The devices 12, 14, 16, and 18 may include one or more of a memory, a secondary storage device, a processor, an input device, a display device, and an output device. The memory may include random access memory (RAM) or a similar type of memory. The memory may also store one or more applications executed by the processor. The applications may correspond to software modules containing computer-executable instructions for performing the processing of the functions described below. The secondary storage device may include a hard disk drive, a floppy disk drive, a CD drive, a DVD drive, a Blu-ray drive, or other types of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. The application, computer readable instructions or program may be stored in memory or secondary storage, or may be received from the Internet or other network 20 .

[0112] An input device may include any device for inputting information into device 12, 14, 16, 18. For example, an input device may be a keyboard, keypad, cursor control device, touch screen, camera, or microphone. A display device may include any type of device for presenting visual information. For example, a display device may be a computer monitor, a flat screen display, a projector, or a display panel. An output device may include any type of device for presenting hard copies of information. For example, a printer. An output device may also include other types of output devices, such as speakers. In some cases, device 12, 14, 16, 18 may include a plurality of any one or more of a processor, an application, a software module, a secondary storage device, a network connection, an input device, an output device, and a display device.

[0113] While devices 12, 14, 16, and 18 are described with various components, those skilled in the art will understand that devices 12, 14, 16, and 18 may, in some cases, include fewer, additional, or different components. Furthermore, while aspects of the implementation of devices 12, 14, 16, and 18 may be described as being stored in memory, those skilled in the art will understand that these aspects may be stored on other types of computer program products or computer-readable media, such as secondary storage devices including hard disks, floppy disks, CDs, or DVDs, carrier waves from the Internet or other networks, or other forms of RAM or ROM. Computer-readable media may include instructions for controlling devices 12, 14, 16, and 18 and / or processors to perform particular methods.

[0114] Devices 12, 14, 16, 18 can be described as performing particular operations. It will be understood that any one or more of these devices can perform an operation in response to manipulation by a user of the device or automatically. That is, a user of the device can manipulate one or more input devices (e.g., a touchscreen, mouse, or buttons) to cause the device to perform a defined operation. In many cases, this aspect may not be described below, but will be understood.

[0115] By way of example, as described below, devices 12, 14, 16, 18 may transmit information to one or more other devices 12, 14, 16, 18. Generally, a device may receive a user interface (e.g., in the form of a web page) from network 20. Alternatively, and additionally, the user interface may be stored locally on the device (e.g., a cached web page or a mobile application).

[0116] The device 12, 14, 16, 18 may be configured to receive a plurality of pieces of information from one or more of the devices 12, 14, 16, 18.

[0117] In response to receiving the information, each device 12, 14, 16, 18 can store the information in a storage database. The storage can correspond to secondary storage of one or more of the other devices 12, 14, 16, 18. Generally, the storage database can be any suitable storage device, such as a hard disk drive, solid state drive, memory card, or disc (e.g., CD, DVD, or Blu-ray). Also, the storage database can be locally connected to the device 12, 14, 16, 18. In some cases, the storage database can be located remotely from the device 12, 14, 16, 18 and accessible to the device 12, 14, 16, 18 over a network. In some cases, the storage database can include one or more storage devices located at a networked cloud storage provider.

[0118] The visual inspection device 12 may be a dedicated machine specifically designed to perform anomaly detection tasks, image analysis tasks, object (e.g., defect) detection tasks, object (e.g., defect) classification tasks, golden sample analysis tasks, object (e.g., defect) tracking tasks, other machine vision or image processing tasks (aggregate thresholding, adaptive cropping, pseudo one-class classification) and related data processing tasks using inspection images captured by the camera device 14.

[0119] The camera device 14 captures image data. The captured image data may be referred to as an "inspection image." The image data may be of a part or object being inspected, or a section or area thereof. The image data may include a single image or multiple images. Multiple images (frames) may be captured as video by the camera 14. To capture an area of ​​the object being inspected (also referred to as an "inspection object" or "object under inspection"), the camera 14 and the object under inspection may move relative to one another. For example, the object may be rotated, and multiple images captured by the camera 14 at different positions may be used to fully inspect the object from multiple angles. The camera 14 is configured to capture multiple frames, each captured at a different position (e.g., if the object is rotating relative to the camera 14).

[0120] Generally, the object being inspected may be an object in which defects are undesirable. Defects in the object being inspected may reduce the functional performance of the object or a larger object (e.g., a system or machine) of which it is a component. Defects in the object being inspected may reduce the visual appeal of the product. Discovering defective products may be an important step for a company to prevent the sale and use of defective products and to identify and correct the root causes associated with the defects.

[0121] The object being inspected may be a manufactured product. The object being inspected may be a product that is prone to defects occurring during manufacturing. The object may be a product that derives value from its visual appearance, and certain defects may adversely affect its visual appearance. Defects in the object being inspected may occur during the manufacturing of the object itself or during other processes (e.g., transportation, testing).

[0122] The object under inspection may be composed of one or more materials such as metal, steel, plastic, composites, wood, glass, etc.

[0123] The object under inspection may be uniform or non-uniform in size and shape. The object may have a curved outer surface.

[0124] An object to be inspected may include multiple sections. The object sections may be further divided into object subsections. The object sections (or subsections) may be determined based on the appearance or function of the object. The object sections may be determined to improve visual inspection of the object and to better identify objects with unacceptable defects.

[0125] Sections of an object may correspond to different parts of the object with different functions. Different sections may have similar or different dimensions. In some cases, an object may contain multiple different section types, and each section type may appear one or more times on the inspected object. Sections may have regular or irregular shapes. Different sections may have different defect specifications (i.e., tolerances for certain defects).

[0126] An object being inspected may be susceptible to multiple types or classes of defects that can be detected using system 10. Example defects may include paint, porosity, dents, scratches, stains, etc. The types of defects may vary from object to object. For example, the types of defects may be specific to an object based on the manufacturing process or material composition of the object. The defects in an object may arise during the manufacturing itself or during subsequent processing of the object.

[0127] User device 16 may be configured to receive and display on a user interface data output generated by vision inspection device 12. User device 16 is configured to receive input data from a user and display data generated by vision inspection device 12, etc., to the user.

[0128] Control device 18 is configured to control the manipulation and physical processing of the object under inspection. This may be done by sending and receiving control instructions to a product manipulation unit (not shown) via a communications link. Such manipulation and physical processing may include rotating the object under inspection for photography or loading and unloading objects into an inspection area. An example of an instruction that control unit 18 may send to the product manipulation unit via a communications link is "Rotate target product 'n' degrees." In some cases, the sending of such instructions may depend on information received from vision inspection device 12. Control device 18 may be configured to generate and send control signals to control the operation of one or more components of the vision inspection device. Such control signals may be determined based on the output of vision inspection device 12. Control device 18 may also communicate with and control the operation of one or more of devices 12, 14, and 16.

[0129] Referring to Figure 2, there is shown a block diagram of a computing device 1000 of system 10 of Figure 1. Computing device 1000 may be, for example, any of devices 12, 14, 16, 18 of Figure 1.

[0130] Computing device 1000 includes several components, such as a processor 1020 that controls the operation of computing device 1000. Communication functions, including data communications, voice communications, or both, may be performed through a communications subsystem 1040. Data received by computing device 1000 may be decompressed and decoded by a decoder 1060. Communications subsystem 1040 may receive messages from and transmit messages to a wireless network 1500.

[0131] The wireless network 1500 may be any type of wireless network, including, but not limited to, a data-centric wireless network, a voice-centric wireless network, and a dual-mode network that supports both voice and data communications.

[0132] Computing device 1000 is a battery-powered device and includes a battery interface 1420 for accepting one or more rechargeable batteries 1440 as shown.

[0133] The processor 1020 interacts with additional subsystems such as random access memory (RAM) 1080, flash memory 1110, a display 1120 (e.g., comprising a touch-sensitive display 1180 with a touch-sensitive overlay 1140 connected to an electronic controller 1160), an actuator assembly 1200, one or more optional force sensors 1220, an auxiliary input / output (I / O) subsystem 1240, a data port 1260, a speaker 1280, a microphone 1300, a short-range communication system 1320, and other device subsystems 1340.

[0134] In some embodiments, user interaction with the graphical user interface may be performed through touch-sensitive overlay 1140. Processor 1020 may interact with touch-sensitive overlay 1140 via electronic controller 1160. Information such as text, characters, symbols, images, icons, and other items that may be displayed or rendered on a computing device generated by processor 1020 may be displayed on touch-sensitive display 1180.

[0135] The processor 1020 may also interact with an accelerometer 1360. The accelerometer 1360 may be utilized to detect the direction of gravity or a reaction force caused by gravity.

[0136] To identify a subscriber for network access according to this embodiment, computing device 1000 may use a SIM / RUIM (Subscriber Identity Module or Removable User Identification) card 1380 inserted into SIM / RUIM interface 1400 for communication with a network (e.g., wireless network 1500). Alternatively, the user identification information may be programmed into flash memory 1110 or implemented using other techniques.

[0137] Computing device 1000 also includes operating system 1460 and software components 1480 that are executed by processor 1020 and may be stored in a non-volatile data storage device, such as flash memory 1110. Additional applications may be loaded into computing device 1000 through wireless network 1500, auxiliary I / O subsystem 1240, data port 1260, short-range communications subsystem 1320, or other suitable device subsystem 1340.

[0138] During use, received signals, such as text messages, email messages, web page downloads, or other data, may be processed by the communications subsystem 1040 and input to the processor 1020. The processor 1020 may then process and output the received signals to the display 1120 or the auxiliary I / O subsystem 1240. A subscriber may also create data items, such as email messages, to send over the wireless network 1500 through the communications subsystem 1040.

[0139] For voice communication, the overall operation of computing device 1000 may be similar: speaker 1280 outputs audible information converted from an electrical signal, and microphone 1300 may convert the audible information into an electrical signal for processing.

[0140] Referring to Figure 3, a computer system 300 for automated visual inspection including anomaly detection is shown. Computer system 300 may be implemented in one or more devices of system 10 of Figure 1. For example, computer system 300 or components thereof may be implemented in any one or more of visual inspection device 12, user device 16, and control device 18 of Figure 1. System 300 may function as an anomaly defect detector and anomaly classifier.

[0141] The system 300 includes a processor 302 for executing software models and modules.

[0142] The system 300 further includes a memory 304 in communication with the processor 302 for storing data, including output data from the processor 302 .

[0143] The system 300 further includes a communication interface 306 for communicating with other devices and can receive and transmit data over a network connection (eg, network 20 of FIG. 1).

[0144] System 300 further includes a display 308 for displaying in a human-readable format various data generated by computer system 300. For example, the display may be configured to display inspection results of the inspected object. Display 308 may be implemented in user device 16 of FIG. 1.

[0145] The memory 304 stores an inspection image 310 including image data. The inspection image 310 may be of an object under inspection. The inspection image 310 may be received by the system 300 via the communication interface 306. The inspection image 310 may be generated and provided by the camera 14 of FIG. 1 or may be received from another source (e.g., a network device, an external storage device, etc.).

[0146] Memory 304 also stores golden sample image 312. Generally, golden sample image 312 is an idealized representation of inspection image 310 (or the region of interest indicated by the input, e.g., in the case of a masked inspection image). Golden sample image 312 may represent an image of an object or part that is free of defects or improper assembly, and golden sample image 312 and inspection image 310 can be compared to identify defects or anomalies in inspection image 310 (and thus the object or part captured in inspection image 310). Generally, comparing inspection image 310 and golden sample image 312 facilitates identifying differences, which may then be analyzed to determine the presence of anomalies.

[0147] Golden sample image 312 may be generated by computer system 300. For example, in some embodiments, computer system 300 includes a golden sample generation module 314 for generating golden sample image 312 using inspection image 310. In one embodiment, golden sample generation module 314 includes a generative model (not shown) that receives inspection image 310 as input and generates golden sample image 312 as output.

[0148] The generative model may be an autoencoder, such as a variational autoencoder (VAE). The generative model is configured to generate a golden sample image 312 from an inspection image 310. The golden sample image 312 generated using the generative model may be considered a "generated golden sample." In other embodiments, a non-generated golden sample image may be used. In an autoencoder embodiment, the generative model may include an encoder component, a code component, and a decoder component (not shown). The encoder component compresses the input (inspection image 310) and generates a code component. The decoder component reconstructs the input using only the code component. In this case, the term "reconstruction" refers to reconstructing a representation of the inspection image 310 that is less noisy than the inspection image 310. Preferably, the reconstructed representation is noise-free. The reconstructed representation is the golden sample image 312 for the particular inspection image 310 used as input to the generative model. The generative model may include an encoding method, a decoding method, and a loss function for comparing the output to a target. Typically, a generative model receives an inspection image 310 that contains a mixture of noisy and noise-free data. A generative model is trained using a training process that helps set the model's weights so that the model can recognize what is noise and what is data. Once trained, the generative model can receive a noisy inspection image 310 and generate a noise-free (or low-noise) image at output (i.e., golden sample image 312). The noise removed from the inspection image 310 can be defects or other deviations (e.g., deviations from a machined surface considered normal). The noise present in the inspection image 310 and that the generative model is configured to remove can have various sources. For example, the noise can be different types of defects or abnormal objects, or droplets of liquid or contamination from such liquids. The environment and air within factories and other manufacturing facilities are generally not clean, and as a result, metal chips, coolant residue, oil droplets, etc. can remain on objects (e.g., camshafts) after machining (e.g., CNC machining) and leave them on the floor for long periods of time.Additionally, the object may be cleaned or covered with a protective material such as a rust inhibitor or oil.

[0149] In other embodiments, the memory 304 may store a bank of golden sample images for obtaining suitable golden sample images 312 (ie, non-generated golden samples).

[0150] In other embodiments, the golden sample image 312 may not be generated by the system 300, but rather may be received from an external device via the communications interface 306. For example, the golden sample image 312 may be received from a network-connected device or an external storage device. In such embodiments, the system 300 may not include the golden sample generator 314.

[0151] The processor 302 further includes an image comparison module 316 and a cropped image classification module 318. The image comparison module 316 is configured to compare the inspection image 310 and the golden sample image 312 and generate an output that is provided to the classification module 318 for classification.

[0152] The image comparison module 316 performs a direct image comparison of the inspection image 310 and the golden sample image 312 (or masked versions 310, 312 as described herein) to generate comparison output data. The comparison output data may include one or more detected anomalies (or artifacts). A detected anomaly in this context refers to an artifact or anomaly that is present in the inspection image 310 but not in the golden sample image 312. In other words, the detected artifact or anomaly represents a detected difference between the images 310, 312.

[0153] The image comparison module 316 includes an image subtraction module 320 .

[0154] Image subtraction module 320 is configured to receive inspection image 310 and golden sample image 312 as inputs and perform an image subtraction operation to generate subtraction image 322 as an output.

[0155] For example, the image subtraction module 320 may subtract the digital values ​​of the pixels of the images 310, 312. In one embodiment, the images 310, 312 may be compared using matrix subtraction or pixel-by-pixel grayscale subtraction to generate an output that identifies artifacts. The image subtraction module 320 may compare the images 310, 312 on a pixel-by-pixel basis. The subtraction image 322 is stored in the memory 304. The subtraction image 322 may be considered an "anomaly map" (because the subtraction process identifies differences between the images 310, 312 that may be considered anomalies).

[0156] In some embodiments, the inspection image 310 and the golden sample image 312 are masked before image subtraction, and image subtraction is performed on the masked versions of the images 310, 312. Masking may include masking or obscuring "regions of interest" (nROIs) in each image. This may limit the analyzed images to the regions of interest ("ROIs"). This minimizes false positives and allows for more efficient use of computer resources. The nROIs may include non-uniform regions of the inspection image depicting the object of inspection. Non-uniform regions may vary in appearance from product to product and may contain components of the object that are not of interest or relevant to the visual inspection. Such relevance determinations may be made in advance by the user or by the system 10 during processing. Non-uniform regions may include improperly illuminated areas or portions. Some visual inspection tasks may require illumination of the object. Such illumination may be translated into an inspection image of the illuminated object. In some cases, the illumination may be complex and require or use multiple illumination sources. Illumination may cause uneven illumination of an object (e.g., properly or brightly lit areas and improperly or dimly lit areas). Uneven illumination may cause problems or inefficiencies in downstream image analysis processes, such as defect and anomaly detection (e.g., by causing false positives). By identifying and masking improperly lit areas that are not of interest to the vision inspection system, the system may provide improved image analysis (e.g., defect detection, anomaly detection). Uneven areas may further include surfaces, such as textured surfaces (e.g., the casting surface of a camshaft), that may produce various anomalies or defects in image analysis. Masking areas covered by such surfaces may reduce false positives and improve overall defect and anomaly detection.

[0157] In particular embodiments, the test image 310 may be provided to an adaptive ROI segmentation module (not shown), which identifies and masks nROIs within the test image 310. Masking the nROIs may be performed by setting the pixels of the nROIs in the test image 310 to black, which may include setting pixels within the nROIs to black and / or setting all pixels in the image outside the ROIs to black. The masked test image (not shown) is provided to a generative model (not shown), which generates a masked golden sample image (not shown) from the masked test image. Advantageously, because the nROIs are masked in the masked test image, the generative model does not generate data with respect to those regions. Thus, the generative model may be more efficient and effective by avoiding unnecessary processing of regions of no interest identified by the adaptive ROI segmentation module.

[0158] Other features of adaptive region of interest segmentation described in International Patent Application No. CA2022050289, which is incorporated herein by reference, may be used in the systems and methods of the present disclosure.

[0159] The image comparison module 316 includes a shape analysis and binarization module 324 .

[0160] The shape analysis and binarization module 324 receives the subtraction image 322 as input and generates a shape analysis and binarized image (SAB output) 326 as output. The SAB output 326 of the shape analysis and binarization module 324 may be referred to as an "anomaly map." In one embodiment, the shape analysis and binarization module 324 may perform binarization, binary image processing (erosion and dilation) to remove defects and anomalies smaller than specified, or defects and anomalies caused by minor surface texture variations. The shape analysis and binarization module 324 performs binarization on the subtraction image 322. Binarization includes using a static threshold to create a black and white image. The shape analysis and binarization module 324 also performs shape analysis on the subtraction image 322. Shape analysis includes a combination of morphological operations.

[0161] The image comparison module 316 further includes an aggregate threshold module 328 for performing an aggregate threshold operation on the SAB output 326 .

[0162] The aggregate threshold module 328 includes a primary threshold map generator 330, a processed primary threshold map generator 332, a secondary threshold map generator 334, and an aggregate threshold map generator 336. In an embodiment, the shape analysis and binarization module 324 may be the same as or integral with the primary threshold map generator 330 and / or the processed primary threshold map generator 332. In an embodiment, the shape analysis and binarization module 324 is distinct from the primary threshold map generator 330 and the processed primary threshold map generator 332, and the SAB output 326 is provided as an input to the primary map generator 330.

[0163] The primary threshold map generator 330 is configured to receive an input and perform a threshold operation on the input using a primary (or first) threshold to obtain a primary (or first) threshold map 338. The input may be the subtraction image 322 or the output of the shape analysis and binarization module 324 (e.g., the SAB output 326). The threshold may be user-selected. The threshold may be set by a user using the user device 16 of FIG. 1.

[0164] The primary threshold map 338 is provided to a processed primary threshold map generator 332, which generates a processed primary (or first) threshold map 340. This may include binarization, binary image processing (erosion and dilation and grouping of nearby anomalies) to remove smaller specified defects and / or anomalies caused by minor surface texture variations.

[0165] The secondary threshold map generator 334 is configured to receive an input and perform a threshold operation on the input using a secondary (or second) threshold to obtain a secondary (or second) threshold map 342. The threshold may be user-selected. The threshold may be set by a user using the user device 16 of FIG. 1. The input to the secondary threshold map generator 334 may be the subtraction image 322. The input to the secondary threshold map generator 334 may be the SAB output 326.

[0166] The primary and secondary thresholds used in modules 330, 334 are different. In general, the primary threshold may be considered a more conservative (lower) threshold (e.g., a very conservative threshold) and the secondary threshold may be considered a more aggressive (higher) threshold. For example, the primary threshold may be a higher number and the secondary threshold may be a lower number, such that the primary threshold catches very dark areas of the SAB output 326 and the secondary threshold catches larger areas of the SAB output 326.

[0167] The processed primary threshold map 340 and secondary threshold map 342 are provided to an aggregate threshold map generator 336 which aggregates the two threshold maps 340 , 342 to obtain an aggregate threshold map 344 .

[0168] The aggregation may be performed according to one or more aggregation rules encoded in the aggregated threshold map generator 336. For example, in one embodiment, the one or more aggregation rules may include: "Include in the aggregated threshold map 344 blobs from the secondary threshold map 342 that overlap with blobs from the processed primary threshold map 340. Do not include in the aggregated threshold map 344 blobs from the secondary threshold map 342 that do not overlap with blobs from the processed primary threshold map 340. Do not include in the aggregated threshold map 344 blobs from the processed primary threshold map 340." In other embodiments, the aggregation rules may differ.

[0169] Generally, the focus of the aggregate threshold map generator 336 is to find overlapping blobs in the two maps 340, 342. For example, the aggregate map generator 336 may check the secondary threshold map 342 to see if any blobs therein overlap with blobs in the primary threshold map 340. If the blobs overlap, the blobs in the secondary threshold map 342 are included in the aggregate threshold map 344. In such cases, no pixels are subtracted or added; rather, the full area of ​​the blob in the secondary threshold map 342 remains (if any portion of that blob overlaps with any portion of a blob in either of the processed primary maps 340). The term "blob" is sometimes also referred to as an "abnormal blob" (i.e., a blob representing an anomaly in an image).

[0170] The aggregate threshold map 344 is used to determine location data (e.g., bounding box coordinates) of regions containing potential anomalies (i.e., artifacts that may be abnormal) within the inspection image 310, which may be used to identify or localize such regions within the inspection image 310. For example, a bounding box may be determined for each blob in the aggregate threshold map 344. The inspection image 310 may then be annotated with the determined bounding boxes (annotated inspection image 346).

[0171] 4, there is shown a schematic diagram of an aggregate thresholding process 400 according to one embodiment. The aggregate thresholding process 400 may be performed by the aggregate threshold module 328 of FIG.

[0172] 4 shows an example of a processed primary threshold map 340 and an example of a secondary threshold map 342. The example of processed primary threshold map 340 and secondary threshold map 342 are aggregated to obtain an aggregate threshold map 344.

[0173] The processed primary threshold map 340 includes blobs 402, 404, 406, 408, 410, and 412. Blobs 408, 410, and 412 represent potential anomalies in the inspection image 310 (i.e., regions of the inspection image 310 that may be determined to contain anomalies by operation of the system 300). Blob 408 includes overlapping blobs 402, 404, and 406. Processing the corresponding primary threshold map 338 may include identifying blobs 402, 404, and 406 as overlapping and generating blob 408. In processing the primary threshold map 338 to obtain the processed primary threshold map 340, the processed primary threshold map generator 332 may remove small potential anomalies (not shown) and merge adjacent anomalies. For example, overlapping mask regions 402, 404, and 406 may be merged to generate mask region 408.

[0174] Secondary threshold map 342 includes blobs 414, 416, and 418. Blobs 414, 416, and 418 represent potential anomalies within inspection image 310 (i.e., regions of inspection image 310 that may be determined to contain anomalies by operation of system 300).

[0175] Differences in the presence of blobs can be seen between the processed primary threshold map 340 and secondary threshold map 342. Such differences are generally due to the different thresholds applied to generate each map 340, 342.

[0176] The maps 340, 342 are aggregated according to one or more aggregation rules to obtain an aggregated threshold map 344. In the embodiment of FIG. 4, the one or more aggregation rules include: "Include in aggregated threshold map 344 any blob in secondary threshold map 342 that overlaps with a blob in processed primary threshold map 340. Do not include in aggregated threshold map 344 any blob in secondary threshold map 342 that does not overlap with a blob in processed primary threshold map 340. Do not include in aggregated threshold map 344 any blob from processed primary threshold map 340." In other embodiments, the aggregation rules may be different. For example, this embodiment may allow any overlap between secondary blobs and primary blobs, while other embodiments may include secondary blobs only if the amount of overlap meets a predetermined overlap threshold. Aggregation may also be referred to as "validation."

[0177] Blobs 414, 416 in aggregate threshold map 344 are localized by bounding boxes 420, 422, respectively, that surround the blobs. The bounding boxes from aggregate threshold map 344 may be used to identify regions of inspection image 310 for subsequent operations (e.g., cropping or classification). For example, the bounding boxes may be used to annotate inspection image 310 and generate annotated inspection image 346.

[0178] The aggregate thresholding process of the present disclosure may provide certain advantages over conventional or existing masking approaches. Conventional masking disadvantageously excludes low-contrast regions of the anomaly mask. For example, one-step thresholding with a low threshold may result in high false positives. Furthermore, one-step thresholding with a high threshold may fail to detect small, high-contrast regions. As shown in FIG. 4 , aggregate thresholding advantageously includes regions of the anomaly mask with different levels of contrast. Aggregate thresholding further advantageously assists the classifier in developing more accurate bounding boxes for anomalies. Aggregate thresholding also advantageously eliminates false positives, i.e., anomalies with only low-contrast pixels. Aggregate thresholding further advantageously assists in improving the delineation of anomaly boundaries, thereby providing more context to the classifier. Aggregate thresholding may also assist in creating more accurate bounding boxes for anomalies.

[0179] Referring again to FIG. 3, the image comparison module 316 also includes an adaptive region cropping module 348 .

[0180] Adaptive region cropping module 348 receives annotated inspection image 346 as input and performs a cropping operation on annotated inspection image 346 to obtain one or more cropped images 350 .

[0181] Generally, adaptive cropping is performed to cut out or extract a region or subset of image data from the inspection image 310 for use in image classification (by the classification module 318). Generally, the cropped image 350 corresponds to a region of the inspection image 310 that contains a potential anomaly (and thus the cropped image 350 may also be referred to as a cropped anomaly 350).

[0182] The adaptive region cropping module 348 performs the cropping using the position data (eg, bounding box coordinates) of the annotated inspection image 346 to obtain a cropped image 350 .

[0183] The adaptive cropping process performed by the adaptive cropping module 348 will now be described.

[0184] First, a bounding box is identified (e.g., from the aggregate threshold map 344 or the annotated inspection image 346). The region defined by the bounding box is sometimes called the "selected region."

[0185] An expansion factor is applied to the selected region to obtain the expansion size. In one embodiment, the expansion factor is defined by the following formula: Expansion Size = (16 (長さ-入力) / (0.88×入力) +1) × length The above is an example of an expansion size formula. In variations, the expansion size equation may vary (e.g., slightly) from application to application.

[0186] Specifically, the bounding box and / or the area defined by the bounding box are dilated to obtain a dilated selection with a dilation size according to the above formula. When performing the dilation operation, the bounding box and the corresponding cropping window are dilated. For example, if application of the dilation formula (e.g., the above formula) produces outputs of 50 and 100 in the x and y directions, respectively, the margin around the actual abnormal blob can jump from 0 pixels to 50 pixels in the x direction and 100 pixels in the y direction.

[0187] In the above equation, "length" refers to the width or height of the anomaly (i.e., when calculating the x-direction, "length" is the width, and when calculating the y-direction, "length" is the height). In the above equation, "input" refers to the input size of the image classifier (i.e., the input size of the classifier model 352).

[0188] Once the dilation size is determined, the cropping size is determined by comparing the dilation size to the input size using the following formula: crop size = min(input,extension size)

[0189] According to the above equation, the dilated image is cropped to the smaller of (i) the input size of the classifier to which the cropped image 350 is provided, or (ii) the current dilation size.

[0190] The adaptive region cropping module 348 may perform collision avoidance after the expansion size is determined. Collision avoidance includes constraining the cropping coordinates to remain within a larger image (e.g., the inspection image 310). Collision avoidance is advantageous to prevent the computer system 300 or software implementing it from making errors, which may occur if the computer system 300 or software implementing it attempts to crop an anomaly with negative dimension values ​​or dimension values ​​greater than the respective dimension of the inspection image 310 (x or y). For example, if the input size is less than or equal to the expansion size, the adaptive region cropping module 348 determines whether the expanded selection and / or bounding box will collide with other bounding boxes and / or exceed the boundaries of the aggregate threshold map 344.

[0191] If the crop size is determined to be the input size, the adaptive region cropping module 348 resizes and downsizes the expanded selection to the input size.

[0192] In one embodiment, if the expansion size is equal to the input size, no resizing and / or downsizing occurs. In one embodiment, resizing and / or downsizing is considered to occur even if the expansion size is equal to the input size.

[0193] If the dilation size is determined to be smaller than the input size (i.e., crop size = dilation size according to the above formula), the adaptive region cropping module 348 zero-pads the dilation size (dilation selection) to generate a zero-padded dilation selection equal to the input size.

[0194] The resulting augmented image may be further downsampled. Downsampling may occur in the unlikely case that the abnormal blob is larger (in x, y, or both directions) than the input size to the classifier. In this scenario, the cropped image 350 is resized and reduced to fit that fixed input size.

[0195] The adaptive region cropping module 348 repeats the above functions for each additional anomaly and / or bounding box-defined region in the aggregate threshold map 344 or the annotated inspection image 346 to generate further cropped images 350 .

[0196] When an image such as aggregate threshold map 344 is conventionally cropped, contextual information such as the size and dimension ratio of the cropped anomaly may be lost. This may occur, for example, when inspection image 310 is cropped according to a fixed window (e.g., 224 pixels by 224 pixels) of the model input, or when a more precise crop of the anomaly is made and then resized to the fixed window of the model input.

[0197] Such an expansion size may advantageously provide unusually sufficient context.

[0198] Using the adaptive cropping method of the present disclosure, the size of the anomalies is advantageously kept consistent throughout the adaptive cropping.

[0199] In one embodiment, the output of the adaptive region cropping module 348 may be a set of one or more smaller images, each corresponding to a region of the inspection image containing a potential anomaly. In one embodiment, the output may be a batch of 224x224 images.

[0200] 5, there is shown a schematic diagram of an adaptive cropping process 500 performed on an image according to one embodiment. The adaptive cropping process 500 may be used to generate the cropped image 350. The adaptive cropping process 500 may be performed by the adaptive region cropping module 348.

[0201] The adaptive cropping process 500 begins with the aggregate threshold map 344. The aggregate threshold map 344 includes a target anomaly 502. "Target" refers to the fact that the anomaly 502 is "targeted" for cropping. Providing a better and / or more detailed view of the target anomaly 502 may be the motivation for applying adaptive cropping to the aggregate threshold map 344. The aggregate threshold map 344 further includes anomalies 504, 506 that are not target anomalies and are not visible in the cropped image 350. The anomalies 504, 506 may be target anomalies in the sense that they may be subject to respective cropping operations (similar to process 500).

[0202] The dilated image data 508 of the region containing the target anomaly 502 in the aggregate threshold map 344 is generated according to the following formula: Dilation Size=(16 (長さ-入力) / (0.88×入力) +1) × length.

[0203] 5, the augmented image data 508 is larger than the aggregate threshold map 344. In some embodiments, the augmented image data 508 is smaller than the aggregate threshold map 344 despite being augmented, for example, because only a portion of the aggregate threshold map 344 is augmented. In some embodiments, the augmented image data 508 is the same or approximately the same size as the aggregate threshold map 344, for example, because only a portion of the aggregate threshold map 344 is augmented, and the augmented portion is selected to be the same or approximately the same size as the aggregate threshold map 344.

[0204] The augmented image data 508 provides a larger depiction of the target anomaly 502. The augmented image data 508 provides a more detailed depiction of the target anomaly 502 than is found in the aggregate threshold map 344.

[0205] The cropped image 350 is generated from the augmented image data 508 .

[0206] If the input size of the classifier model 352 is smaller than the dilation size, the cropping size is equal to the input size of the classifier model 352, generating a cropped image 350a.

[0207] If the dilation size is smaller than the input size of the classifier model 352, the cropping size is equal to the dilation size. To provide a cropped image 350 of the same size as the input size of the classifier model 352, zero padding is applied, producing a zero padded region 510 in the generated cropped image 350b.

[0208] Adaptive cropping can provide more background context to the cropped image 350. Adaptive cropping can preserve the dimensions and size of the target anomaly 502 in the cropped image 350. Therefore, classification accuracy can be improved. Without adaptive cropping, context information such as the size and dimension ratio of the cropped anomaly may be lost.

[0209] 3, the cropped image 350 is provided to the image classification module 318 for classification. The classification module 318 may be considered a "pseudo one-class classifier" for reasons explained below and in the text.

[0210] The image classification module 318 includes an image classifier model 352. The classifier model 352 may be a neural network. The neural network may be a convolutional neural network (CNN). The image classifier model 352 may be any suitable classifier model configured to perform image classification. The classifier 352 may be a combination of a convolutional layer (from a pre-trained CNN) and a support vector machine ("SVM") classifier. The SVM classifier may be trained on a small set of project-specific images. With sufficient training samples, such a hybrid CNN may be replaced with a fine-tuned CNN.

[0211] The image classifier model 352 is configured to receive image data as input (i.e., the cropped image 350). In one embodiment, the image classifier model 352 is configured to classify the cropped image 350 (and any anomalies contained therein) as an abnormal deviation, a normal deviation, or a novel deviation. During operation, the image classifier model 352 analyzes the cropped image 350 and determines a preliminary class label 354 for the cropped image 350 and a confidence level 356 for the assignment of the preliminary class label 354. The possible preliminary class labels may include an “OK” label corresponding to an “OK” (or good) class (first label / class) and an “NG” label corresponding to an “NG” (or bad) class (second label / class). The labels may be represented in any suitable format (e.g., string, numeric). The confidence level for the class label assignment may be represented in any suitable format (e.g., a number between 0 and 1, where 0 represents a low confidence level and 1 represents a high confidence level). The preliminary class labels 354 and confidence levels 356 are stored in memory 304 .

[0212] Each preliminary class has an associated confidence threshold, e.g., the OK class has an associated OK class confidence threshold (first confidence threshold), the NG class has an associated NG class threshold (second confidence threshold), etc.

[0213] If the assigned preliminary class label 354 is an OK class label, the classification module 318 compares the corresponding confidence level 356 to an OK class confidence threshold to determine whether the confidence level 356 meets the confidence threshold. If the confidence level 356 meets the OK class confidence threshold, the classification module 318 assigns a final class label 362 of "OK Anomaly" (representing the OK Anomaly class or the Normal Deviation class). If the confidence level 356 does not meet the OK class confidence threshold, the classification module 318 assigns a final class label 362 of "New Anomaly" (representing the "New Anomaly" or "New Deviation" class).

[0214] If the assigned preliminary class label 354 is an NG class label, the classification module 318 compares the corresponding confidence level 356 to the NG confidence threshold to determine whether the confidence level 356 meets the NG class confidence threshold. If the confidence level 356 meets the NG class confidence threshold, the classification module 318 assigns a final class label 362 of "NG Anomaly" (representing the NG Anomaly class or the Anomaly Deviation class). If the confidence level 356 does not meet the NG class confidence threshold, the classification module 318 assigns a final class label 362 of "New Anomaly" (representing the "New Anomaly" or "New Deviation" class). The OK class confidence threshold and the NG class confidence threshold may differ.

[0215] The confidence threshold may be manually set as a parameter by a user (eg, a human expert).

[0216] Accordingly, the classification module 318 compares the confidence level 356 of the assignment of the preliminary class label 354 to an appropriate confidence threshold and assigns a final class label 362. The final class label 362 is stored in the memory 304. There may be more potential final class labels 362 than potential preliminary class labels 354. For example, if there are two possible preliminary class labels 354, there may be three or more possible final class labels 362. In some cases, the class represented by the preliminary class label 354 may be represented as a class in the final class label 362, along with one or more additional classes not represented in the preliminary class labels. By way of example, the preliminary class labels may be OK anomaly and NG anomaly, and the final class labels may be OK anomaly / normal deviation, NG anomaly / abnormal deviation, and new anomaly / new deviation.

[0217] If the preliminary class label 354 is a first class label (OK class label) and the confidence level 356 meets a first confidence threshold (OK class confidence threshold), the cropped image 350 is assigned a first final class label 362 corresponding to the first class (e.g., OK abnormal class / normal deviation class).

[0218] If the preliminary class label 354 is a first class label (OK class label) and the confidence level 356 does not meet the first confidence threshold (OK class confidence threshold), the cropped image 350 is assigned a second final class label 362 corresponding to a second class (e.g., new anomaly class / new deviation class).

[0219] If the preliminary class label 354 is a second class label (NG class label) and the confidence level 356 meets a second confidence threshold (NG class confidence threshold), the cropped image 350 is assigned a third final class label 362 corresponding to a third class (e.g., NG anomaly class / novel deviation class).

[0220] If the preliminary class label 354 is a second class label (NG class label) and the confidence level 356 does not meet the second confidence threshold (NG class confidence threshold), the cropped image 350 is assigned a second final class label 362 corresponding to the second class (e.g., new anomaly class / new deviation class).

[0221] Thresholds, such as the first and second confidence thresholds, may be implemented in any manner and are preferably manually set as parameters by a user (e.g., a human expert). Such thresholds are typically used to indicate assignment to one class or another. In some embodiments, meeting a threshold includes equaling or exceeding the threshold. In some embodiments, meeting a threshold means strictly exceeding the threshold.

[0222] In some embodiments, the evaluation of the preliminary class label 354 with respect to the confidence threshold may be performed by the classifier model 352 (if the classifier model is configured to do so). In other embodiments, the evaluation of the preliminary class label 354 with respect to the confidence threshold may be performed by the classification module 318 using instructions or logic external to the classifier model 352 that are used to process the output of the classifier model 352.

[0223] The processor 302 may then use the final class labels 362 and the position data from the annotated inspection image 346 to generate a second annotated inspection image 364 .

[0224] For example, processor 302 may generate an inspection image (second annotated inspection image 364) in which each region identified for cropping and classification is defined by a bounding box and labeled with a final class label 362. The bounding box data may come from aggregate threshold map 344 or annotated inspection image 346. When displayed in a user interface, second annotated inspection image 364 may identify anomalies in inspection image 310 and their respective assigned classes for a user to review. In some embodiments, processor 302 may be configured to generate a user interface that includes second annotated inspection image 364 and display the user interface via display 308.

[0225] The aforementioned classification module 318 (pseudo one-class classifier) ​​may offer certain advantages over other types of classifiers, such as one-class or binary classifiers. The classifier may enable a multi-class (or binary) classifier to be converted into a true anomaly detector. The classifier may provide reliable detection of well-known defects (if available). The classifier may enable automated inspection to begin without defective parts. In particular, a binary OK vs. NG classifier may miss the detection of new defects, and as a result, anomaly detection systems and algorithms incorporating the same may miss defects that appear to be OK anomalies.

[0226] 6A-6C, a set of cropped images are shown classified by a pseudo-one-class classifier (FIG. 6A) of the present disclosure, a binary classifier (FIG. 6B), and a one-class classifier (FIG. 6C), according to one embodiment. The classification outputs shown in FIGS. 6A-6C may be generated by classification module 318 of FIG. 3.

[0227] Referring first to Figure 6A, Figure 6A shows classification results 600a for cropped images 602-1 through 602-20. The classifier in Figure 6A first classifies images 602-1 through 602-12 into an OK class 604 (preliminary class label) with an associated confidence level 606a, and classifies images 602-13 through 602-20 into an NG class 608 (preliminary class label) with an associated confidence level 606b. Images 602-1 through 602-20 are collectively and generically referred to as images 602.

[0228] Images classified in the OK class 604 are further evaluated with respect to a first confidence threshold 610. Images 602 assigned to the OK class 604 with a confidence level below the threshold 610 are assigned to the novel anomaly (novel deviation) class 612 (final class label). Images 602 assigned to the OK class 604 with a confidence level that meets the threshold 610 are assigned to the OK anomaly (normal deviation) class 614.

[0229] Images 602 assigned to the NG class 608 are further evaluated with respect to a second confidence threshold 616. Images 602 assigned to the NG class 608 with a confidence level below the threshold 616 are assigned to a new anomaly (new deviation) class 612 (final class label). Images 602 assigned to the NG class 608 with a confidence level that meets the second threshold 616 are assigned to the NG anomaly (abnormal deviation) class 615.

[0230] Referring to Figure 6B, the same image 602 has been classified using a binary classifier: As in Figure 6A, images 602-1 through 602-12 are classified into OK class 604 with confidence level 606a, and images 602-13 through 602-20 are classified into NG class 608 with confidence level 606b.

[0231] The subset of images 602 classified into NG class 608 forms group 617. These images 602 fall within the high confidence level range of the NG class and are therefore anomalous deviations. Note that the binary classifier in Figure 6B does not have a criterion for how to handle data points with confidence levels below a threshold.

[0232] Referring to Figure 6C, the same images 602 as in Figures 6A and 6B are classified using a one-class classifier. All images 602 are classified into the OK class 604 with a confidence level 606a. The classified images 602 form two groups 630, 632. Group 630 is the image classified into the OK class 604 with a low confidence level, and group 632 is the image classified into the OK class 604 with a high confidence level. Note that Figure 6C is a typical one-class classifier that can report all samples outside of one class as anomalous.

[0233] The pseudo one-class classifier 352 used in FIG. 6A effectively distinguishes between new anomalies and known anomaly categories, as well as between known anomaly categories. Traditional binary classifiers cannot distinguish between new anomalies and therefore inevitably classify all anomalies into one of two categories. Traditional one-class classifiers, by definition, cannot distinguish between multiple anomaly classes and therefore classify all anomalies into a single known category and a new category. The pseudo one-class classifier 352 effectively overcomes the limitations and shortcomings of known classifiers in providing at least the aforementioned capabilities.

[0234] The pseudo one-class classifier 352 may be created through the transformation of a multi-class or binary classifier. Advantageously, the pseudo one-class classifier 352 can begin autonomous inspection without any existing samples of defective parts. The amount of good data provided as a sample may vary depending on how variable a machine part without deviations is.

[0235] Referring again to FIG. 3, in some embodiments, the output or data generated by the computer system 300 may be combined or used with other AI vision inspection data generated from the same inspection image 310.

[0236] For example, in one embodiment, the inspection image 310 may be input to an object detection component that includes an object detection model configured to detect and classify objects within the inspection image 310. The detected objects may be described by position data (e.g., bounding boxes) and a class label (e.g., defect type or class) that identify the detected objects within the inspection image 310. The data describing the detected objects may then be compared to data describing the detected anomalies via the computer system 300 (e.g., comparing position data such as bounding box coordinates to determine overlap between outputs). In some cases, only anomalies with a particular final class label 362 may be compared to the detected objects. This comparison may enable confirmation of the detected objects using the anomaly detection output (i.e., to confirm the presence of a defect). In some embodiments, the object detection and comparison of the object detection output to the anomaly detection output may be performed by the computer system 300. The computer system 300, and the systems and methods of the present disclosure generally, may be used as part of an AI vision inspection system, such as that described in PCT Application No. PCT / CA2022 / 050100, the contents of which are incorporated herein by reference.

[0237] Referring to Figure 7, there is shown an example anomaly detection pipeline 700 executed by computer system 300 of Figure 3. There may be additional steps and outputs that are not shown.

[0238] The inspection image 310 is provided to an image subtraction module 320 for comparison with a golden sample image 312. The golden sample image 312 is generated by inputting the inspection image 310 into a generative model 702.

[0239] The subtracted image 322 is provided to a shape analysis and binarization module 324 .

[0240] The SAB output 326 is provided to an aggregate threshold module 328. Shown are a processed primary threshold map 340 and a secondary threshold map 342. The processed primary and secondary maps 340, 342 are aggregated and an aggregate threshold map 344 (not shown) is used by an adaptive region cropping module 348.

[0241] The adaptive region cropping module 348 performs adaptive cropping to generate a cropped image (cropped anomaly) 350. In this particular case, the cropped image 350 is zero-padded. The cropped image 350 is provided to a classification module 318 that includes a pseudo-one-class classifier 352. A second annotated inspection image 364 is generated that includes the bounding box data determined from the aggregation threshold and the final class label 362 determined by the classification module 318.

[0242] Referring to FIG. 8, a method 800 for visual inspection is shown.

[0243] In step 802, the method 800 includes acquiring an inspection image. The inspection image may be inspection image 310.

[0244] In step 804, the method 800 includes generating a golden sample image from the inspection image. The golden sample image may be golden sample image 312.

[0245] In step 806, method 800 includes performing an image subtraction operation on the inspection image and the golden sample image to obtain a subtracted image. The subtracted image may be subtracted image 322.

[0246] In step 808, the method 800 includes performing an aggregate thresholding process on the subtracted image to generate an aggregate threshold image for identifying anomalies. The aggregate threshold map may be aggregate threshold map 344. The aggregate threshold map may include a bounding box surrounding each anomaly in the image.

[0247] In step 810, the method 800 includes performing adaptive cropping on the anomaly map to obtain a cropped image of the anomaly in the anomaly map. The cropped image may be cropped image 350 in FIG.

[0248] In step 812, the method includes classifying the cropped image with a pseudo one-class classifier, which may be classification module 318 or classifier model 352 of FIG.

[0249] Referring to Figure 9, a method 900 of aggregate thresholding is shown. Method 900 may be used as part of a machine vision anomaly detection method as described herein. Method 900 may be performed by computer system 300 of Figure 3. In particular, method 900 may be performed by aggregate threshold module 328 of Figure 3.

[0250] In step 902, the method 900 includes providing an anomaly map generated from a comparison of the inspection image and the golden sample image. The anomaly map may be the subtracted image 322 of Figure 3. This comparison may include performing image subtraction.

[0251] In step 904, the method 900 includes performing a first image thresholding operation on the anomaly map using a first threshold value to obtain a primary threshold map.

[0252] In step 906, the method 900 includes processing the primary threshold map to obtain a processed primary threshold map.

[0253] In step 908, the method 900 includes performing a second image thresholding on the anomaly map using a second threshold to obtain a secondary threshold map.

[0254] The first threshold may be a conservative threshold and the second threshold may be an aggressive threshold.

[0255] At step 910, the method 900 includes aggregating the primary and secondary threshold maps according to a set of one or more aggregation rules to obtain an aggregated threshold map. In one embodiment, the aggregation rules include retaining in the aggregated threshold map blobs present in the secondary threshold map that overlap with blobs present in the processed primary threshold map and excluding all other blobs present in the processed primary or secondary threshold maps.

[0256] Referring to Figure 10, a method 1000 of adaptive image region cropping is shown. Method 1000 may be used as part of a machine vision anomaly detection process. Method 1000 may be performed by computer system 300 of Figure 3. In particular, method 1000 may be performed by adaptive region cropping module 348 of Figure 3.

[0257] In step 1002, method 1000 includes providing an aggregate threshold map. The aggregate threshold map includes one or more bounding box-defined regions that contain each anomaly (detected via image subtraction and thresholding). The aggregate threshold map may be aggregate threshold map 344 of FIG. 3. The aggregate threshold map may be provided by method 900.

[0258] In step 1004, the method 1000 includes applying an expansion factor to the region defined by the bounding box of the aggregate threshold map to obtain an expanded selection having an expanded size.

[0259] In step 1006, the method 1000 includes determining whether the input size of the image classification model is less than or equal to the augmented size.

[0260] In step 1008, method 1000 branches based on whether the input size of the classifier is less than or equal to the augmented size. If the input size is less than or equal to the augmented size, method 1000 proceeds to step 1012. If the input size is not less than or equal to the augmented size, method 1000 proceeds to step 1018.

[0261] In some embodiments, after step 1008, method 1000 may include performing collision avoidance using the extended selection. Collision avoidance may be performed between steps 1008 and 1012 of method 1000.

[0262] In step 1012, the method 1000 includes resizing and downsizing the augmented selection to the input size of the image classification model.

[0263] In step 1014, the method 1000 includes providing the resized and / or downsized expanded selection.

[0264] In step 1018, the method 1000 includes zero-padding the expanded selection to the input size of the image classification model.

[0265] In step 1020, the method 1000 includes providing a zero-padded extension selection.

[0266] After step 1014 or 1020, the method 1000 proceeds to step 1016. In step 1016, the method includes repeating steps 1004-1020 for the region defined by each additional bounding box in the aggregate threshold map.

[0267] Referring to Figure 11, a method 1100 for classifying an image is shown. Method 1100 may be used as part of a machine vision anomaly detection process. Method 1100 may be performed by computer system 300 of Figure 3. In particular, method 1100 may be performed by image classification module 318 or classifier model 352 of Figure 3.

[0268] In step 1102, the method 1100 includes providing a cropped image. The cropped image may be cropped image 350. The cropped image may be generated by the method 1000 of FIG.

[0269] In step 1104, the method 1100 includes using a classifier model to determine a preliminary class label for the cropped image and a confidence level for the preliminary class label determination.

[0270] In step 1106, the method 1100 branches based on whether a first class label or a second class label was assigned by the classifier.

[0271] If the first label is assigned (eg, the OK class), the method 1100 proceeds to step 1108 .

[0272] If a second label is assigned (eg, the NG class), the method 1100 proceeds to step 1116 .

[0273] In step 1108, the method 1100 includes comparing the confidence level of the preliminary class label determination to a first confidence level threshold.

[0274] In step 1110, method 1100 branches based on whether the trust level meets a first trust level threshold. If the trust level meets the first trust level threshold, method 1100 proceeds to step 1112. If the trust level does not meet the first trust level threshold, method 1100 proceeds to step 1114.

[0275] In step 1112, the method 1100 includes assigning a final class label indicating the first class assignment (eg, an OK abnormal class).

[0276] At 1114, the method 1100 includes assigning a final class label indicating the new class assignment (eg, the new abnormal class).

[0277] At 1116, the method 1100 includes comparing the confidence level of the preliminary class label determination to a second confidence level threshold.

[0278] At 1118, method 1100 branches based on whether the trust level meets a second trust level threshold. If the trust level meets the second trust level threshold, method 1100 proceeds to 1120. If the trust level does not meet the second trust level threshold, method 1100 proceeds to 1114.

[0279] At 1120, the method 1100 includes assigning a final class label indicating the second class (e.g., the NG anomaly class).

[0280] At 1114, the method 1100 includes assigning a final class label indicating the new class assignment (eg, the new abnormal class).

[0281] Following one or more of methods 700, 800, 900, 1000, and / or 1100, post-processing may occur based on the output. For example, parts with anomalies detected and / or confirmed in aggregate threshold map 344 of FIG. 3 may be discarded automatically or by a human operator. Additionally, computer resources may be allocated according to cropped image 350, and the allocation may be made or modified automatically or by a human operator. Furthermore, if the classification of an anomaly according to the final class label in method 1100 of FIG. 11 belongs to the first class, the second class, or a new class, a computer system or device implementing method 1100 (such as computer system 300 of FIG. 3) or a human operator reviewing the classification may not further review the part with the classified anomaly or may flag it for further review.

[0282] As a further example of post-processing, an anomaly detection device (such as computer system 300) may send data regarding the detected anomaly to a control device (not shown). The control device may generate and further send a control signal to a physical processing component or physical device. The physical processing component may perform post-processing based on the received control signal. The physical processing component may perform all or part of the post-processing unless it receives a control signal. Thus, receiving a control signal may advantageously improve the efficiency of a computer implementing the aforementioned function and provide a practical application of the function in that the aforementioned function reduces computer processing.

[0283] As a further example of post-processing, a physical processing component may be a physical device (e.g., a robot) that performs one action upon receiving a first control signal. Upon receiving a second control signal, the physical device may perform a second action or a different action instead of the second action. Such actions may include refraining from certain actions. For example, a physical device may transport or allow a physical part to be transported upon receiving a first control signal indicating that all abnormalities detected within the part are of the "OK" class, but may instead physically discard the part upon receiving a second control signal indicating that the abnormalities detected within the part are of the "NG" class. After transporting or discarding the part, the physical device may flag the part for further review upon receiving a third control signal in addition to or instead of the first or second control signal. The third control signal may indicate that the abnormality detected within the part is of a new class (i.e., not "OK" or "NG"). The physical device may receive a control signal for each part. The physical device may receive a control signal for each abnormality on or within each part. The physical device may continue to perform an action (or refrain from an action) according to the last received control signal until receiving a further or different control signal. For example, after receiving a second control signal to discard a part, the robot may continue to discard parts until receiving a further or different control signal.

[0284] In some embodiments, the anomaly detection device, the control device, and the physical device are all separated from one another. Any of the anomaly detection device, the control device, and the physical device may be located remotely from one another. For example, both the anomaly detection device and the physical device may be physically located near the component, and the control device may be located remotely from the anomaly detection device and the physical device. For example, the anomaly detection device, the control device, and the physical device may all be physically located near the component.

[0285] While the above description provides examples of one or more devices, methods, or systems, it will be understood that other devices, methods, or systems may be included within the scope of the claims as interpreted by one skilled in the art.

Claims

1. 1. A system for inspecting an inspection image, comprising: a memory for receiving or storing the inspection image; a golden sample generator for generating a golden sample image from the inspection image; an image subtraction module that generates a subtraction image using the inspection image and the golden sample image; an aggregate thresholding module that generates an aggregate threshold image for identifying anomalies; an adaptive region cropping module that obtains a cropped image of the anomaly; a cropped image classification module that classifies the cropped image using a pseudo one-class classifier; Including, the system.

2. The system of claim 1 , further comprising a camera configured to capture the inspection image.

3. The system of claim 1 or claim 2, wherein the inspection image shows a part or object under inspection, or a portion of an area thereof.

4. The system of any one of claims 1 to 3, wherein the inspection image is part of a video captured by a camera device.

5. The system of any one of claims 1 to 4, wherein the inspection image is used to analyze the presence of defects.

6. The system of any one of claims 1 to 5, further comprising an adaptive ROI segmentation module for masking the inspection image and the golden sample image.

7. The system of any one of claims 1 to 6, further comprising a shape analysis and binarization module for receiving the subtracted image and generating a shape analyzed and binarized image.

8. 8. The system of claim 7, wherein the shape analysis and binarization module performs binarization and binary image processing to remove defects and anomalies in the subtraction image that are smaller than a specified size and / or caused by minor surface texture variations.

9. The system of claim 8 , wherein the binary image processing includes erosion and dilation.

10. 1. A method of inspecting an inspection image, comprising: acquiring the inspection image; generating a golden sample image from the inspection image; performing an image subtraction operation on the inspection image and the golden sample image to obtain a subtraction image; performing aggregate thresholding on the subtraction image to generate an aggregate threshold image for identifying anomalies; adaptively cropping the aggregate threshold image to obtain a cropped image of the anomaly; classifying the cropped image of the anomaly with a pseudo one-class classifier; method.

11. The method of claim 10 further comprising annotating the aggregate threshold map.

12. The method of claim 10 or claim 11, further comprising the step of discarding parts containing anomalies detected and / or confirmed in the aggregate threshold image.

13. The method of any one of claims 10 to 12, wherein the aggregate threshold map comprises a bounding box surrounding each anomaly.

14. The method of claim 13 , further comprising identifying each region defined by each of the bounding boxes.

15. 1. A device for inspecting an inspection image, comprising: a memory for receiving or storing the inspection image; a golden sample generator for generating a golden sample image from the inspection image; an image subtraction module for generating a subtraction image from the inspection image and the golden sample image; an aggregate thresholding module for generating an aggregate threshold image for identifying anomalies; an adaptive region cropping module for obtaining a cropped image of the anomaly; a cropped image classification module for classifying the cropped image with a pseudo one-class classifier; Including, the device.

16. The device of claim 15 , wherein the golden sample generator comprises a generative model.

17. The device of claim 16 , wherein the generative model is an autoencoder.

18. 20. The device of claim 17, wherein the autoencoder includes an encoder component for compressing the inspection image to generate code components, and a decoder component for reconstructing the inspection image using the code components.

19. The device of claim 15 , wherein the golden sample generator obtains an appropriate pre-stored golden sample image.

20. The device of claim 15 , wherein the golden sample generator receives the golden sample image.

21. 1. A system for aggregate thresholding, comprising: an image subtraction module for generating an anomaly map from a comparison of the inspection image and the golden sample image; a primary threshold map generator for applying a first threshold to the anomaly map to obtain a primary threshold map; a processed primary threshold map generator for processing the first threshold map to obtain a processed primary threshold map; a secondary threshold map generator for applying a second threshold to the anomaly map to obtain a second threshold map; an aggregate threshold map generator for aggregating the first threshold map and the second threshold map according to a set of one or more aggregation rules to obtain an aggregate threshold map; Including, the system.

22. 22. The system of claim 21, further comprising a camera configured to capture the inspection image.

23. 23. The system of claim 21 or claim 22, further comprising an adaptive ROI segmentation module for masking the inspection image and the golden sample image.

24. 24. The system of claim 21, further comprising a shape analysis and binarization module for receiving a subtraction image and performing binarization and binary image processing on the subtraction image to remove defects and anomalies smaller than a specified size and / or caused by minor surface texture variations, wherein the binary image processing includes erosion and dilation.

25. The system of any one of claims 21 to 24, wherein the first threshold is set by a user.

26. The system of any one of claims 21 to 25, wherein the second threshold is set by a user.

27. The system of any one of claims 21 to 26, wherein the first threshold is more conservative than the second threshold.

28. 1. A method for aggregate thresholding, comprising: Provides an anomaly map generated from a comparison of the inspection image and the golden sample image, performing a first image thresholding operation on the anomaly map using a first threshold to obtain a first threshold map; processing the first threshold map to obtain a processed first threshold map; performing a second image thresholding operation on the anomaly map using a second threshold to obtain a second threshold map; aggregating the first threshold map and the second threshold map according to a set of one or more aggregation rules to obtain an aggregate threshold map. method.

29. 30. The method of claim 28, further comprising discarding parts containing anomalies detected and / or confirmed in the aggregate threshold image.

30. 30. The method of claim 28 or claim 29, wherein the aggregate threshold map comprises a bounding box surrounding each anomaly.

31. The method of any one of claims 28 to 30, wherein the first threshold is set by a user.

32. The method of any one of claims 28 to 31, wherein the second threshold is set by a user.

33. The method of any one of claims 28 to 32, wherein the first threshold is more conservative than the second threshold.

34. 1. A device for aggregate thresholding, comprising: an image subtraction module for generating an anomaly map from a comparison of the inspection image and the golden sample image; a primary threshold map generator for applying a first threshold to the anomaly map to obtain a primary threshold map; a processed primary threshold map generator for processing the first threshold map to obtain a processed primary threshold map; a secondary threshold map generator for applying a second threshold to the anomaly map to obtain a second threshold map; an aggregate threshold map generator for aggregating the first threshold map and the second threshold map according to a set of one or more aggregation rules to obtain an aggregate threshold map; Including, the device.

35. 35. The device of claim 34, wherein the golden sample image generator includes a generative model, the generative model being an autoencoder including an encoder component for compressing the inspection image to generate code components and a decoder component for reconstructing the inspection image using the code components.

36. 35. The device of claim 34, wherein the golden sample image generator obtains a suitable pre-stored golden sample image.

37. 35. The device of claim 34, wherein a golden sample image generator receives the golden sample image.

38. A device according to any one of claims 34 to 37, wherein the first threshold is set by a user.

39. A device according to any one of claims 34 to 38, wherein the second threshold is set by a user.

40. A device according to any one of claims 34 to 39, wherein the first threshold is more conservative than the second threshold.

41. 1. A system for adaptive region cropping, comprising: an adaptive cropping region module for receiving an aggregate threshold map based on the inspection image; The adaptive cropping region module is included in or implemented in a processor and memory, and computer-executable instructions stored in the memory cause the processor, when executed, to: For each bounding box defined area of ​​the aggregate threshold map: applying an expansion factor to the area defined by the bounding box to obtain an expanded selection having an expanded size; When the image classification model input size of the image classification model is equal to or less than the expansion size, resizing and downsizing the augmented selection to the image classification model input size; providing the resized and / or downsized augmented selection as an input to the image classification model; if the image classification model input size is not less than or equal to the augmented size, zero-pad the expanded selection to the image classification model input size; providing the zero-padded augmented selection as an input to the image classification model; A system that executes processing.

42. The expansion coefficient is Expansion size = (16 （長さ-入力）／（０．８８×入力） +1) x length and "Input" is the image classification model input size; 42. The system of claim 41.

43. Resizing and downsizing Formula: Cropping size = min(input, extension size) Proceed according to "Input" is the image classification model input size; 43. A system according to claim 41 or claim 42.

44. The system of any one of claims 41 to 43, further comprising a collision avoidance module for constraining cropping coordinates to stay within the inspection image.

45. 45. The system of claim 41, further comprising a downsampling module for downsampling the aggregate threshold map if abnormal blobs indicated in the aggregate threshold map are larger than the image classification model input size.

46. 46. ​​The system of claim 45, wherein the aggregate threshold map comprises a bounding box surrounding each abnormal blob.

47. The system of any one of claims 41 to 46, wherein the inspection image is taken by a camera.

48. 1. A method for adaptive region cropping, comprising: providing an aggregate threshold map; For each bounding box defined region of the aggregate threshold map based on the test image, applying an expansion factor to obtain an expansion selection having an expansion size; When the image classification model input size of the image classification model is equal to or less than the expansion size, resizing and downsizing the augmented selection to the image classification model input size; providing the resized and / or downsized augmented selection as an input to the image classification model; if the image classification model input size is not less than or equal to the augmented size, zero-pad the expanded selection to the image classification model input size; providing the zero-padded augmented selection as an input to the image classification model; method.

49. The expansion coefficient is Expansion size = (16 （長さ-入力）／（０．８８×入力） +1) x length and "Input" is the image classification model input size; 49. The method of claim 48.

50. Resizing and downsizing Formula: Cropping size = min(input, extension size) Proceed according to "Input" is the image classification model input size; 50. The method of claim 48 or claim 49.

51. and performing collision avoidance by constraining cropping coordinates to remain within the inspection image. The method according to any one of claims 48 to 50.

52. and downsampling the aggregate threshold map if an abnormal blob indicated in the aggregate threshold map is larger than the image classification model input size. A method according to any one of claims 48 to 51.

53. 53. The method of claim 52, wherein the aggregate threshold map comprises a bounding box surrounding each abnormal blob.

54. A method according to any one of claims 48 to 53, wherein the inspection image is taken by a camera.

55. 1. A device for adaptive region cropping, comprising: an adaptive cropping region module for receiving an aggregate threshold map based on the inspection image; The adaptive cropping region module is included in or implemented in a processor and memory, and is configured by computer-executable instructions stored in the memory that, when executed, cause the processor to: For each bounding box defined area of ​​the aggregate threshold map: applying an expansion factor to obtain an expansion selection having an expansion size; When the image classification model input size of the image classification model is equal to or less than the expansion size, resizing and downsizing the augmented selection to the image classification model input size; providing the resized and / or downsized augmented selection as an input to the image classification model; If the image classification model input size is not equal to or less than the expansion size, zero-pad the expanded selection to the image classification model input size; providing the zero-padded augmented selection as an input to the image classification model; A device that performs processing.

56. The expansion coefficient is Expansion size = (16 （長さ-入力）／（０．８８×入力） +1) x length and "Input" is the image classification model input size; 56. The device of claim 55.

57. Resizing and downsizing Formula: Cropping size = min(input, extension size) Proceed according to "Input" is the image classification model input size; 57. A device according to claim 55 or claim 56.

58. a collision avoidance module for constraining cropping coordinates to remain within the inspection image. A device according to any one of claims 55 to 57.

59. 59. The device of claim 55, further comprising a downsampling module for downsampling the aggregate threshold map if abnormal blobs indicated in the aggregate threshold map are larger than the image classification model input size.

60. the aggregate threshold map includes a bounding box surrounding each abnormal blob; 60. The device of claim 59.

61. 1. A system for classifying an input image according to a pseudo-one-class classifier, comprising: a crop image classification module including a classifier model; The crop image classification module is included in or implemented in a processor and memory by computer-executable instructions stored in the memory that, when executed, cause the processor to: Receive the cropped image, determining a preliminary class label for the cropped image and a confidence level associated with determining the preliminary class label using the classifier model; If the preliminary class label is the first class, comparing the confidence level of the preliminary class label determination to a first confidence threshold; If the confidence level satisfies the first confidence threshold, assigning a final class label to the cropped image that indicates the first class; If the confidence level does not satisfy the first confidence threshold, assigning a final class label to the cropped image that indicates a new class; If the preliminary class label is the second class, comparing the confidence level of the preliminary class label determination to a second confidence threshold; If the confidence level satisfies the second confidence threshold, assigning a final class label to the cropped image that indicates the second class; If the confidence level does not satisfy the second confidence threshold, assigning a final class label to the cropped image that indicates a new class; A system that executes processing.

62. 62. The system of claim 61, wherein the first confidence threshold is set by a user.

63. 63. The system of claim 61 or claim 62, wherein the second confidence threshold is set by a user.

64. 64. The system of any one of claims 61 to 63, wherein the image classifier model is a convolutional neural network.

65. the first class represents normal deviations; the second class represents an abnormal deviation; The novel class represents a novel deviation. A system according to any one of claims 61 to 64.

66. The confidence level satisfying the confidence threshold includes the confidence level being equal to the confidence threshold, and further includes the confidence level being greater than the confidence threshold. A system according to any one of claims 61 to 65.

67. The confidence level meeting the confidence threshold includes the confidence level exceeding the confidence threshold, but does not include the confidence level being equal to the confidence threshold. A system according to any one of claims 61 to 65.

68. 1. A method for classifying an input image according to a pseudo one-class classifier, comprising: Provide a cropped image, determining a preliminary class label and a confidence level associated with determining the preliminary class label for the cropped image using a classifier model; If the preliminary class label is the first class, comparing the confidence level of the preliminary class label determination to a first confidence threshold; If the confidence level satisfies the first confidence threshold, assigning the cropped image a first final class label indicative of the first class; If the confidence level does not satisfy the first confidence threshold, assigning a second final class label to the cropped image that indicates a new class; If the preliminary class label is the second class, comparing the confidence level of the preliminary class label determination to a second confidence threshold; If the confidence level satisfies the second confidence threshold, assigning the cropped image a third final class label indicative of the second class; If the confidence level does not satisfy the second confidence threshold, assigning the second final class label to the cropped image, the second final class label indicating a new class; method.

69. 69. The method of claim 68, wherein the first confidence threshold is set by a user.

70. 70. The method of claim 68 or claim 69, wherein the second confidence threshold is set by a user.

71. The confidence level satisfying the confidence threshold includes the confidence level being equal to the confidence threshold, and further includes the confidence level being greater than the confidence threshold. A method according to any one of claims 68 to 70.

72. The confidence level meeting the confidence threshold includes the confidence level exceeding the confidence threshold, but does not include the confidence level being equal to the confidence threshold. A method according to any one of claims 68 to 70.

73. The method of any one of claims 68 to 72, further comprising generating a second annotated inspection image using the final class labels.

74. The method of any one of claims 68 to 73, further comprising flagging parts for review based on the final class labels.

75. 1. A device for classifying an input image according to a pseudo-one-class classifier, comprising: a crop image classification module including a classifier model; The crop image classification module is included in or implemented in a processor and memory, and is configured with computer-executable instructions stored in the memory that, when executed, cause the processor to: Receive the cropped image, determining a preliminary class label and a confidence level associated with determining the preliminary class label for the cropped image using a classifier model; If the preliminary class label is the first class: comparing the confidence level of the preliminary class label determination to a first confidence threshold; If the confidence level satisfies the first confidence threshold, assigning a final class label to the cropped image that indicates the first class; If the confidence level does not satisfy the first confidence threshold, assigning a final class label to the cropped image that indicates a new class; If the preliminary class label is the second class, comparing the confidence level of the preliminary class label determination to a second confidence threshold; If the confidence level satisfies the second confidence threshold, assigning a final class label to the cropped image that indicates the second class; If the confidence level does not satisfy a second confidence threshold, assigning a final class label to the cropped image that indicates the new class; A device that performs processing.

76. 76. The device of claim 75, wherein the first confidence threshold is set by a user.

77. 77. A device according to claim 75 or claim 76, wherein the second confidence threshold is set by a user.

78. The confidence level satisfying the confidence threshold includes the confidence level being equal to the confidence threshold, and further includes the confidence level being greater than the confidence threshold. A device according to any one of claims 75 to 77.

79. The confidence level meeting the confidence threshold includes the confidence level exceeding the confidence threshold, but does not include the confidence level being equal to the confidence threshold. A device according to any one of claims 75 to 78.

80. A device according to any one of claims 75 to 79, wherein the image classifier model is a convolutional neural network.

Images