Computer system and method for automated visual inspection using segmentation-based anomaly detection

Abstract

Systems, and methods, and devices for segmentation-based anomaly detection are provided. The method includes providing image data to a segmentation model, receiving segment data from the segmentation model, and providing segment data to a defect model as an input, receiving defect data from the defect model as an output, and providing defect data and segment data to an anomaly model as an input and receiving anomaly data from the anomaly model as an output.

Description

COMPUTER SYSTEM AND METHOD FOR AUTOMATED VISUAL INSPECTION USING SEGMENTATION-BASED ANOMALY DETECTIONTechnical Field

[0001] The following relates generally to systems and methods for automated visual inspection of objects, and more particularly to systems and methods for anomaly detection within images, using segmentation models.Introduction

[0002] Current methods and systems for anomaly detection are limited in functionality and may provide for relatively poor or unreliable performance. Many limitations may be exacerbated by inherent limitations in the availability of and ease of acquiring training data.

[0003] Accordingly, there is a need for an improved system and method for anomaly detection that overcomes at least some of the disadvantages of existing systems and methods.Summary

[0004] Systems, methods, and devices for segmentation-based anomaly detection are provided. The method includes performing image segmentation on a digital image using a segmentation model to obtain a segmentation output. The segmentation output includes segment location data localizing each segment in the digital image. The method further includes performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output. The defect detection output includes defect location data localizing each defect in the digital image. The method further includes comparing the segment location data to the defect location data to identify any segments that do not correspond to a detected defect. Segments that do not correspond to a detected defect are identified as anomalies or anomalous regions in the digital image.

[0005] In an embodiment, the segmentation model comprises a neural network.

[0006] In an embodiment, the segmentation model is an instance segmentation model.

[0007] In an embodiment, an anomaly is a detected condition or state that is deemed abnormal but does not align with a defect type that the defect model has been trained to detect.

[0008] In an embodiment, the method further comprises training the defect model to detect the segment as a new defect type.

[0009] In an embodiment, the method further includes grouping similar anomalies into one or more groups or clusters using an unsupervised machine learning technique and training the defect model with the anomaly data corresponding to the one or more groups or clusters.

[0010] In an embodiment, the segment location data of a segment includes a bounding box, a contour, or a pixel map.

[0011] In an embodiment, the segmentation output further includes segment class data for each detected segment and a segment image crop including the image data lying within the defined segment region of the digital image.

[0012] In an embodiment, the defect model is a neural network.

[0013] In an embodiment, the defect model is an instance segmentation model.

[0014] In an embodiment, the defect data includes defect class data identifying a defect class of the defect.

[0015] In an embodiment, performing defect detection on the digital image includes selecting the defect detection model from a plurality of defect detection models based on a segment class of a first segment in the segmentation output, the defect detection model trained to detect defects in that segment class.

[0016] In an embodiment, the plurality of defect detection models include a first defect detection model and a second defect detection model that use different computer vision techniques to perform defect detection.

[0017] In an embodiment, the anomaly or anomalous region is used to train a new defect detection model or retrain the defect detection model.

[0018] In an embodiment, the digital image includes views of one or more surfaces or regions of interest of an inspection object being visually inspected.

[0019] In an embodiment, comparing the segment location data to the defect location data includes using golden segment data.

[0020] A system for segmentation-based anomaly detection is also provided. The system includes a memory for storing data including a digital image and at least one processor in communication with the memory. The processor is configured to execute a method comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data localizing each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data localizing each defect in the digital image; comparing the segment location data to the defect location data to identify any segments that do not correspond to a detected defect; and identifying each segment that does not correspond to a detected defect as an anomaly or anomalous region.

[0021] In an embodiment, the segmentation model is an instance segmentation model.

[0022] Also provided is a least one non-transitory computer-readable storage medium storing processor-executable instructions that when executed by at least one processor, cause the at least one processor to perform a method of segmentation-based anomaly detection, the method comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data localizing each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data localizing each defect in the digital image; comparing the segment location data to the defect location data toidentify any segments that do not correspond to a detected defect; and identifying each segment that does not correspond to a detected defect as an anomaly or anomalous region.

[0023] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0024] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0025] Figure 1 is a schematic diagram of a networked computer system for segmentation-based anomaly detection, according to an embodiment;

[0026] Figure 2 is a block diagram of a computer device, according to an embodiment;

[0027] Figure 3 is a system block diagram of an anomaly detection system, according to an embodiment;

[0028] Figure 4 is a schematic diagram of data associated with the anomaly detection system of Figure 3, according to an embodiment;

[0029] Figure 5 is a diagram of an example image of a golden sample associated with the anomaly detection system of Figure 3, according to an embodiment;

[0030] Figure 6 is a diagram of an example image of an inspection object processed by the anomaly detection system of Figure 3, according to an embodiment;

[0031] Figure 7 is a diagram of an example image of an inspection object processed by the anomaly detection system of Figure 3, according to an embodiment;

[0032] Figure 8 is a diagram of an example image of an inspection object processed by the anomaly detection system of Figure 3, according to an embodiment; and

[0033] Figure 9 is a flowchart of a method of segmentation-based anomaly detection, according to an embodiment.Detailed Description

[0034] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0035] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0036] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0037] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0038] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. Inother words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0039] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0040] The following relates generally to an automated method of visual inspection, and more particularly, to methods and systems directed to a segmentation-based machine learning model system, which may detect defects and anomalies of an object by analyzing images of the object.

[0041] The system and methods described herein may function as follows: a set of inspection objects may be provided. Each inspection object may be of the same object type and but may vary in condition. For example, each inspection object may be a camshaft of the same specification, but each inspection object may be a separate physical object, wherein 100 separate physical camshafts may be inspected. Each separate physical object is inherently of a different condition and / or character, as each physical object is unique.

[0042] Digital images may be captured of each inspection object. This may include imaging all surfaces and portions of interest, or some subset thereof.

[0043] Inspection object images may be provided to the anomaly detection system. In particular, these inspection images may be provided as input to a segmentation model. The segmentation model may be an instance segmentation model. The segmentation model may output bounding boxes, contours or pixel maps around or over known regions, “segmenting” the image into discrete areas. This segmentation information may be stored or organized as segment data (or segmentation output data).

[0044] Next, inspection object images may be subsequently provided to a defect model. Segment data may also be provided to the defect model. The defect model may output defect data, which locates defects, and classifies located defects. Defects may include porosity, scratches, cracks, tolerancing issues, or other defects.

[0045] Subsequently, inspection images, segment data, and defect data may be provided to an anomaly detection module model or anomaly model. The anomaly detection module may compare the segment data and the defect data. In doing so, the anomaly detection module may locate areas within the inspection image with segment data that does not correspond to any defects as per the defect data (“irregular segment data”). The anomaly detection module may flag these areas as including an “anomaly”, and a description of these areas may be stored as anomaly data.

[0046] An anomaly may be defined as a detected condition or state that is deemed abnormal but does not align with any known defect and / or detected defect. For example, an anomaly may comprise an object defect which has not yet been identified, classified, and / or contemplated, or a measurement, equipment or process error. For example, anomalies may correspond to poorly calibrated equipment, or the presence of dust, debris, insects, or other foreign matter in the inspection space.

[0047] In examples where anomalies correspond to defects which have not yet been identified, classified, and / or contemplated, the detection of such anomalies may be applied to train the defect model to detect these defects that have not been explicitly trained for detection by the defect model.

[0048] For example, a certain form of porosity may be present in the image. This may not be a trained defect, as such a defect may not have been seen before. The systems and methods described herein may detect this novel defect as an anomaly, by the application of segmentation methods described above. Similar anomalies in other locations or inspection objects may be grouped together (manually, or using machine learning methods e.g. methods for the unsupervised grouping of images). Once grouped together, this group or set of anomaly data corresponding to a similar novel defect may be then used to further train the defect model, such that the system may detect this novel defect as a known defect instead of an anomaly.

[0049] The methods and systems described herein may advantageously detect novel defects which have not been particularly contemplated or trained for. This may be especially advantageous when applying smaller training sets to the models described herein, as rare defects are less likely to be present in a smaller sample size.

[0050] Models described above may refer to machine learning models, such as neural networks (e.g. convolutional neural networks).

[0051] Referring now to Figure 1 , shown therein is a segmentation-based anomaly detection system 10, in accordance with an embodiment. The system 10 includes an anomaly detection device 12, and an operator device 14, connected via a network 20. In some embodiments, the devices 12, 14 may be part of a larger visual inspection system including additional components. For example, the system 10 may be implemented as part of any one or more of the systems described in US Patent Application No. 17765222, International Application No. PCT / CA2022 / 050100, International Application No. PCT / CA2022 / 050101 , and International Application No. PCT / CA / 2022050289.

[0052] The devices 12, 14, may be a server computer, node computing device (e.g., JETSON computing device or the like), embedded device, desktop computer, notebook computer, tablet, PDA, smartphone, or another computing device. The devices 12, 14, may include a connection with the network 20 such as a wired or wireless connection to the Internet. In some cases, the network 20 may include other types of computer or telecommunication networks. The devices 12, 14 may include one or more of a memory, a secondary storage device, a processor, an input device, a display device, and an output device. Memory may include random access memory (RAM) or similar types of memory. Also, memory may store one or more applications for execution by processor. Applications may correspond with software modules comprising computer executable instructions to perform processing for the functions described below. Secondary storage device may include a hard disk drive, floppy disk drive, CD drive, DVD drive, Blu-ray drive, or other types of non-volatile data storage. Processor may execute applications, computer readable instructions or programs. The applications, computer readable instructions or programs may be stored in memory or in secondary storage or may be received from the Internet or other network 20.

[0053] Input device may include any device for entering information into device 12, 14. For example, input device may be a keyboard, keypad, cursor-control device, touchscreen, camera, or microphone. Display device may include any type of device for presenting visual information. For example, display device may be a computer monitor, a flat-screen display, a projector, or a display panel. Output device may include any type of device for presenting a hard copy of information, such as a printer for example. Output device may also include other types of output devices such as speakers, for example. In some cases, device 12, 14 may include multiple of any one or more of processors, applications, software modules, second storage devices, network connections, input devices, output devices, and display devices.

[0054] Although devices 12, 14 are described with various components, one skilled in the art will appreciate that the devices 12, 14, may in some cases contain fewer, additional or different components. In addition, although aspects of an implementation of the devices 12, 14 may be described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices, including hard disks, floppy disks, CDs, or DVDs; a carrier wave from the Internet or other network; or other forms of RAM or ROM. The computer-readable media may include instructions for controlling the devices 12, 14 and / or processor to perform a particular method.

[0055] Devices 12, 14 can be described performing certain acts. It will be appreciated that any one or more of these devices may perform an act automatically or in response to an interaction by a user of that device. That is, the user of the device may manipulate one or more input devices (e.g., a touchscreen, a mouse, or a button) causing the device to perform the described act. In many cases, this aspect may not be described below, but it will be understood.

[0056] As an example, it is described below that the devices 12, 14 may send information to one or more other device 12, 14. For example, a user using the operator device 14 may manipulate one or more inputs (e.g., a mouse and a keyboard) to interact with a user interface displayed on a display of the device 14. Generally, the device mayreceive a user interface from the network 20 (e.g., in the form of a webpage or data for a native application). Alternatively, or in addition, a user interface may be stored locally at a device (e.g., a cache of a webpage or a mobile application).

[0057] The devices 12, 14, may be configured to receive a plurality of information, from one or more of the plurality of devices 12, 14.

[0058] In response to receiving information, the respective device 12, 14 may store the information in storage database. The storage may correspond with secondary storage of one or more other devices 12, 14. Generally, the storage database may be any suitable storage device such as a hard disk drive, a solid-state drive, a memory card, or a disk (e.g., CD, DVD, or Blu-ray etc.). Also, the storage database may be locally connected with the device 12, 14. In some cases, storage database may be located remotely from the device 12, 14 and accessible to the device 12, 14 across a network for example. In some cases, storage database may comprise one or more storage devices located at a networked cloud storage provider.

[0059] The anomaly detection device 12 may communicate with other devices over network 20 to retrieve image data and store generated data. Once the device 14 has generated anomaly data, the device 12 may transmit the anomaly data to over network 20 for further use or processing.

[0060] The operator device 14 comprises a computer terminal that may be used by a human operator to control system 10. The operator device 14 may be coupled to device 12 such that operator device 14 may adjust the parameters and control the operation of device 12. The operator device 14 includes a user interface component (or module) (e.g., a human-machine interface).

[0061] The user interface component of the operator device 14 may also render one or more user interface elements for receiving input from the operator or displaying outputs of the system 10 (e.g. model performance metrics). For example, the user interface component may provide a yes / no or similar binary option for receiving user input data indicating a selection of an option.

[0062] Referring now to Figure 2, Figure 2 shows a simplified block diagram of components of a computing device 1000, such as a mobile device or portable electronic device, according to an embodiment. Software modules described in the disclosure herein may be configured to run on a computing device, such as device 1000 of Figure 2. The device 1000 includes multiple components such as a processor 1020 that controls the operations of the device 1000. Communication functions, including data communications, voice communications, or both may be performed through a communication subsystem 1040. Data received by the device 1000 may be decompressed and decrypted by a decoder 1060. The communication subsystem 1040 may receive messages from and send messages to a wireless network 1500.

[0063] The wireless network 1500 may be any type of wireless network, including, but not limited to, data-centric wireless networks, voice-centric wireless networks, and dual-mode networks that support both voice and data communications.

[0064] The device 1000 may be a battery-powered device and as shown includes a battery interface 1420 for receiving one or more rechargeable batteries 1440.

[0065] The processor 1020 also interacts with additional subsystems such as a Random Access Memory (RAM) 1080, a flash memory 1100, a display 1120 (e.g. with a touch-sensitive overlay 1140 connected to an electronic controller 1160 that together comprise a touch-sensitive display 1180), an actuator assembly 1200, one or more optional force sensors 1220, an auxiliary input / output (I / O) subsystem 1240, a data port 1260, a speaker 1280, a microphone 1300, short-range communications systems 1320 and other device subsystems 1340.

[0066] In some embodiments, user-interaction with the graphical user interface may be performed through the touch-sensitive overlay 1140. The processor 1020 may interact with the touch-sensitive overlay 1140 via the electronic controller 1160. Information, such as text, characters, symbols, images, icons, and other items that may be displayed or rendered on a portable electronic device generated by the processor 102 may be displayed on the touch-sensitive display 118.

[0067] The processor 1020 may also interact with an accelerometer 1360 as shown in Figure 2. The accelerometer 1360 may be utilized for detecting direction of gravitational forces or gravity-induced reaction forces.

[0068] To identify a subscriber for network access according to the present embodiment, the device 1000 may use a Subscriber Identity Module or a Removable User Identity Module (SIM / RUIM) card 1380 inserted into a SIM / RUIM interface 1400 for communication with a network (such as the wireless network 1500). Alternatively, user identification information may be programmed into the flash memory 1100 or performed using other techniques.

[0069] The device 1000 also includes an operating system 1460 and software components 1480 that are executed by the processor 1020 and which may be stored in a persistent data storage device such as the flash memory 1100. Additional applications may be loaded onto the device 1000 through the wireless network 1500, the auxiliary I / O subsystem 1240, the data port 1260, the short-range communications subsystem 1320, or any other suitable device subsystem 1340.

[0070] For example, in use, a received signal such as a text message, an e-mail message, web page download, or other data may be processed by the communication subsystem 1040 and input to the processor 1020. The processor 1020 then processes the received signal for output to the display 1120 or alternatively to the auxiliary I / O subsystem 1240. A subscriber may also compose data items, such as e-mail messages, for example, which may be transmitted over the wireless network 1500 through the communication subsystem 1040.

[0071] For voice communications, the overall operation of the portable electronic device 1000 may be similar. The speaker 1280 may output audible information converted from electrical signals, and the microphone 1300 may convert audible information into electrical signals for processing.

[0072] Referring now to Figures 3 and 4, shown therein is a computer system 300 for segmentation-based anomaly detection, according to an embodiment (Figure 3), and a set of block diagrams which visually depict segment data 310, defect data 312 and anomaly data 316 (Figure 4) generated or processed by the computer system 300,according to an embodiment. System 300 may be implemented at device 12 of Figure 1 . System 300 comprises data storage 302, segmentation model 304, defect model 306, and anomaly detection module 314.

[0073] Data storage 302 comprises a storage device for data, which may be used to store data and retrieve stored data. Data storage 302 may comprise a software abstraction, such as a database, cloud storage, database, sever location or data location. In other examples, data storage 302 may comprise a physical storage device, such as a hard drive, solid state drive, NAND flash, tape drive, or any other physical drive known in the art.

[0074] Stored on data storage 302 is image data 308. Image data 308 comprises digital image data including views of surfaces or portions of interest of an inspection object. An inspection object may include any object which it may be desirable to visually inspect. Such objects may include any manufactured object, such as machine parts, electronic devices, tools, food, or any other object, substance, or material which may be inspected through visual means, either with visual inspection by a skilled operator, or through the application of visual light camera-based inspection. In some cases, the inspection object may include a plurality of objects.

[0075] For example, if the inspection object is an engine camshaft, image data 308 may comprise digital image data including views of all surfaces of the camshaft which may comprise defects which impact functionality or aesthetics. In some examples, this may include views of all external surfaces of the camshaft, or other inspection object. Such images may be taken with a digital camera, such as a CCD or CMOS based camera. In some cases, the camshaft or other inspection object may be manipulated for imaging, such as by rotating or otherwise moving the inspection object.

[0076] Segmentation model 304 comprises a software module configured to receive image data (e.g. image data 308), and output segment data (e.g. segment data 310) corresponding to the inputted images. Segmentation model 304 is configured to implement a machine learning-based image segmentation method.

[0077] Segment data 310 comprises segment location data 318. Segment location data 318 may comprise, for example, a bounding box, contour, or pixel map definedaround or over a region of interest, or multiple regions of interest (i.e. segment, or segments) of the inspection object. For example, an inspection object may comprise two distinct areas: a first area and a second area. Areas may correspond to regions of interest that can be identified in an image of an inspection object. A separate bounding box, contour or pixel map may be defined around or over each segment, one for the first area, and one for the second area, describing two different segments.

[0078] Multiple segments may be of interest, as different portions of an inspection object may comprise different materials, functionality, importance, or character, such that different defects may be present within different segments, or defects present in one segment may be of a different type or of a greater impact than defects in another segment. Accordingly, system 300 may be configured to assess each segment for defects separately and may be trained or configured to detect defects using different methods (e.g. computer vision techniques such as “object detection”, “instance segmentation”, or the like), models, or information for each segment.

[0079] Additionally, as visible in Figure 4, segment data 310 may comprise segment class data 320 corresponding to each detected segment, a segment image crop 322, including the image data lying within the defined segment region of the image, as well as segment metadata 324, which may include, without limitation, inspection object parameters, serial numbers, lot numbers, part numbers, facility identifiers, object manufacturers, material data, and dates and times of inspection and / or processing.

[0080] In some examples, segmentation model 304 may comprise a trained machine learning model, such as a neural network (e.g. convolutional neural network). In such examples, segmentation model 304 may be trained using supervised, unsupervised, or semi-supervised machine learning methods. In a particular embodiment, the segmentation model 304 may be an instance segmentation model.

[0081] In examples including supervised and / or semi-supervised training methods, training image data may be provided to an untrained segmentation model 304. Training image data may include a plurality of digital images of a set of inspection objects of the same specification, and corresponding segment data (e.g. bounding boxes, contours orpixel maps, and segment class data). Training data may be generated using any suitable technique, including manually by skilled operators for each image.

[0082] Defect model 306 comprises a software module configured to receive image data 308, and output defect data 312 describing defects detected in the image data 308. In some examples, defect model 306 may further receive segment data 310 as an input.

[0083] Defect data 312 comprises defect location data 326, which may comprise, for example, a bounding box, contour, or pixel map defined around or over a defect as described. In an example, an inspection object may include two distinct defects. A separate bounding box, contour or pixel map may be defined around or over each defect, one for the first defect, and one for the second defect, describing two different defects. In other examples, defect data 312 may describe any number of defects.

[0084] Additionally, defect data 312 may comprise defect class data 328 (or class label or assignment) corresponding to each detected defect (e.g. scratch, porosity, crack, or any other defect type or class that may be detected). Defect data 312 may include a defect image crop 330, including the image data lying within a defined defect region of the image, as well as defect metadata 332, which may include, without limitation, inspection object parameters, serial numbers, lot numbers, part numbers, facility identifiers, object manufacturers, material data, and dates and times of inspection and / or processing.

[0085] In some examples, defect model 306 may comprise a trained machine learning model, such as a neural network (e.g. convolutional neural network). In such examples, defect model 306 may be trained using supervised, unsupervised, or semisupervised machine learning methods. Generally, the defect model 306 may be configured to implement any suitable computer vision technique for detecting “objects” in a digital image. For example, the defect model 306 use an object detection or image segmentation technique. In a particular embodiment, the defect model 306 may be an instance segmentation model.

[0086] In examples including supervised and / or semi-supervised training methods, training image data may be provided to an untrained defect model 306. Training image data may include a plurality of digital images of a set of inspection objects, andcorresponding defect data, describing the defects present within each image. Training data may be generated using any suitable technique, including manually by skilled operators.

[0087] Anomaly detection module 314 (also referred to as anomaly model) comprises a software module configured to receive and analyze segment data 310 and defect data 312 and output anomaly data 316. In some examples, anomaly module 314 may further receive image data 308 as an input, as well as golden segment data (for example, golden sample data as described in PCT Patent Application No. PCT / CA2022 / 050100, PCT Patent Application No. PCT / CA2022 / 050289, and United States Provisional Patent Application No. 63 / 413,056.

[0088] Anomaly data 316 may comprise anomaly location data 334, which includes a bounding box, contour, or pixel map defined around or over an anomaly, or multiple anomalies of the inspection object.

[0089] Additionally, anomaly data 316 may comprise anomaly class data 336 corresponding to each detected anomaly. Anomaly data 316 may include an anomaly image crop 338, including the image data lying within the defined anomaly region of the image, as well as anomaly metadata 340, which may include, without limitation, inspection object parameters, serial numbers, lot numbers, part numbers, facility identifiers, object manufacturers, material data, and dates and times of inspection and / or processing.

[0090] Golden segment data comprises data corresponding the segments that are expected to be detected for an inspection object free of defects (the golden sample). For example, a golden sample inspection object may comprise 3 segments, each of specific size, shape, and position. Golden segment data may contain information which may communicate the specific size, shape, and position of each segment.

[0091] Anomaly module 314 is configured to compare generated segment data 310 for a specific inspection object, to determine if discrepancies exist between golden segments, and segment location data 318 generated by segmentation model 304. If such discrepancies exist, the areas or regions that do not overlap between golden segments, and segment location data 318 may be deemed suspected anomalies. If a defect isdetected within each of these areas or regions, the anomaly module 314 may not label the suspected anomalies as anomalies. However, if no defect is detected within these regions or areas, the suspected anomalies are deemed to be anomalies, and stored within or as anomaly data 316 by anomaly module 314.

[0092] In some examples, anomaly class data 336 may not be determined at the moment an anomaly is detected. Instead, anomaly class data 336 may be determined in a subsequent step. For example, a clustering model may be applied to detected anomalies, to automatically group like anomalies together.

[0093] Newly detected anomalies may be applied to train, re-train, or further train defect model 306. For example, similar or clustered anomalies, or anomalies of the same class may be provided to defect model 306 to train, re-train, or further train defect model 306 to detect these specific anomalies. Once anomalies (or conditions previously labelled as anomalies) are detectable by defect model 306, subsequent cycles of system 300 may not detect such conditions as anomalies, as such conditions may be labelled as defects by defect model 306.

[0094] In operation of system 300, image data 308 is provided to segmentation model 304 from data storage 302. Segmentation model 304 in turn outputs segment data 310. Segment data 310 and image data 308 are provided to defect model 306, which generates and outputs defect data 312. Defect data 312 and segment data 310 are provided to anomaly module 314. Optionally, image data 308 and golden segment data may also be provided to anomaly detection module 314. Anomaly module 314 may output anomaly data 316. Anomaly data 316 may be further processed, applied to train defect model 306, or reviewed manually by a skilled human operator.

[0095] Anomalies may be detected by system 300 by comparing segment data 310, golden segment data and defect data 312 using anomaly module 314. Areas of segments within segment data 310 not aligned with golden segment data, and not containing any defects may be deemed to be anomalies, and possibly contain novel defects, contamination, or other unusual condition, which should be further processed or reviewed.

[0096] Referring now to Figure 5, shown therein is an example image 500 of a golden inspection object 402, overlayed with segment data 310 according to an embodiment.

[0097] Segment data 310 of Figure 5 includes segment location data 318, defining two segment locations, golden segment location 404 and golden segment location 406.

[0098] Golden inspection object 402 is an example of an inspection object, deemed to be free of defects, to which other inspection objects are to be compared against. Golden inspection object 402 may be asserted or certified as free of defects by a skilled human operator, or other automated system. Segments defined in relation to golden inspection object 402 (e.g. 404, 406) may be the benchmarks to which later generated segments may be compared to, to determine if detected segments are unusual or irregular in any manner. Segments of other inspection objects which do not correspond to golden segment locations 404, 406 may indicate that an inspection object comprises an anomaly or suspected anomaly.

[0099] Referring now to Figure 6, shown therein is an example image 600 of an inspection object 502, overlayed with segment data 310, and defect data 312, according to an embodiment. Inspection object 502 is of an unknown condition, but of the same specification as golden inspection object 402.

[0100] Segment data 310 of Figure 6 includes segment location data 318, defining two segment locations, segment location 504 and segment location 506.

[0101] Segment location 506 is aligned with golden segment location 406, such that segment location 506 and golden segment location 406 are substantially the same size, shape, and in the same position, relative to each respective inspection object.

[0102] Segment location 504 is not substantially aligned with golden segment location 404. Segment location 506 and golden segment location 406 are substantially in the same position, relative to each respective inspection object, however, a region 528 of the lower right portion of golden segment location 404 is not present in segment location 504. Region 528 may include a set of pixels that have been segmented by thesegmentation model (i.e. segmented out from segment 504). Depending on the contents of region 528, region 528 may be deemed by anomaly module 314 as an anomaly.

[0103] Defect data 310 of Figure 6 includes defect location data 326, defining four defect locations, defect locations 508, 510, 512, and 514, corresponding to defects 516, 518, 520, and 522 respectively. Further, defect data 310 of Figure 6 includes defect 524, and corresponding defect location 526.

[0104] As defect 524 was detected within region 528 (as visible in Figure 6 and labelled by defect location 526), anomaly module 314, when processing image 500, and associated defect data 310 and segment data 312, may attribute the mismatch between segment location 504 and golden segment location 404 to the detected defect 524. Accordingly, inspection object 502 is deemed to be free of anomalies by anomaly model 514 (i.e. region 528 is not identified as an anomaly). All suspected anomalies have been attributed to known causes.

[0105] Referring now to Figure 7, shown therein is an example image 700 of an inspection object 602, overlayed with segment data 308, defect data 312, and anomaly data 316, according to an embodiment. Inspection object 602 is of an unknown condition, but of the same specification as golden inspection object 402, and inspection object 502.

[0106] Segment data 308 of Figure 7 includes segment location data 318, defining two segment locations, segment location 604 and segment location 606.

[0107] Segment location 606 is aligned with golden segment location 406, such that segment location 606 and golden segment location 406 are substantially the same size, shape, and in the same position, relative to each respective inspection object.

[0108] Segment location 604 is not substantially aligned with golden segment location 404. Segment location 606 and golden segment location 406 are substantially in the same position, relative to each respective inspection object, however, a region of the lower right portion of golden segment location 404 is not present in segment location 604. The region may include a set of pixels that have been segmented by the segmentation model (i.e. segmented out from segment 504). Depending on the contents of the region, the region may be deemed by anomaly module 314 to be an anomaly.

[0109] Defect data 312 of Figure 7 includes defect location data 326, defining four defect locations, defect locations 608, 610, 612, and 614, corresponding to defects 616, 618, 620, and 622 respectively.

[0110] No defect was detected within the mismatched region between segment location 604 and golden segment location 404. Anomaly module 314, when processing image 600, and associated defect data 312 and segment data 310, may attempt to attribute the mismatch between segment location 604 and golden segment location 404 to a detected defect within this region. However, no defect has been detected within this region by defect model 306.

[0111] Accordingly, inspection object 602 is deemed to include an anomaly, with anomaly location 626. Anomaly location 626 may be stored within anomaly data 316 (e.g. as anomaly location data 334), associated with inspection object 602. Additional anomaly data 316, such as an image crop of anomaly location 626 of image 600 may be stored as an anomaly image crop 338, and additional anomaly metadata 340 may be stored in association with this detected anomaly as anomaly data 316.

[0112] Referring now to Figure 8, shown therein is an image 800 of an inspection object, overlaid with defect data and segment data, according to an embodiment. In this example, the inspection object is a camshaft and the image includes a portion of the camshaft. Segments have been detected within image 700, as seen by regions labelled with “ROI model”. Defects have been detected with a defect model. The ROI model may be analogous to the segmentation model 304 and the defect model may be analogous to the defect model 306 as described herein. The ROI model has detected a VTC region 802 via a segmentation technique. In doing so, pixels in a portion of the VT region 802 in the top right corner have been segmented out of the VTC region 802. The portion of “bad” pixels that have been segmented out of the VTC region 802 may be described by the segmentation model including location data and size data. The defect model has detected defect 804 in the image. The defect 804 is described by location data and size data. The anomaly module 314 is configured to compare the location and size data of the pixels segmented out of the VTC region 802 (i.e. corresponding to the top right portion) to the location and size data of the defect 804. The comparison may be performed according toa set of one or more rules for determining a “match”. If a match is not determined, the “bad” pixels segmented out of the VTC region 802 are labeled an anomaly by the anomaly detection module 314. If a match is not determined, the “bad” pixels are attributed to the defect 804 and are not labeled as an anomaly.

[0113] Referring now to Figure 9, shown therein is a flowchart depicting a method 900 of segmentation-based anomaly detection, according to an embodiment. Method 900 may be performed by the device 12 of Figure 1 or the system 300 of Figure 3.

[0114] At 902, image data is provided to segmentation model.

[0115] At 904, segment data is received from segmentation model, and segment data is provided to the defect model.

[0116] At 906, defect data is received from the defect model, and defect data and segment data are provided to anomaly module.

[0117] At 908, anomaly data is received from anomaly model.

[0118] While the systems and methods described herein included a segmentation model and a defect model, in some examples, additional visual inspection models may be present. For examples, additional models may be present to detect features which are not considered defects, such as holes, slots, or other geometric features which may be present on an inspection object.

[0119] While the systems and methods described herein are described in reference to the application of the automated inspection of parts during manufacturing processes, in other embodiments, the systems and methods described herein may be applied to other processes or applications. For example, the systems and methods described herein may be applied to any computer vision-based feature or object detection system, such as a wildlife detection system.

[0120] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:1 . A method of segmentation-based anomaly detection, the method comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data localizing each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data localizing each defect in the digital image; comparing the segment location data to the defect location data to identify any segments that do not correspond to a detected defect; and identifying each segment that does not correspond to a detected defect as an anomaly or anomalous region.

2. The method of claim 1 , wherein the segmentation model comprises a neural network.

3. The method of claim 1 , wherein the segmentation model is an instance segmentation model.

4. The method of claim 1 , wherein an anomaly is a detected condition or state that is deemed abnormal but does not align with a defect type that the defect model has been trained to detect.

5. The method of claim 1 , further comprising training the defect model to detect the segment as a new defect type.The method of claim 1 , further comprising grouping similar anomalies identified by the method of claim 1 into one or more groups or clusters using an unsupervised machine learning technique and training the defect model with the anomaly data corresponding to the one or more groups or clusters. The method of claim 1 , wherein the segment location data of a segment includes a bounding box, a contour, or a pixel map. The method of claim 9, wherein the segmentation output further includes segment class data for each detected segment and a segment image crop including the image data lying within the defined segment region of the digital image. The method of claim 1 , wherein the defect model is a neural network. The method of claim 1 , wherein the defect model is an instance segmentation model. The method of claim 1 , wherein the defect data includes defect class data identifying a defect class of the defect. The method of claim 1 , wherein the performing defect detection on the digital image includes selecting the defect detection model from a plurality of defect detection models based on a segment class of a first segment in the segmentation output, the defect detection model trained to detect defects in that segment class. The method of claim 13, wherein the plurality of defect detection models include a first defect detection model and a second defect detection model that use different computer vision techniques to perform defect detection. The method of claim 1 , wherein the anomaly or anomalous region is used to train a new defect detection model or retrain the defect detection model.The method of claim 1 , wherein the digital image includes views of one or more surfaces or regions of interest of an inspection object being visually inspected. The method of claim 1 , wherein comparing the segment location data to the defect location data includes using golden segment data. A system for segmentation-based anomaly detection, the system comprising: a memory for storing data including a digital image; and at least one processor in communication with the memory, the processor configured to execute a method comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data localizing each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data localizing each defect in the digital image; comparing the segment location data to the defect location data to identify any segments that do not correspond to a detected defect; and identifying each segment that does not correspond to a detected defect as an anomaly or anomalous region. The system of claim 18, wherein the segmentation model is an instance segmentation model.The method of claim 1 , wherein the method performed by the at least one processor further comprises training the defect model to detect the segment as a new defect type. At least one non-transitory computer-readable storage medium storing processorexecutable instructions that when executed by at least one processor, cause the at least one processor to perform a method of segmentation-based anomaly detection, the method comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data localizing each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data localizing each defect in the digital image; comparing the segment location data to the defect location data to identify any segments that do not correspond to a detected defect; and identifying each segment that does not correspond to a detected defect as an anomaly or anomalous region.

Images