Computer system and method for automated visual inspection using segmentation-based anomaly detection

Abstract

Systems, methods, and devices for segmentation-based anomaly detection are provided, the methods including providing image data to a segmentation model, receiving segment data from the segmentation model, providing the segment data as input to a defect model, and receiving defect data from the defect model as output, providing the defect data and the segment data as input to the anomaly model, and receiving the anomaly data from the anomaly model as output.

Description

[Technical Field]

[0001] The following relates generally to systems and methods for automated visual inspection of objects, and more particularly to systems and methods for detecting anomalies in images using segmentation models. [Background technology]

[0002] Introduction

[0002] Current methods and systems for anomaly detection are limited in functionality and can provide relatively poor or unreliable performance, many limitations that can be exacerbated by inherent limitations in the availability and ease of obtaining training data. Summary of the Invention [Problem to be solved by the invention]

[0003]

[0003] Therefore, there is a need for improved systems and methods for anomaly detection that overcome at least some of the shortcomings of existing systems and methods. [Means for solving the problem]

[0004] overview Systems, methods, and devices are provided for segmentation-based anomaly detection. The method includes performing image segmentation on a digital image using a segmentation model to obtain a segmentation output. The segmentation output includes segment location data identifying a location of each segment within the digital image. The method further includes performing defect detection on the digital image using a defect detection model configured to detect and classify defects within the digital image to obtain a defect detection output. The defect detection output includes defect location data identifying a location of each defect within the digital image. The method further includes comparing the segment location data to the defect location data to identify any segments that do not correspond to detected defects. Segments that do not correspond to detected defects are identified as anomalies or anomalous regions within the digital image.

[0005] In one embodiment, the segmentation model includes a neural network.

[0006] In one embodiment, the segmentation model is an instance segmentation model.

[0007]

[0007] In one embodiment, an anomaly is a detected condition or state that is considered to be abnormal but does not match the defect type that the defect model is trained to detect.

[0008]

[0008] In one embodiment, the method further includes training a defect model to detect the segment as a new defect type.

[0009]

[0009] In one embodiment, the method further includes grouping similar anomalies into one or more groups or clusters using unsupervised machine learning techniques, and training a defect model with anomaly data corresponding to the one or more groups or clusters.

[0010] In one embodiment, the segment position data for a segment comprises a bounding box, a contour or a pixel map.

[0011] In one embodiment, the segmentation output further includes segment class data for each detected segment and a segment image crop containing image data that is within the defined segment region of the digital image.

[0012] In one embodiment, the defect model is a neural network.

[0013] In one embodiment, the defect model is an instance segmentation model.

[0014] In one embodiment, the defect data includes defect class data that identifies a defect class of the defect.

[0015]

[0015] In one embodiment, performing defect detection on a digital image includes selecting a defect detection model from a plurality of defect detection models based on a segment class of a first segment in the segmentation output, and the defect detection model is trained to detect defects of that segment class.

[0016] In one embodiment, the plurality of defect detection models includes a first defect detection model and a second defect detection model that perform defect detection using different computer vision techniques.

[0017]

[0017] In one embodiment, the anomaly or anomaly region is used to train a new defect detection model or retrain a defect detection model.

[0018] In one embodiment, the digital image includes a view of one or more surfaces or regions of interest of the inspection object being visually inspected.

[0019] In one embodiment, comparing the segment location data to the defect location data includes using golden segment data.

[0020] A system for segmentation-based anomaly detection is also provided. The system includes a memory for storing data including a digital image and at least one processor in communication with the memory. The processor is configured to execute a method including: performing image segmentation on the digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data identifying a location of each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data identifying a location of each defect in the digital image; comparing the segment location data to the defect location data to identify any segments that do not correspond to detected defects; and identifying each segment that does not correspond to a detected defect as an anomaly or an anomaly region.

[0021] In one embodiment, the segmentation model is an instance segmentation model.

[0022]

[0022] Also provided is at least one non-transitory computer-readable storage medium storing processor-executable instructions which, when executed by at least one processor, cause the at least one processor to perform a method for segmentation-based anomaly detection, the method including: performing image segmentation on the digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment location data identifying a location of each segment in the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data identifying a location of each defect in the digital image; comparing the segment location data with the defect location data to identify any segments that do not correspond to a detected defect; and identifying each segment that does not correspond to a detected defect as an anomaly or anomaly region.

[0023]

[0023] Other aspects and features will become apparent to those skilled in the art upon review of the following description of several exemplary embodiments.

[0024] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings included herein are intended to illustrate various examples of the articles, methods and apparatus herein. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram of a networked computer system for segmentation-based anomaly detection, according to one embodiment. [Figure 2]

[0026] FIG. 1 is a block diagram of a computing device according to one embodiment. [Figure 3]

[0027] FIG. 1 is a system block diagram of an anomaly detection system according to one embodiment. [Figure 4]

[0028] FIG. 4 is a schematic diagram of data associated with the anomaly detection system of FIG. 3, according to one embodiment. [Figure 5]

[0029] FIG. 4 is a diagram of an example image of a golden sample associated with the anomaly detection system of FIG. 3, according to one embodiment. [Figure 6]

[0030] 4 is a diagram of an example image of an inspection object processed by the anomaly detection system of FIG. 3, according to one embodiment. [Figure 7]

[0031] 4 is a diagram of an example image of an inspection object processed by the anomaly detection system of FIG. 3, according to one embodiment. [Figure 8]

[0032] 4 is a diagram of an example image of an inspection object processed by the anomaly detection system of FIG. 3, according to one embodiment. [Figure 9]

[0033] 1 is a flowchart of a method for segmentation-based anomaly detection, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Detailed Description

[0034] Various devices or processes are described below to provide examples of each claimed embodiment. No embodiment described below limits any claimed embodiment, and any claimed embodiment may encompass processes or devices different from those described below. A claimed embodiment is not limited to a device or process having all of the features of any one device or process described below or to features common to some or all of the devices described below.

[0027]

[0035] One or more systems described herein may be implemented in a computer program executing on a programmable computer, each of which includes at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, but not limited to, the programmable computer may be a programmable logic unit, a mainframe computer, a server or personal computer, a cloud-based program or system, a laptop, a personal data assistant, a mobile phone, a smartphone, or a tablet device.

[0028]

[0036] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In either case, the language can be a compiled or interpreted language. Each such computer program is preferably stored on a general-purpose or special-purpose programmable computer-readable storage medium or device for configuring and operating the computer when the storage medium or device is read by a computer to perform the procedures described herein.

[0029]

[0037] A description of an embodiment in which several components are in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0030]

[0038] Furthermore, although process steps, method steps, algorithms, etc. may be described (in this disclosure and / or claims) in a sequential order, such processes, methods, and algorithms may be configured to function in an alternating order. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. Steps of processes described herein may be performed in any order that is practical. Furthermore, some steps may be performed simultaneously.

[0031]

[0039] Where a single device or article is described herein, it will be readily apparent that two or more devices / articles (whether or not they cooperate) may be used in place of the single device / article. Similarly, where two or more devices or articles are described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the two or more devices or articles.

[0032]

[0040] The following relates generally to automated methods for visual inspection, and more particularly to methods and systems directed to segmentation-based machine learning model systems that can detect defects and anomalies in objects by analyzing images of the objects.

[0033]

[0041] The systems and methods described herein may function as follows: A set of test objects may be provided. Each test object may be the same object type, but may be in varying conditions. For example, each test object may be a camshaft of the same specifications, but each test object may be a separate physical object, such that 100 separate physical camshafts may be tested. Because each physical object is individual, each separate physical object is of a disparate condition and / or characteristics.

[0034]

[0042] A digital image of each test object can be captured, which can include imaging all surfaces and portions of the object or some subset thereof.

[0035]

[0043] Inspection object images may be provided to an anomaly detection system. In particular, these inspection images may be provided as input to a segmentation model. The segmentation model may be an instance segmentation model. The segmentation model may output bounding boxes, contours, or pixel maps around or on known regions to "segment" the image into discrete areas. This segmentation information may be stored or organized as segment data (or segmentation output data).

[0036]

[0044] The inspection object image may then be provided to a defect model. The segmentation data may also be provided to the defect model. The defect model may output defect data that identifies the location of the defects and classifies the located defects. The defects may include holes, scratches, cracks, tolerance issues, or other defects.

[0037]

[0045] The inspection image, segment data, and defect data may then be provided to an anomaly detection module model, or anomaly model. The anomaly detection module may compare the segment data and the defect data. In doing so, the anomaly detection module may locate areas in the inspection image having segment data that does not correspond to any defect according to the defect data ("anomalous segment data"). The anomaly detection module may flag these areas as containing "anomalies," and descriptions of these areas may be stored as anomaly data.

[0038]

[0046] An anomaly may be defined as a detected condition or state that is considered abnormal but does not match any known and / or detected defect. For example, an anomaly may include an object defect or a measurement, equipment, or process error that has not yet been identified, classified, and / or expected. For example, an anomaly may correspond to improperly calibrated equipment or the presence of dust, debris, insects, or other foreign objects in the inspection space.

[0039]

[0047] In instances where anomalies correspond to defects that have not yet been identified, classified, and / or anticipated, detection of such anomalies may be applied to train a defect model to detect these defects that have not been explicitly trained for detection by the defect model.

[0040]

[0048] For example, a hole of a particular morphology may be present in an image. This may not be a trained defect because such a defect may not have been seen before. The systems and methods described herein may detect this new defect as an anomaly by applying the segmentation methods described above. Similar anomalies at other locations or in the inspected object may be grouped together (manually or using machine learning methods, such as methods for unsupervised grouping of images). Once grouped together, this group or set of anomaly data corresponding to similar new defects can then be used to further train the defect model so that the system may detect this new defect as a known defect instead of an anomaly.

[0041]

[0049] The methods and systems described herein may advantageously detect novel defects that were not specifically anticipated or trained for, which may be particularly advantageous when applying smaller training sets to the models described herein, since rare defects are less likely to be present in smaller sample sizes.

[0042]

[0050] The above-mentioned models may refer to machine learning models such as neural networks (e.g., convolutional neural networks).

[0043]

[0051] Referring now to FIG. 1 , a segmentation-based anomaly detection system 10 according to one embodiment is shown therein. System 10 includes an anomaly detection device 12 and an operator device 14 connected via a network 20. In some embodiments, devices 12, 14 may be part of a larger visual inspection apparatus that includes additional components. For example, system 10 may be implemented as part of any one or more of the systems described in U.S. Patent Application Publication No. 17765222, International Application No. PCT / CA2022 / 050100, International Application No. PCT / CA2022 / 050101, and International Application No. PCT / CA / 2022050289.

[0044]

[0052] The devices 12, 14 may be server computers, node computing devices (e.g., JETSON computing devices), embedded devices, desktop computers, notebook computers, tablets, PDAs, smartphones, or other computing devices. The devices 12, 14 may include a connection to a network 20, such as a wired or wireless connection to the Internet. In some cases, the network 20 may include other types of computer or communication networks. The devices 12, 14 may include one or more of a memory, a secondary storage device, a processor, an input device, a display device, and an output device. The memory may include random access memory (RAM) or a similar type of memory. The memory may also store one or more applications for execution by the processor. The applications may correspond to software modules containing computer-executable instructions for performing processing for functions described below. The secondary storage device may include a hard disk drive, a floppy disk drive, a CD drive, a DVD drive, a Blu-ray drive, or other type of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. The application, computer readable instructions or program may be stored in memory or secondary storage, or may be received from the Internet or other network 20 .

[0045]

[0053] Input devices may include any device for inputting information into device 12, 14. For example, input devices may be a keyboard, keypad, cursor control device, touch screen, camera, or microphone. Display devices may include any type of device for presenting visual information. For example, display devices may be a computer monitor, flat screen display, projector, or display panel. Output devices may include any type of device for presenting hard copies of information, such as a printer. For example, output devices may also include other types of output devices, such as speakers. In some cases, device 12, 14 may include any one or more of a processor, application, software module, secondary storage device, network connection, input device, output device, and display device.

[0046]

[0054] While devices 12, 14 are described with various components, those skilled in the art will understand that devices 12, 14 may, in some cases, include fewer, additional, or different components. Additionally, while aspects of the implementation of devices 12, 14 may be described as being stored in memory, those skilled in the art will understand that these aspects may be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices including hard disks, floppy disks, CDs, or DVDs, carrier waves from the Internet or other networks, or other forms of RAM or ROM. The computer-readable media may include instructions for controlling devices 12, 14 and / or processors to perform particular methods.

[0047]

[0055] Devices 12, 14 may be described that perform several operations. It will be recognized that any one or more of these devices may perform the operations automatically or in response to interaction by a user of the device. That is, a user of the device may manipulate one or more input devices (e.g., a touchscreen, mouse, or buttons) to cause the device to perform the described operations. In many cases, this aspect may not be described below, but will be understood.

[0048]

[0056] As an example, it is described below that a device 12, 14 may transmit information to one or more other devices 12, 14. For example, a user using an operator device 14 may manipulate one or more inputs (e.g., a mouse and keyboard) to interact with a user interface displayed on a display of the device 14. Generally, a device may receive a user interface (e.g., in the form of a web page or data of a native application) from the network 20. Alternatively or additionally, the user interface may be stored locally on the device (e.g., a web page or mobile application cache).

[0049]

[0057] The device 12, 14 may be configured to receive information from one or more of the devices 12, 14.

[0050]

[0058] In response to receiving the information, each device 12, 14 may store the information in a storage database. The storage may correspond to a secondary storage of one or more other devices 12, 14. Generally, the storage database may be any suitable storage device, such as a hard disk drive, solid state drive, memory card, or disk (e.g., CD, DVD, or Blu-ray). The storage database may also be locally connected to the device 12, 14. In some cases, the storage database may be located remotely from the device 12, 14 and accessible to the device 12, 14, for example, over a network. In some cases, the storage database may include one or more storage devices located at a network cloud storage provider.

[0051]

[0059] Anomaly detection device 12 may communicate with other devices on network 20 to retrieve image data and store the generated data. Once device 14 generates anomaly data, device 12 may transmit the anomaly data over network 20 for further use or processing.

[0052]

[0060] The operator device 14 includes a computer terminal that can be used by a human operator to control the control system 10. The operator device 14 can be coupled to the device 12 such that the operator device 14 can adjust parameters and control the operation of the device 12. The operator device 14 includes user interface components (or modules) (e.g., a human-machine interface).

[0053]

[0061] The user interface component of the operator device 14 may also render one or more user interface elements for receiving input from an operator or displaying output (e.g., model performance metrics) of the system 10. For example, the user interface component may provide yes / no or similar binary options for receiving user input data indicating the selection of the option.

[0054]

[0062] Referring now to FIG. 2, FIG. 2 shows a simplified block diagram of components of a computing device 1000, such as a mobile or portable electronic device, according to one embodiment. Software modules described in this disclosure may be configured to execute on a computing device, such as the device 1000 of FIG. 2. The device 1000 includes multiple components, such as a processor 1020, which controls the operation of the device 1000. Communication functions, including data communication, voice communication, or both, may be performed through a communication subsystem 1040. Data received by the device 1000 may be decompressed and decrypted by a decoder 1060. The communication subsystem 1040 may receive messages from and transmit messages to a wireless network 1500.

[0055]

[0063] Wireless network 1500 can be any type of wireless network, including, but not limited to, data-oriented wireless networks, voice-oriented wireless networks, and dual-mode networks that support both voice and data communications.

[0056]

[0064] The device 1000 may be a battery-powered device and, as shown, includes a battery interface 1420 for accommodating one or more rechargeable batteries 1440 .

[0057]

[0065] The processor 1020 also interacts with additional subsystems such as random access memory (RAM) 1080, flash memory 1100, display 1120 (e.g., having a touch-sensitive overlay 1140 connected to an electronic controller 1160 which together include a touch-sensitive display 1180), actuator assembly 1200, one or more optional force sensors 1220, auxiliary input / output (I / O) subsystem 1240, data port 1260, speaker 1280, microphone 1300, short-range communication system 1320, and other device subsystems 1340.

[0058]

[0066] In some embodiments, user interaction with the graphical user interface may occur through touch-sensitive overlay 1140. Processor 1020 may interact with touch-sensitive overlay 1140 via electronic controller 1160. Information generated by processor 102, such as text, characters, symbols, images, icons, and other items that may be displayed or rendered on the portable electronic device, may be displayed on touch-sensitive display 118.

[0059]

[0067] The processor 1020 may also interact with an accelerometer 1360, as shown in Figure 2. The accelerometer 1360 may be used to detect the direction of gravity or a reaction force caused by gravity.

[0060]

[0068] To identify a subscriber for network access according to this embodiment, device 1000 may use a subscriber identity module or removable user identity module (SIM / RUIM) card 1380 inserted into SIM / RUIM interface 1400 for communication with a network (such as wireless network 1500). Alternatively, user identity information may be programmed into flash memory 1100 or implemented using other techniques.

[0061]

[0069] Device 1000 also includes an operating system 1460 and software components 1480 executed by processor 1020, which may be stored in a persistent data storage device such as flash memory 1100. Additional applications may be loaded onto device 1000 through wireless network 1500, auxiliary I / O subsystem 1240, data port 1260, short-range communications subsystem 1320, or any other suitable device subsystem 1340.

[0062]

[0070] For example, in use, received signals such as text messages, email messages, web page downloads, or other data may be processed by communications subsystem 1040 and input to processor 1020. Processor 1020 then processes the received signals for output to display 1120 or alternatively auxiliary I / O subsystem 1240. Subscribers may also organize data items, such as email messages, that may be transmitted over wireless network 1500 through communications subsystem 1040.

[0063]

[0071] With respect to voice communication, the overall operation of the portable electronic device 1000 may be similar: the speaker 1280 may output audible information converted from an electrical signal, and the microphone 1300 may convert the audible information into an electrical signal for processing.

[0064]

[0072] 3 and 4, there is shown a pair of block diagrams visually depicting a computer system 300 for segmentation-based anomaly detection according to one embodiment (FIG. 3) and segment data 310, defect data 312, and anomaly data 316 (FIG. 4) generated or processed by the computer system 300 according to one embodiment. The system 300 may be implemented in the device 12 of FIG. 1. The system 300 includes a data store 302, a segmentation model 304, a defect model 306, and an anomaly detection module 314.

[0065]

[0073] Data storage 302 may include a data storage device and may be used to store and retrieve stored data. Data storage 302 may include a software abstraction such as a database, cloud storage, database, server location, or data location. In other examples, data storage 302 may include a physical storage device such as a hard drive, solid state drive, NAND flash, tape drive, or any other physical drive known in the art.

[0066]

[0074] Image data 308 is stored on data storage device 302. Image data 308 includes digital image data that includes a view of a target surface or portion of an inspection object. The inspection object may include any object for which visual inspection may be desired. Such objects may include any manufactured object, such as a machine part, an electronic device, a tool, food, or any other object, substance, or material that can be inspected via visual means, such as by visual inspection by a skilled operator or through the application of visible light camera-based inspection. In some cases, the inspection object may include multiple objects.

[0067]

[0075] For example, if the inspection object is an engine camshaft, image data 308 may include digital image data including views of all surfaces of the camshaft, which may contain defects that affect functionality or aesthetics. In some examples, this may include views of all exterior surfaces of the camshaft or other inspection object. Such images may be captured by a digital camera, such as a CCD or CMOS-based camera. In some cases, the camshaft or other inspection object may be manipulated for imaging, such as by rotating or otherwise moving the inspection object.

[0068]

[0076] The segmentation model 304 includes a software module configured to receive image data (e.g., image data 308) and output segment data (e.g., segment data 310) corresponding to an input image. The segmentation model 304 is configured to implement a machine learning-based image segmentation method.

[0069]

[0077] The segment data 310 includes segment position data 318. The segment position data 318 may include, for example, a bounding box, contour, or pixel map defined around or on a region or regions of interest (i.e., one or more segments) of the inspected object. For example, the inspected object may include two distinct areas: a first area and a second area. These areas may correspond to regions of interest that may be identified in an image of the inspected object. Separate bounding boxes, contours, or pixel maps describing the two different segments, one for the first area and one for the second area, may be defined around or on each segment.

[0070]

[0078] Multiple segments may be of interest because different portions of the inspection object may contain different materials, functionality, importance, or character, such that different defects may exist in different segments, or defects present in one segment may be of a different type or greater impact than defects in another segment. Accordingly, system 300 may be configured to evaluate each segment for defects separately and may be trained or configured to detect defects using various methods (e.g., computer vision techniques such as "object detection," "instance segmentation," etc.), models, or information from each segment.

[0071]

[0079] Additionally, as can be seen in FIG. 4, the segment data 310 may include segment class data 320 corresponding to each detected segment, a segment image crop 322 containing image data that is within the defined segment area of ​​the image, and segment metadata 324, which may include, but is not limited to, inspection object parameters, serial number, lot number, part number, facility identifier, object manufacturer, material data, and inspection and / or processing date and time.

[0072]

[0080] In some examples, the segmentation model 304 may include a trained machine learning model, such as a neural network (e.g., a convolutional neural network). In such examples, the segmentation model 304 may be trained using supervised, unsupervised, or semi-supervised machine learning methods. In particular embodiments, the segmentation model 304 may be an instance segmentation model.

[0073]

[0081] In examples involving supervised and / or semi-supervised training methods, training image data may be provided to the untrained segmentation model 304. The training image data may include multiple digital images of a set of test objects of the same specifications and corresponding segment data (e.g., bounding boxes, contours or pixel maps and segment class data). The training data may be generated using any suitable technique, such as manually by a skilled operator for each image.

[0074]

[0082] The defect model 306 includes a software module configured to receive the image data 308 and output defect data 312 that describes the defects detected in the image data 308. In some examples, the defect model 306 further receives segment data 310 as input.

[0075]

[0083] Defect data 312 includes defect location data 326, which may include, for example, bounding boxes, contours, or pixel maps defined around or on the defects, as described. In one example, an inspection object may include two separate defects. Separate bounding boxes, contours, or pixel maps describing the two different defects, one for the first defect and one for the second defect, may be defined around or on each defect. In other examples, defect data 312 may describe any number of defects.

[0076]

[0084] Additionally, defect data 312 may include defect class data 328 (or class label or assignment) corresponding to each detected defect (e.g., a scratch, hole, crack, or any other defect type or class that may be detected). Defect data 312 may include a defect image crop 330 that includes image data that is within the defined defect area of ​​the image and defect metadata 332 that may include, but is not limited to, inspection object parameters, serial number, lot number, part number, facility identifier, object manufacturer, material data, and date and time of inspection and / or processing.

[0077]

[0085] In some examples, the defect model 306 may include a trained machine learning model, such as a neural network (e.g., a convolutional neural network). In such examples, the defect model 306 may be trained using a supervised machine learning method, an unsupervised machine learning method, or a semi-supervised machine learning method. In general, the defect model 306 may be configured to implement any suitable computer vision technique for detecting “objects” in a digital image. For example, the defect model 306 uses object detection or image segmentation techniques. In particular embodiments, the defect model 306 may be an instance segmentation model.

[0078]

[0086] In examples involving supervised and / or semi-supervised training methods, training image data may be provided to the untrained defect model 306. The training image data may include multiple digital images of a set of inspection objects and corresponding defect data describing defects present in each image. The training data may be generated using any suitable technique, such as manually by a skilled operator.

[0079]

[0087] The anomaly detection module 314 (also referred to as an anomaly model) includes a software module configured to receive and analyze the segment data 310 and the defect data 312 and output anomaly data 316. In some examples, the anomaly module 314 further receives as input the image data 308 and golden segment data (e.g., golden sample data as described in PCT Patent Application Nos. PCT / CA2022 / 050100, PCT Patent Application No. PCT / CA2022 / 050289, and U.S. Provisional Patent Application No. 63 / 413,056).

[0080]

[0088] The anomaly data 316 may include anomaly location data 334 that includes a bounding box, contour, or pixel map defined around or on the anomaly or anomalies of the inspected object.

[0081]

[0089] Additionally, the anomaly data 316 may include anomaly class data 336 corresponding to each detected anomaly. The anomaly data 316 may include an anomaly image crop 338 containing image data that is within the defined anomaly region of the image and anomaly metadata 340, which may include, but is not limited to, inspection object parameters, serial number, lot number, part number, facility identifier, object manufacturer, material data, and date and time of inspection and / or processing.

[0082]

[0090] The golden segment data includes data corresponding to segments (golden samples) that are expected to be detected on a defect-free inspection object. For example, a golden sample inspection object may include three segments, each of a specific size, shape, and location. The golden segment data may include information that can convey the specific size, shape, and location of each segment.

[0083]

[0091] The anomaly module 314 is configured to compare the generated segment data 310 of a particular inspected object with the segment position data 318 generated by the segmentation model 304 to determine whether a discrepancy exists between the golden segments. If such a discrepancy exists, areas or regions that do not overlap between the golden segments and the segment position data 318 may be considered suspected anomalies. If defects are detected in each of these areas or regions, the anomaly module 314 may not label the suspected anomaly as an anomaly. However, if no defects are detected within these areas or regions, the suspected anomaly is considered an anomaly and stored in or as the anomaly data 316 by the anomaly module 314.

[0084]

[0092] In some examples, the anomaly class data 336 may not be determined at the time the anomaly is detected. Instead, the anomaly class data 336 may be determined at a subsequent step. For example, a clustering model may be applied to the detected anomalies to automatically group similar anomalies together.

[0085]

[0093] Newly detected anomalies may be applied to train, retrain, or further train the defect model 306. For example, similar or clustered anomalies or anomalies of the same class may be provided to the defect model 306 to train, retrain, or further train the defect model 306 to detect these particular anomalies. If an anomaly (or a condition previously labeled as an anomaly) is detectable by the defect model 306, subsequent cycles of the system 300 may not detect such a condition as an anomaly because such a condition may be labeled as a defect by the defect model 306.

[0086]

[0094] During operation of the system 300, image data 308 is provided from the data storage device 302 to the segmentation model 304. The segmentation model 304 then outputs segment data 310. The segment data 310 and the image data 308 are provided to a defect model 306, which generates and outputs defect data 312. The defect data 312 and the segment data 310 are provided to an anomaly module 314. Optionally, the image data 308 and golden segment data may also be provided to the anomaly detection module 314. The anomaly module 314 may output anomaly data 316. The anomaly data 316 may be further processed, applied to train the defect model 306, or manually reviewed by a skilled human operator.

[0087]

[0095] Anomalies may be detected by the system 300 by comparing the segment data 310, golden segment data, and defect data 312 using an anomaly module 314. Areas of segments in the segment data 310 that do not match the golden segment data and do not contain any defects may be considered anomalies, possibly containing new defects, contamination, or other abnormal conditions that should be further processed or investigated.

[0088]

[0096] Referring now to FIG. 5, there is shown an example image 500 of the golden test object 402 overlaid with segment data 310, according to one embodiment.

[0089]

[0097] The segment data 310 of FIG. 5 includes segment position data 318 that defines two segment positions, golden segment position 404 and golden segment position 406 .

[0090]

[0098] The golden test object 402 is an example of a test object that is deemed defect-free, against which other test objects are compared. The golden test object 402 may be claimed or certified defect-free by a trained human operator or other automated system. The segments (e.g., 404, 406) defined relative to the golden test object 402 may be benchmarks against which subsequently generated segments may be compared to determine whether the detected segments are abnormal or anomalous in any way. Segments of other test objects that do not correspond to the golden segment locations 404, 406 may indicate that the test object contains an anomaly or suspected anomaly.

[0091]

[0099] 6, there is shown an example image 600 of inspection object 502 overlaid with segment data 310 and defect data 312, according to one embodiment. Inspection object 502 is of unknown condition but of the same specifications as golden inspection object 402.

[0092]

[0100] The segment data 310 of FIG. 6 includes segment position data 318 that defines two segment positions, segment position 504 and segment position 506 .

[0093]

[0101] Segment locations 506 are aligned with golden segment locations 406 such that segment locations 506 and golden segment locations 406 are substantially the same size, shape, and in the same location relative to each respective test object.

[0094]

[0102] Segment location 504 is substantially misaligned with golden segment location 404. Segment location 506 and golden segment location 406 are in substantially the same location with respect to their respective test objects, but region 528 in the lower right portion of golden segment location 404 does not lie within segment location 504. Region 528 may include a set of pixels segmented by the segmentation model (i.e., segmented from segment 504). Depending on the contents of region 528, region 528 may be deemed anomalous by anomaly module 314.

[0095]

[0103] 6 includes defect location data 326 that defines four defect locations, defect locations 508, 510, 512, and 514, corresponding to defects 516, 518, 520, and 522, respectively. Additionally, defect data 310 of FIG. 6 includes defect 524 and corresponding defect location 526.

[0096]

[0104] Because defect 524 was detected within region 528 (as seen in FIG. 6 and labeled by defect location 526), ​​when anomaly module 314 processes image 500 and associated defect data 310 and segment data 312, it can attribute the mismatch between segment location 504 and golden segment location 404 to the detected defect 524. Therefore, inspection object 502 is deemed anomaly-free by anomaly model 514 (i.e., region 528 is not identified as an anomaly). All suspected anomalies are attributed to known causes.

[0097]

[0105] 7, there is shown an example image 700 of inspection object 602 overlaid with segment data 308, defect data 312, and anomaly data 316, according to one embodiment. Inspection object 602 is of unknown condition but of the same specifications as golden inspection object 402 and inspection object 502.

[0098]

[0106] The segment data 308 of FIG. 7 includes segment position data 318 that defines two segment positions, segment position 604 and segment position 606 .

[0099]

[0107] Segment location 606 is aligned with golden segment location 406 such that segment location 606 and golden segment location 406 are substantially the same size, shape, and in the same location relative to each respective test object.

[0100]

[0108] Segment location 604 is substantially misaligned with golden segment location 404. Segment location 606 and golden segment location 406 are in substantially the same location with respect to their respective test objects, but a region to the lower right of golden segment location 404 does not lie within segment location 604. This region may contain a set of pixels segmented by the segmentation model (i.e., segmented from segment 504). Depending on the content of the region, this region may be deemed anomalous by anomaly module 314.

[0101]

[0109] Defect data 312 of FIG. 7 includes defect location data 326 that defines four defect locations, defect locations 608, 610, 612, and 614, corresponding to defects 616, 618, 620, and 622, respectively.

[0102]

[0110] No defects were detected in the region of mismatch between segment location 604 and golden segment location 404. When anomaly module 314 processes image 600 and associated defect data 312 and segment data 310, it may attempt to attribute the mismatch between segment location 604 and golden segment location 404 to a detected defect in this region. However, no defect was detected in this region by defect model 306.

[0103]

[0111] Thus, the inspection object 602 is deemed to contain an anomaly at an anomaly location 626. The anomaly location 626 may be stored (e.g., as anomaly location data 334) within the anomaly data 316 associated with the inspection object 602. Additional anomaly data 316, such as an image crop of the image 600 at the anomaly location 626, may be stored as an anomaly image crop 338, and additional anomaly metadata 340 may be stored in association with this detected anomaly as anomaly data 316.

[0104]

[0112] Referring now to FIG. 8 , an image 800 of an inspection object overlaid with defect data and segment data is shown therein, according to one embodiment. In this example, the inspection object is a camshaft, and the image includes a portion of the camshaft. A segment has been detected in the image 700, as seen by the region labeled “ROI model.” The defect has been detected by the defect model. The ROI model may be similar to the segmentation model 304, and the defect model may be similar to the defect model 306 described herein. The ROI model has detected a VTC region 802 through segmentation techniques, whereby pixels within a portion of the VTC region 802 in the upper right corner have been segmented from the VTC region 802. The portion of the “bad” pixels segmented from the VTC region 802 can be explained by the segmentation model, which includes positional and dimensional data. The defect model has detected a defect 804 in the image. The defect 804 is explained by the positional and dimensional data. The anomaly module 314 is configured to compare the position and dimension data of a pixel segmented from the VTC region 802 (i.e., corresponding to the upper right portion) with the position and dimension data of the defect 804. This comparison may be performed according to one or more sets of rules for determining a "match." If a match is not determined, the "bad" pixel segmented from the VTC region 802 is labeled as an anomaly by the anomaly detection module 314. If a match is determined, the "bad" pixel is attributed to the defect 804 and is not labeled as an anomaly.

[0105]

[0113] 9, there is shown a flowchart depicting a method 900 of segmentation-based anomaly detection according to one embodiment. Method 900 may be performed by device 12 of FIG. 1 or system 300 of FIG.

[0106]

[0114] At 902, image data is provided to a segmentation model.

[0107]

[0115] At 904, segment data is received from a segmentation model and the segment data is provided to a defect model.

[0108]

[0116] At 906, defect data is received from the defect model and the defect data and segment data is provided to an anomaly module.

[0109]

[0117] At 908, anomaly data is received from the anomaly model.

[0110]

[0118] While the systems and methods described herein have included a segmentation model and a defect model, in some examples, additional visual inspection models may exist, for example, to detect features that are not considered defects, such as holes, slots, or other geometric features that may be present on the inspected object.

[0111]

[0119] Although the systems and methods described herein are described with reference to applications in automated inspection of parts during a manufacturing process, in other embodiments, the systems and methods described herein may be applied to other processes or applications. For example, the systems and methods described herein may be applied to any computer vision-based feature or object detection system, such as a wildlife detection system.

[0112]

[0120] While the above description provides one or more example devices, methods, or systems, it will be understood that other devices, methods, or systems may be within the scope of the claims, as interpreted by one of ordinary skill in the art.

Claims

1. 1. A method for segmentation-based anomaly detection, comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment position data identifying a location of each segment within the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data identifying a location of each defect in the digital image; comparing the segment location data with the defect location data to identify any segments that do not correspond to detected defects; identifying each segment that does not correspond to a detected defect as an anomaly or an anomaly region; A method comprising:

2. The method of claim 1 , wherein the segmentation model comprises a neural network.

3. The method of claim 1 , wherein the segmentation model is an instance segmentation model.

4. The method of claim 1 , wherein an anomaly is a detected condition or state that is considered to be abnormal but does not match a defect type that the defect model is trained to detect.

5. The method of claim 1 , further comprising training the defect model to detect the segment as a new defect type.

6. 10. The method of claim 1, further comprising: grouping similar anomalies identified by the method of claim 1 into one or more groups or clusters using unsupervised machine learning techniques; and training the defect model with anomaly data corresponding to the one or more groups or clusters.

7. The method of claim 1 , wherein the segment position data for a segment comprises a bounding box, a contour, or a pixel map.

8. 10. The method of claim 9, wherein the segmentation output further includes segment class data for each detected segment and a segment image crop containing image data that is within a defined segment region of the digital image.

9. The method of claim 1 , wherein the defect model is a neural network.

10. The method of claim 1 , wherein the defect model is an instance segmentation model.

11. The method of claim 1 , wherein the defect data includes defect class data that identifies a defect class of the defect.

12. 2. The method of claim 1, wherein performing defect detection on the digital image includes selecting the defect detection model from a plurality of defect detection models based on a segment class of a first segment in the segmentation output, the defect detection model being trained to detect defects of the segment class.

13. The method of claim 13 , wherein the plurality of defect detection models comprises a first defect detection model and a second defect detection model that use different computer vision techniques to perform defect detection.

14. The method of claim 1 , wherein the anomaly or anomaly region is used to train a new defect detection model or retrain the defect detection model.

15. The method of claim 1 , wherein the digital image includes a view of one or more surfaces or regions of a subject of an inspection object being visually inspected.

16. The method of claim 1 , wherein comparing the segment location data with the defect location data includes using golden segment data.

17. 1. A system for segmentation-based anomaly detection, comprising: a memory for storing data including a digital image; at least one processor in communication with the memory, performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment position data identifying a location of each segment within the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data identifying a location of each defect in the digital image; comparing the segment location data with the defect location data to identify any segments that do not correspond to detected defects; identifying each segment that does not correspond to a detected defect as an anomaly or an anomaly region; at least one processor configured to perform a method comprising: A system including:

18. The system of claim 18 , wherein the segmentation model is an instance segmentation model.

19. The method of claim 1 , wherein the method performed by the at least one processor further comprises training the defect model to detect the segment as a new defect type.

20. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for segmentation-based anomaly detection, the method comprising: performing image segmentation on a digital image using a segmentation model to obtain a segmentation output, the segmentation output including segment position data identifying a location of each segment within the digital image; performing defect detection on the digital image using a defect detection model configured to detect and classify defects in the digital image to obtain a defect detection output, the defect detection output including defect location data identifying a location of each defect in the digital image; comparing the segment location data with the defect location data to identify any segments that do not correspond to detected defects; identifying each segment that does not correspond to a detected defect as an anomaly or an anomaly region; At least one non-transitory computer-readable storage medium, including:

Images