Workpiece inspection and defect detection systems including monitoring of workpiece images
By using light sources, lenses and cameras in machine vision inspection systems, combined with the classification characteristics of abnormal detectors, the problem of low accuracy in workpiece defect detection is solved, and higher detection accuracy and system robustness are achieved.
Patent Information
- Application Number
- CN202110646169.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-06-15
- Filing Date
- 2021-06-10
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-06-10
AI Technical Summary
Existing machine vision inspection systems have problems with low accuracy in workpiece defect detection, especially in the case of workpiece type or surface changes, making it difficult to effectively detect and classify defects.
Using a system including a light source, a lens, a camera, a processor and a memory, by acquiring multiple training images, the training image features and the classification characteristics of the abnormality detector are determined, and the operation mode image of the workpiece is abnormal, and the image is adjusted according to the characteristics. conditions to improve detection accuracy.
It improves the accuracy and reliability of workpiece defect detection, can effectively distinguish abnormal and non-abnormal images, reduce false positives, adapt to workpiece surface changes, and improve the robustness of the system.
Smart Images

Figure CN113866165B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to workpiece inspection systems, and more particularly to precision machine vision systems for inspecting workpieces and detecting defects. Background Art
[0002] Precision non-contact workpiece inspection systems such as machine vision inspection systems (or simply "vision systems") can be used to obtain images of workpieces for inspection. Such systems can be used for various types of applications (e.g., general workpiece inspection, metrology applications for determining precise dimensional measurements of workpieces, etc.). Such systems typically include a computer, a camera, and an optical system. In certain configurations, a moving mechanism (e.g., a precision worktable, a conveyor, etc.) may be included that moves to allow the workpiece to be traversed and inspected. An exemplary prior art machine vision inspection system is the QUICK Series of PC-based vision systems and For example, the QVPAK 3D CNC Vision Measurement Machine User Guide, published in January 2003, generally describes QUICK Series of vision systems and Features and operation of the software, which is hereby incorporated herein by reference in its entirety. This type of system uses a microscope-type optical system and moves a stage to provide an inspection image of the workpiece.
[0003] Such machine vision inspection systems often face various types of challenges for inspecting workpieces (e.g., due to variations in the type or surface of the workpiece being inspected, varying inspection conditions, etc.) A system that can provide improvements to such issues for certain types of inspection operations (e.g., for workpiece defect detection and / or improving the accuracy of defect detection) would be desirable. Summary of the invention
[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0005] A workpiece inspection and defect detection system is provided, the system comprising a light source, a lens, a camera, one or more processors and a memory. The lens inputs image light caused by the surface of the workpiece illuminated by the light source and transmits the image light along an imaging light path. The camera receives the imaging light transmitted along the imaging light path and provides an image of the workpiece.
[0006] The memory is coupled to the one or more processors and stores program instructions that, when executed by the one or more processors, cause the one or more processors to at least: acquire a plurality of training images, wherein at least some of the plurality of training images are used to train a defect detection portion; determine a plurality of training image features from the plurality of training images; determine a plurality of anomaly detector classification features based at least in part on the determined training image features; acquire a first operating mode image of a workpiece using a camera; determine a plurality of features from the first operating mode image of the workpiece; and determine whether the first operating mode image of the workpiece is to be classified as abnormal using the anomaly detector classification features based at least in part on the features determined from the first operating mode image of the workpiece.
[0007] In various embodiments, the anomaly detector classification characteristic can be used to determine that the second operating mode image of the workpiece will be classified as non-anomalous. In addition, the defect detection portion can be used to analyze the second operating mode image to determine whether the second operating mode image of the workpiece will be classified as a defect image including a defect of the workpiece.
[0008] In various embodiments, the first operating mode image of the workpiece may be classified as abnormal in accordance with the utilization of the classification characteristics of the anomaly detector. In addition, a message indicating that at least one of the imaging conditions, lighting conditions, or workpiece conditions is problematic may be generated based at least in part on the classification of the first operating mode image of the workpiece as abnormal. In addition, a determined reason may be determined as to why the first operating mode image of the workpiece has been classified as abnormal. In addition, a message indicating the reason why the first operating mode image of the workpiece was classified as abnormal may be generated. In various embodiments, after determining the reason why the first operating mode image was classified as abnormal, at least one of the brightness of the light source, or the position of the light source, or the position of the lens or lens element relative to at least one of the camera or the workpiece may be adjusted.
[0009] In various embodiments, a synthetic data set may be created in which a plurality of training image features are altered to simulate abnormal images but not altered to simulate non-abnormal images. Additionally, a two-class classifier may be trained to be able to distinguish between abnormal images and non-abnormal images in the synthetic data set.
[0010] In various embodiments, a method for operating a workpiece inspection and defect detection system (e.g., a computer-implemented method operating under the control of one or more computing systems configured with executable instructions) may be provided, the method comprising: acquiring a plurality of training images using a light source, a lens, and a camera of the workpiece inspection and defect detection system, wherein at least some of the plurality of training images are used to train a defect detection portion; determining a plurality of training image features from the training images; determining a plurality of anomaly detector classification features based at least in part on the training image features; acquiring a first operating mode image of the workpiece; determining a plurality of features from the first operating mode image of the workpiece; and determining whether the first operating mode image of the workpiece is to be classified as abnormal using the anomaly detector classification features based at least in part on the features determined from the first operating mode image of the workpiece.
[0011] In various embodiments, a non-transitory computer-readable medium may be provided that stores program instructions that, when executed by one or more processors, cause the one or more processors to at least: acquire a plurality of training images using a light source, a lens, and a camera of a workpiece inspection and defect detection system, wherein at least some of the plurality of training images are used to train a defect detection portion; determine a plurality of training image features from the plurality of training images; determine a plurality of anomaly detector classification features based at least in part on the determined training image features; acquire a first operating mode image of a workpiece; determine a plurality of features from the first operating mode image of the workpiece; and determine, based at least in part on the features determined from the first operating mode image of the workpiece, using the anomaly detector classification features to determine whether the first operating mode image of the workpiece is to be classified as abnormal. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The foregoing aspects and many of the attendant advantages of the present invention will become more readily understood and better appreciated when reference is made to the following detailed description taken in conjunction with the accompanying drawings, in which:
[0013] Figure 1 is a diagram showing various typical components of a general precision machine vision inspection system;
[0014] Figure 2 is with Figure 1 A block diagram of a control system portion and a vision component portion of a machine vision inspection system similar to and including certain features disclosed herein;
[0015] Figures 3A-3F If you can use Figure 1 An embodiment of a machine vision inspection system similar to a machine vision inspection system for acquiring images of portions of a workpiece that do not include defects.
[0016] Figures 4A-4E If you can use Figure 1 An embodiment of a machine vision inspection system is similar to a machine vision inspection system for acquiring images of portions of a workpiece including defects.
[0017] Figures 5A-5D If you can use Figure 1 Embodiments of a machine vision inspection system similar to a machine vision inspection system for acquiring images of similar portions of a workpiece with and without defects.
[0018] Figure 6 is a diagram illustrating utilizing one or more video tools to perform metrology operations on an image of a workpiece including defects.
[0019] Figures 7A-7J Is it possible to Figure 1 Embodiments of a machine vision inspection system of similar images of similar portions of a workpiece with and without defects and with varying brightness levels acquired by a similar machine vision inspection system.
[0020] Figures 8A-8F Is it possible to Figure 1 Embodiments of a machine vision inspection system of images of similar portions of a workpiece with and without defects and with varying focus levels acquired by a similar machine vision inspection system.
[0021] Fig. 9 is an embodiment of an image enhancement user interface according to the present disclosure.
[0022] Fig.10 and Fig.11 shows the operation for Figure 1 A machine vision inspection system is similar to a flow chart of a method of a machine vision inspection system. DETAILED DESCRIPTION
[0023] Figure 1 1 is a block diagram of an exemplary machine vision inspection system 10 that may be used as an imaging system in accordance with the methods described herein. The machine vision inspection system 10 includes a vision inspection machine 12 operably connected to exchange data and control signals with a control computer system 14. The control computer system 14 is further operably connected to exchange data and control signals with a monitor or display 16, a printer 18, a joystick 22, a keyboard 24, and a mouse 26. The monitor or display 16 may display a user interface suitable for controlling and / or programming the operation of the machine vision inspection system 10. It should be appreciated that in various exemplary embodiments, a touch screen tablet and / or similar devices, etc. may replace and / or redundantly provide the functionality of any or all of the elements 14, 16, 22, 24, and 26.
[0024] Those skilled in the art will appreciate that any suitable computing system or device including a distributed or networked computing environment, etc., can generally be used to implement the control computer system 14 and / or other control systems described herein. Such computing systems or devices may include one or more general or special processors (e.g., non-customized or customized devices) that execute software to perform the functions described herein. The software may be stored in a memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory, etc., or a combination of such components. The software may also be stored in one or more storage devices such as an optical disk, a flash memory device, or any other type of non-volatile storage medium for storing data. The software may include one or more program modules, which include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In a distributed computing environment, the functionality of the program modules may be combined or distributed across multiple computing systems or devices, and the functionality of the program modules may be accessed via service calls in a wired or wireless configuration.
[0025] The visual inspection machine 12 includes a movable workpiece stage 32 and an optical imaging system 34, which may include a zoom lens or an interchangeable objective lens. The zoom lens or interchangeable objective lens typically provides various magnifications for the image provided by the optical imaging system 34. Various exemplary embodiments of the machine vision inspection system 10 are also described in commonly assigned U.S. Patent Nos. 7,454,053, 7,324,682, 8,111,905, and 8,111,938, each of which is hereby incorporated by reference in its entirety.
[0026] Figure 2 is a method that includes certain features disclosed herein and Figure 1 1 and 10. The control system portion 120 and the vision component portion 200 of the machine vision inspection system 100 are similar to the machine vision inspection system of FIG. 1. As will be described in more detail below, the control system portion 120 is used to control the vision component portion 200. The control system portion 120 can be arranged to exchange data and control signals with the vision component portion 200. The vision component portion 200 includes an optical assembly portion 205, light sources 220, 230, 240, 300, and a workpiece stage 210 having a central transparent portion 212. The workpiece stage 210 can be controllably moved along an x-axis and a y-axis, which are located in a plane generally parallel to the surface of the stage on which the workpiece 20 can be positioned.
[0027] Optical assembly portion 205 includes camera system 260 and interchangeable objective lens 250. In some embodiments, optical assembly portion 205 may optionally include a variable focus length (VFL) lens, for example, an adjustable acoustic gradient (TAG) such as disclosed in U.S. Pat. No. 9,143,674, which is hereby incorporated by reference in its entirety.
[0028] In various exemplary embodiments, the optical assembly portion 205 may also include a turret lens assembly 223 having lenses 226 and 228. As an alternative to the turret lens assembly, in various exemplary embodiments, a fixed or manually interchangeable magnification changing lens or zoom lens configuration, etc. may be included. In various exemplary embodiments, the interchangeable objective lens 250 may be selected from a set of fixed magnification objective lenses included as part of the variable magnification lens portion (e.g., a set of objective lenses corresponding to magnifications such as 0.5x, 1x, 2x or 2.5x, 5x, 10x, 20x or 25x, 50x, 100x, etc.).
[0029] The optical assembly portion 205 can be controllably moved along a z-axis, which is generally orthogonal to the x-axis and y-axis, by using a controllable motor 294, which drives an actuator to move the optical assembly portion 205 along the z-axis to change the focus of the image of the workpiece 20. The controllable motor 294 is connected to the input / output interface 130 via a signal line 296 to change the focus of the image over a certain range. The workpiece 20 can be placed on a workpiece stage 210. The workpiece stage 210 can be controlled to move relative to the optical assembly portion 205 so that the field of view of the interchangeable objective lens 250 is moved between various positions on the workpiece 20 and / or among multiple workpieces 20.
[0030] One or more of the stage light source 220, the coaxial light source 230, and the surface light source 240 (e.g., a ring light) may emit source light 222, 232, and / or 242, respectively, to illuminate one or more workpieces 20. For example, during image exposure, the coaxial light source 230 may emit source light 232 along a path including a beam splitter 290 (e.g., a partial reflector). The source light 232 is reflected or transmitted as workpiece light 255, and the workpiece light for imaging passes through the interchangeable objective lens 250 and the turret lens assembly 223 and is collected by the camera system 260. The workpiece image exposure including the image of the workpiece 20 is captured by the camera system 260 and output to the control system portion 120 on the signal line 262.
[0031] Various light sources (e.g., light sources 220, 230, 240, 300) can be connected to the lighting control interface 133 of the control system portion 120 via relevant signal lines (e.g., buses 221, 231, 241, 331, respectively). The control system portion 120 can control the turret lens assembly 223 to rotate along the axis 224 via the signal line or bus 223' to select the turret lens to change the image magnification.
[0032] like Figure 2 As shown, in various exemplary embodiments, the control system portion 120 includes a controller 125, an input / output interface 130, a memory 140, a workpiece program generator and actuator 170, and a power supply portion 190. Each of these components and the additional components described below can be interconnected through one or more data / control buses and / or application programming interfaces or through direct connections between various components. The input / output interface 130 includes an imaging control interface 131, a motion control interface 132, an illumination control interface 133, and a lens control interface 134.
[0033] The illumination control interface 133 may include illumination control elements 133a-133n that control, for example, the selection, power, and on / off switching of various corresponding light sources for the machine vision inspection system 100. The illumination control interface 133 also includes an illumination control element 133sip, which, in the illustrated embodiment, may work in conjunction with the structured illumination pattern (SIP) generation portion 300 to provide structured illumination during image acquisition. In various embodiments, a projection pattern may be output from the SIP generation portion 300 to be input to the beam splitter 290, where the projection pattern is directed as coaxial light through the objective lens 250 to provide SIP structured light 232' to illuminate the field of view.
[0034] The memory 140 may include an image file memory portion 141, a defect detection portion 140dp, a workpiece program memory portion 142 which may include one or more part programs, etc., and a video tool portion 143. The video tool portion 143 includes a video tool portion 143a and other video tool portions (e.g., 143n) that determine a GUI, image processing operations, etc. for each of the corresponding video tools, and a region of interest (ROI) generator 143roi that supports automatic, semi-automatic, and / or manual operations that define various ROIs that can be operated in various video tools included in the video tool portion 143. Embodiments of the operation of such video tools for locating edge features and performing other workpiece feature inspection operations are described in more detail in certain previously incorporated references and in U.S. Pat. No. 7,627,162, which is hereby incorporated by reference in its entirety.
[0035] The video tool portion 143 also includes an autofocus video tool 143af that determines a GUI, image processing operations, etc. for focus height measurement operations. In various exemplary embodiments, the autofocus video tool 143af may additionally include a high-speed focus height tool that can be used to measure focus height at high speed using hardware, as described in more detail in U.S. Patent No. 9,143,674, which is hereby incorporated by reference in its entirety. In various exemplary embodiments, the high-speed focus height tool can be a special mode of the autofocus video tool 143af that can be operated in other ways according to conventional methods for autofocus video tools, or the operation of the autofocus video tool 143af can include only those operations of the high-speed focus height tool. High-speed automatic focus and / or focus position determination for one or more image regions of interest can be based on analyzing images according to known methods to determine corresponding quantitative contrast metrics for each region. For example, such methods are disclosed in U.S. Patent Nos. 8,111,905, 7,570,795 and 7,030,351, which are hereby incorporated by reference in their entirety.
[0036] In the context of the present disclosure, and as known by those of ordinary skill in the art, the term "video tool" generally refers to a relatively complex set of automated or programmed operations that a machine vision user can implement through a relatively simple user interface. For example, a video tool may include a set of complex pre-programmed image processing operations and calculations that are applied and customized in a particular instance by adjusting several variables or parameters that govern the operations and calculations. In addition to the underlying operations and calculations, the video tool also includes a user interface that allows the user to adjust those parameters for a particular instance of the video tool. It should be noted that sometimes visible user interface features are referred to as video tools while implicitly including the underlying operations.
[0037] One or more display devices 136 (e.g., Figure 1 16) and one or more input devices 138 (e.g., Figure 1 The display device 136 and the input device 138 may be used to display a user interface, which may include various graphical user interface (GUI) features that may be used to perform inspection operations, and / or create and / or modify part programs, view images captured by the camera system 260, and / or directly control the vision component portion 200.
[0038] In various exemplary embodiments, when a user utilizes the machine vision inspection system 100 to create a part program for a workpiece 20, the user generates part program instructions by operating the machine vision inspection system 100 in a learning mode to provide a required image acquisition training sequence. For example, the training sequence may include locating a specific workpiece feature of a representative workpiece in a field of view (FOV), setting light levels, focusing or autofocusing, acquiring an image, and providing an inspection training sequence applied to the image (e.g., an instance of using one of the video tools for the workpiece feature). The learning mode operates so that the sequence is captured or recorded and converted into corresponding part program instructions. When the part program is executed, these instructions will cause the machine vision inspection system to reproduce the trained image acquisition and cause the inspection operation to automatically inspect that the specific workpiece features (i.e., corresponding features in corresponding positions) are located on one or more operating mode workpieces that match the representative workpiece used when creating the part program. In various exemplary embodiments, certain types of training modes (e.g., training modes for training a defect detection portion to detect defects, training modes for training anomaly detector portions to detect abnormal images that are unacceptable to the defect detection process, etc.) may be utilized in the same or alternative manner.
[0039] The video tool portion 143 also includes a Z height measurement tool portion 143z, which provides various operations and features related to Z height measurement operations. In one embodiment, the Z height measurement tool portion 143z may include a Z height tool 143zt. For example, the Z height tool 143zt may include an autofocus tool 143af and a multi-point autofocus tool 143maf. The Z height tool 143zt may control certain aspects of image stack acquisition and related structured light pattern generation operations in conjunction with a Z height tool configured in a mode for determining the best focus height and / or Z height measurement results. Typically, the Z height measurement tool portion 143z may perform at least some operations similar to known Z height measurement tools, for example, performing operations in a learning mode and / or a run mode or other mode to generate all or part of a focus curve and find its peak as the best focus position. For example, certain known operations for a Z height measurement tool are described in U.S. Patent No. 10,520,301, which is hereby incorporated herein by reference in its entirety.
[0040] The defect detection portion 140dp performs various defect detection operations, as will be described in more detail below. Prior to performing such defect detection operations, the defect detection portion 140dp is trained using a training image set captured using specific (e.g., current) imaging, lighting, and workpiece conditions. When the defect detection portion 140dp is subsequently used for defect detection in an online or offline environment over time, these conditions may change significantly (e.g., drift). If the appearance of the images used during defect detection is substantially different from the appearance of the images in the training image set, the performance of the trained defect detection portion 140dp may deteriorate. Thus, the defect detection portion 140dp may include a monitoring (e.g., anomaly detection) system that may be created / defined (e.g., using some or all of the training image set when training the defect detection portion 140dp) and is capable of distinguishing between normal (e.g., acceptable) input images and abnormal (e.g., unacceptable) input images, and alerting a user if the appearance of the input image during defect detection is no longer satisfactory and may negatively affect the accuracy of the trained defect detection portion 140dp, thereby effectively improving the reliability of defect detection performed by the defect detection portion 140dp. With respect to classification of images, the terms "normal" and "non-abnormal" are generally considered to have the same meaning and are used interchangeably herein.
[0041] In various exemplary embodiments, the defect detection portion 140dp may implement a defect detection process that is performed in conjunction with a metrology process. In various embodiments, it may be desirable to include the defect detection portion 140dp in conjunction with a Figure 1 In a machine vision inspection system similar to the machine vision inspection system 100 of the present invention, the system is configured to perform a measurement process because the system is capable of generating image data that is input into a defect detection process implemented by the defect detection portion 140dp. Therefore, a single machine is configured to perform both a measurement process and a defect detection process, which can provide advantages over conventional measurement systems. For example, if a defect is detected in a workpiece while a defect detection process is being performed, there may be no reason to perform a measurement process on the workpiece, which can save time. More specifically, if a defect is detected in a workpiece while a defect detection process is being performed, it may not be necessary to measure a part that is undoubtedly defective. Therefore, it may be advantageous to perform a defect detection process before starting the measurement process.
[0042] Additionally, some defects may require further measurement or inspection to determine additional defect parameters. For example, a 2D image may enable quick identification of a likely defect and enable quick ascertainment of the XY location of the defect and the approximate XY area of the defect. If the 3D nature of the potential defect is important, the defect detection portion 140dp may cause additional processing (e.g., metrology operations) to be performed to determine whether the potential defect is an actual defect. For example, if a scratch in the surface of a workpiece must be deeper than a certain threshold to be considered a defect, the defect detection portion 140dp may cause a more time-consuming 3D point cloud of the affected area to be acquired (e.g., using a z-height measurement tool portion 143z) to understand whether the scratch is deep enough to cause the part to be rejected. In various embodiments, different actions may be performed as a result of the initial defect classification (e.g., such as may be programmed to occur automatically), such as (1) continuing a standard metrology process, (2) stopping or pausing the defect detection process and performing a metrology process that includes more informative measurements of the potential defect (e.g., 3D, different lighting, touch probe measurements such as surface roughness, etc.), (3) sending the workpiece to scrap (e.g., discarding or recycling the workpiece), (4) sending the workpiece for additional manual inspection, (5) providing feedback to the production line indicating that a machine or process error may be occurring, etc.
[0043] In various exemplary embodiments, the defect detection portion 140dp of the memory 140 stores model data and program instructions for various tools and algorithms of the defect detection system, which can be used to infer whether various types of defects are present in an image of a workpiece (i.e., as indicated on a portion of the workpiece surface included in the image). While the machine vision inspection system is operated in a training or learning mode, the defect detection system trains the defect detection portion 140dp using a training image set including defective images and non-defective images. The training image set is captured using specific (e.g., current) imaging, lighting, and workpiece conditions. After the defect detection portion 140dp is initially trained, the defect detection portion 140dp is operated in an operating mode to infer whether defects are included in new, previously unseen workpiece images, and to classify each workpiece image as defective or non-defective accordingly.
[0044] As will be described in more detail below, Figures 3A-3F , Figures 4A-4E and Figures 5A-5DSome examples of workpiece images are shown that may be acquired by the machine vision inspection system 100 and may be used in some embodiments to train the defect detection portion 140dp during the training mode (and / or some images may be examples of workpiece images acquired later by the machine vision inspection system 100 during the operating mode and may be analyzed by the trained defect detection portion 140dp). In particular, Figures 3A-3F and Figures 4A-4E The workpiece image of the embodiment has different cross sections (i.e., at different XY positions) on the processed aluminum plate, and for this Figures 3A-3F An example of a defect-free image is shown, while Figures 4A-4E An example of defect images is shown (e.g., for which a training image set will typically include both a number of defective workpiece images and non-defective workpiece images), as will be described in more detail below. The differences between the images help explain, in part, why it may be desirable to train the defect detection portion 140dp using a number of training images. More specifically, due in part to the types of workpieces that may be acquired for inspection (e.g., as a set of workpiece images for inspection of a type of aluminum sheet). Figures 3A-3F and Figures 4A-4E The different possible characteristics of different workpiece images (illustrated as the difference between) can improve the accuracy of the defect detection portion 140dp for detecting defects by utilizing various workpiece images for training, which can be similar to various images that can be acquired later during the operating mode and / or can otherwise better enable detection of defects in such images. For example, with specific regard to Figures 3A-3F , Figures 4A-4E and Figures 5A-5D In an embodiment, such training can help the defect detection portion 140dp distinguish between defects (e.g., which in the illustrated embodiment may include various different types of scratch defects, etc.) and features of a normal workpiece surface (e.g., which in the illustrated embodiment may include various different types of processing marks formed on the surface of the plate that appear as a hash texture that varies across the surface, etc.).
[0045] While the defect detection portion 140dp is operated in the run mode, the imaging, lighting, and workpiece conditions used in the machine vision inspection system may change significantly (e.g., drift, etc.) over time, for example, due to aging of one or more light sources illuminating the workpiece when obtaining image data, accumulation of dust (e.g., on the lens), optical misalignment, changing manufacturing conditions, and other random or systematic factors. As a result, the images input to the defect detection portion 140dp during the run mode analysis may become substantially different from the images used to train the defect detection portion 140dp, which may cause significantly worse defect detection performance. For example, more false positives may be caused by the defect detection portion 140dp interpreting the changed imaging conditions on a non-defective image as a defect. As another example, due to the changed image conditions, the defect detection portion 140dp may have difficulty detecting or otherwise distinguishing actual defects from normal workpiece surface features (e.g., machining marks, etc.).
[0046] Thus, the defect detection portion 140dp includes an image data monitor and an anomaly detector subsystem that creates a generalized model of a training image set (e.g., including defective and non-defective images, and may include all or some of the available training images) and stores the generalized model with the defect detection portion 140dp for subsequent use in monitoring the quality of the images during defect detection. As will be described in more detail below, in various embodiments, the determination of the generalized model may include determining training image features from the training images and determining anomaly detector classification characteristics based on the training image features. Depending on the purpose of the model, if the appearance / quality of the image used for defect detection deviates too far from the appearance / quality of the image used to train the defect detection portion 140dp (e.g., the image may be classified as abnormal for this), a designed action may be performed (e.g., a user may be alerted and / or the system may be adjusted, etc.). For example, the user or the system may be able to adjust imaging conditions, lighting conditions, and / or workpiece conditions, or retrain the defect detection portion 140dp to compensate for the deviation and maintain good performance of the defect detection system.
[0047] It will be appreciated that, as described above (and as indicated by Figures 3A-3F , Figures 4A-4E and Figures 5A-5DThe various potential training images (illustrated by various examples of ) help illustrate the importance of utilizing a set of training images to determine a generalized model. More specifically, because training images can have varying characteristics, it is beneficial to determine an accurate representation of the differences (e.g., as ranges or other measures corresponding to different training image features and associated anomaly detector classification characteristics) in accordance with the principles disclosed herein, which helps ensure the accuracy of the anomaly detection process used to classify certain images as anomalous (e.g., as having characteristics that have deviated beyond some acceptable range, threshold, etc. of characteristics as determined based on the training images, etc.).
[0048] As pointed out above, Figures 3A-3F and Figures 4A-4E is an embodiment of a workpiece image of a cross section of a workpiece of a relatively "flat" machined aluminum plate. Each image is taken from the same angular viewpoint (e.g., directly above the plate at a 90 degree angle), but each is taken at a different XY position on the plate. Each image shows a view of the surface of the plate that is approximately 2.5 mm by 1.9 mm (XY). Thus, each image shows a magnified view of a portion of the surface of the plate. The machining marks formed on the surface of the plate create a hash texture that generally varies across the surface. In the image, at such a magnification ratio, a relatively flat plate may appear uneven. In the current example, the height of some of the hash machining mark ridges may be approximately 5 microns or less.
[0049] Figures 3A-3F The cross section of the machined aluminum plate shown does not include defects. In other words, Figures 3A-3F An example of an image of a cross section of a "defect-free" machined aluminum plate is shown. In contrast, Figures 4A-4E An example of an image of a cross section of a workpiece including a defect is shown. Figures 4A-4E The image shown is similar to Figures 3A-3F The images shown are similar except that Figures 4A-4E The images include defects 402 formed on the surface of the processed aluminum plate. The defects 402 in these examples are scratches formed in the surface of the processed aluminum plate. More specifically, Figure 4A Scratch defects 402A1 and 402A2 are shown. Figure 4B Scratch defects 402B1 and 402B2 are shown. Figure 4C Scratch defects 402C1 and 402C2 are shown. Figure 4D A scratch defect 402D is shown, and Figure 4E Scratch defects 402E1 and 402E2 are shown. As noted above, Figures 3A-3F and Figures 4A-4EThe various characteristics of the defects and workpiece surfaces illustrated help illustrate, in part, why it may be desirable to utilize many training images to train the defect detection portion 140dp (e.g., for which the training image set will typically include many defective workpiece images and non-defective workpiece images), and accordingly why it may be desirable to utilize a variety of training images to determine training image features and related anomaly detector classification characteristics that help improve the accuracy of the anomaly detection process.
[0050] Figures 5A-5D If you can use Figure 1 Embodiments of a machine vision inspection system similar to a machine vision inspection system for acquiring images of similar portions of a workpiece with and without defects. Figure 5A An example cross section of a panel that may be classified as "defect-free" is shown. Figures 5B-5D An example of images of a cross section of a panel that may be classified as a "defect" is shown, where each image includes a scratch defect 502 (e.g., including corresponding scratch defects 502B, 502C, and 502D). Figures 5A-5D In an example embodiment, the images are of similar portions of the workpiece (e.g., whereby the type of machining marks formed on the surface of each portion are similar or nominally the same in each image, and the primary difference between the images is the characteristics of the corresponding scratch defects 502B, 502C, and 502D). Figures 5A-5C The images may be included as part of a training image set for training the defect detection portion 140dp. Figure 5D An image of may be an example of an operating mode image that may be analyzed by the defect detection portion 140dp to determine whether it should be classified as a defect image or a non-defect image. In various embodiments, the defect detection portion 140dp may have been well trained to be able to classify Figure 5D The image is appropriately classified as a defective image (e.g., as has been Figures 5A-5C In various embodiments, certain additional processes may be performed with respect to the example scratch defect 502D. For example, one or more metrology processes may be performed in conjunction with the defect detection process, for which various sizes or other characteristics of the example scratch defect 502D may be determined, as will be described below with respect to Figure 6 As will be described below with respect to Figures 7A-7J and Figures 8A-8F As described in more detail, it is possible to have various operating mode images (e.g., which in certain embodiments may be similar and consistent with Figure 5A and Figure 5D Abnormal changes occur in the image for comparison).
[0051] Figure 6 1 is a diagram illustrating the use of one or more video tools to perform metrology operations (e.g., to determine the size of the defect, etc.) on an image of a workpiece including a defect. As shown in the figure, for an image 601 (e.g., which may be similar to or the same as the scratch defect 502D) including a scratch defect 602 (e.g., which may be similar to or the same as the scratch defect 502D), Figure 5D The video box tool 606 includes scan lines 608 (e.g., which may also or alternatively represent a video point tool, etc.) that are used to determine edge locations, sizes, and / or other aspects of the scratch defect 602. In various exemplary embodiments, the video box tool 606 may be resized, positioned, and rotated until the box tool 606 indicates or defines an area of interest (e.g., an area within the box tool 606), and the area of interest may be determined using the video box tool 606. Figure 6 Arrows (e.g., representing scan lines, point tools, etc.) are shown to determine the edges of the scratch defect 602. In various exemplary embodiments, the video box tool 606 may generally use one or more general edge gradients along the edges of the defect 602 in the region of interest, and may determine the edges of the defect 602 based on the local magnitude of the edge gradients along various scan lines 608, etc.
[0052] In various exemplary embodiments, such metrology operations may also include performing certain morphological or other filtering (e.g., to distinguish between the edges of scratches and the machining pattern of the workpiece, certain types of such filtering are described in U.S. Pat. No. 7,522,763, which is hereby incorporated by reference in its entirety). Figure 6 As shown, in the display area included in the image, the edge position (e.g., outer edge or perimeter) of the scratch defect 602 is determined using a box tool 606 with a scan line 608. Based on such a determination, the video tool and / or other measurement operation may include determining a dimension D1 of the scratch defect (e.g., corresponding to the length or other dimension of the scratch defect 602). In various exemplary embodiments, other dimensions (e.g., width, depth, etc.) of the scratch defect 602 may be determined using the box tool 606, the scan line 608, and / or other video tools and / or measurement operations. For example, as described above, the video tool portion 143 may include a Z height measurement tool portion 143z, for which the corresponding video tool or operation may be used to determine the Z height dimension of the scratch defect (e.g., including determining the depth of the scratch relative to other portions or features of the workpiece surface, etc.).
[0053] As part of the general operation of the defect detection portion, some of the defects detected may require further measurement or inspection to determine additional defect parameters. For example, as noted above, various types of analysis and / or processing of the defect image including the scratch defect 602 may enable the XY location and approximate XY area and / or other dimensions of the defect 602 to be determined (e.g., using video tools and / or other operations as described above). If the 3D nature of the potential defect is important (e.g., if the scratch must be deeper than a certain value to be considered a defect), the defect detection portion 140dp may initiate a process for utilizing a Z height measurement tool (e.g., of the Z height measurement tool portion 143z) or other 3D sensing process (e.g., for acquiring a 3D point cloud of the affected area to determine the depth of the scratch, etc.).
[0054] As will be described in more detail below, Figures 7A-7J Is it possible to Figure 1 similar machine vision inspection systems with and without defects (e.g., Figure 5A and Figure 5DWith respect to varying brightness levels, in various exemplary embodiments, a set of training images may be analyzed to determine training image features (e.g., related to brightness and / or other features), and corresponding anomaly detector classification characteristics may be determined based on the training image features, as described above. In accordance with such determinations, in a particular exemplary embodiment, the machine vision inspection system 100 may be determined to be capable of providing robust accuracy performance relative to the training images, to approximately ±15% of the baseline brightness of the table light source 220, the coaxial light source 230, and / or the surface light source 240 used to illuminate the workpiece 20. As the table light source 220, the coaxial light source 230, and the surface light source 240 are used over time, the brightness of each light source may degrade. Additionally or alternatively, the brightness of the table light source 220, the coaxial light source 230, and the surface light source 240 may be inadvertently set to a value that is significantly different from the brightness used to acquire the training images, resulting in poor image defect detection accuracy performance. Additionally or alternatively, the camera aperture or other light-transmitting optical element (e.g., lens, reflector, filter, etc.) can be changed according to the conditions or settings used to acquire the training image, thereby causing more or less light to reach the camera imager. Additionally or alternatively, the ambient room light can be changed according to its intensity when acquiring the training image, thereby causing more or less light to reach the camera imager. Therefore, it may be desirable to examine the image data generated by the machine vision inspection system 100 to determine whether the brightness of the image being inspected is greater than ±15% of the baseline brightness (e.g., ±15% of this baseline brightness may represent an anomaly detector classification characteristic), and perform a specified action (e.g., to alert a user that there is a problem with the lighting conditions of the workpiece 20 and / or to adjust one or more light settings, etc.) if the brightness is outside the range that results in robust defect detection accuracy performance.
[0055] Therefore, the defect detection section 140dp may include an image data monitor and anomaly detector subsystem that is capable of processing image data corresponding to an image of a workpiece and detecting whether the image data is generated with a brightness outside a range that results in robust accuracy performance (e.g., ±15% of a baseline brightness). After the image data monitor and anomaly detector subsystem have been trained using a training image set, the image data monitor and anomaly detector subsystem is capable of detecting whether the operational image data is generated with a brightness outside the range that results in robust accuracy performance.
[0056] Figures 7A-7J Is it possible to Figure 1 Embodiments of a machine vision inspection system similar to that of a machine vision inspection system such as images of similar portions of a workpiece with and without defects and with varying brightness levels acquired during an operational mode and / or as may be used as training images. Figures 7A-7J In embodiments, the images are of similar portions of the workpiece (eg, whereby the types of machining marks formed on the surface of each portion are similar or nominally the same in each image and the primary difference between the images is brightness level). Figures 7A-7E The image is a defect-free image (e.g., as can be seen with Figure 5A ), while Figure 7F-7J The image is a defect image (for example, as can be seen with Figure 5D as for comparison and as including corresponding defects 702F-702J similar to defect 502D).
[0057] In one example implementation, the image data monitor and anomaly detector subsystem may be trained to classify image data as “normal” or “non-anomalous” if the image data indicates a brightness that is ±15% of a baseline brightness (e.g., average brightness), and to classify image data as “abnormal” or “abnormal” if the image data indicates a brightness that is not ±15% of the baseline brightness. For example, if the image data monitor and anomaly detector subsystem determines that the image data indicates a brightness that is ±25% of the baseline brightness, the image data monitor and anomaly detector subsystem may classify the image data as “abnormal” or “abnormal.”
[0058] Fig. 7A is a defect-free image of a workpiece having a baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem may Fig. 7A The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0059] Figure 7B is a defect-free image of a workpiece whose brightness is 25% lower than the baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem may Figure 7B The image data corresponding to the image is classified as "abnormal" or "abnormal".
[0060] Figure 7C is a defect-free image of a workpiece whose brightness is 15% lower than the baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem may Figure 7C The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0061] Fig.7D is a defect-free image of a workpiece with a brightness 15% higher than the baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem may Fig.7D The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0062] Fig. 7E is a defect-free image of a workpiece captured while illuminating the workpiece with a brightness 25% higher than the baseline brightness. Fig. 7E The image data corresponding to the image is classified as "abnormal" or "abnormal".
[0063] Figure 7F is a defect image of a workpiece having a baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem may Figure 7F The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0064] Figure 7G is a defect image of a workpiece whose brightness is 25% lower than the baseline brightness. Figure 7G The image data corresponding to the image is classified as "abnormal" or "abnormal".
[0065] Figure 7H is a defect image of a workpiece whose brightness is 15% lower than the baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem can Figure 7H The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0066] Fig.7I is a defect image of a workpiece whose brightness is 15% higher than the baseline brightness. According to this embodiment, the image data monitor and anomaly detector subsystem can Fig.7I The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0067] Figure 7J is a defect image of a workpiece whose brightness is 25% higher than the baseline brightness. Figure 7J The image data corresponding to the image is classified as "abnormal" or "abnormal".
[0068] It should be appreciated that although certain image pairs (e.g., Figure 7B and Figure 7C , Fig.7D and Fig. 7E , Figure 7G and Figure 7H and Fig.7I and Figure 7J) may not appear significantly different to the human eye in some cases, but such differences may be significant to the operation of the defect detection portion 140dp. For example, in some embodiments, a 25% difference in brightness level from a baseline brightness level may be determined to inhibit the defect detection portion 140dp from accurately distinguishing between defects (e.g., such as defects 702G and 702J) and features of a normal workpiece surface (e.g., Figure 7G and Figure 7J other parts of the surface in the image and / or with respect to determining the Figure 7B and Fig. 7E the ability to detect whether such defects exist in images of those that do not.
[0069] As will be described in more detail below, Figures 8A-8F Is it possible to Figure 1 similar machine vision inspection systems with and without defects (e.g., Figure 5A and Figure 5D With respect to varying focus levels, in various exemplary embodiments, a set of training images may be analyzed to determine training image features (e.g., related to focus and / or other features), and corresponding anomaly detector classification characteristics may be determined based on the training image features. Pursuant to such determination, in one particular exemplary embodiment, if images are acquired of a workpiece surface of interest within an acceptable depth of focus (DOF) range for the camera system 260, the machine vision inspection system 100 may be determined to be capable of providing robust accuracy performance relative to the training images. The camera system DOF may be compared to the distance of the workpiece from the best focus distance to indicate the relative amount of focus or defocus in the image. Images captured of the workpiece within a range of one DOF centered about the best focus distance may be considered to have excellent focus, whereas images captured of the workpiece at some larger distance from the best focus distance but below some threshold DOF range (e.g., 4 times DOF) may be considered to have an acceptable amount of defocus that still provides robust defect detection accuracy performance. In contrast, images captured of the workpiece outside a threshold DOF range (e.g., outside a 4x DOF range, such as at 6x DOF farther or closer to the camera system than the best focus distance) may be considered to have an unacceptable amount of defocus that will not provide robust defect detection accuracy performance.
[0070] Figures 8A-8F Is it possible to Figure 1 Embodiments of a machine vision inspection system similar to that of a machine vision inspection system such as images of similar portions of a workpiece with and without defects and with varying focus levels acquired during an operational mode and / or as may be used as training images. Figures 8A-8F In embodiments, the images are of similar portions of the workpiece (eg, whereby the types of machining marks formed on the surface of each portion are similar or nominally the same in each image, and the primary difference between the images is focus level). Figures 8A-8C The image is a defect-free image (e.g., as can be seen with Figure 5A ), while Figures 8D-8F The image is a defect image (for example, as can be seen with Figure 5D as for comparison and as including corresponding defects 802D-802F similar to defect 502D).
[0071] In one example implementation, the image data monitor and anomaly detector subsystem may be trained to classify image data as "normal" or "non-anomalous" if the image data indicates defocus within four times the system DOF, and to classify image data as "abnormal" or "abnormal" if the image data does not indicate defocus within four times the baseline DOF. For example, if the image data monitor and anomaly detector subsystem determines that the image data indicates defocus of eight times the system DOF, the image data monitor and anomaly detector subsystem may classify the image data as "abnormal" or "abnormal" (e.g., and may perform specified actions therefor, such as providing a warning to a user and / or automatically adjusting the system, such as adjusting the position of a lens or workpiece, etc. to achieve better focus).
[0072] Fig. 8A is a defect-free image of the workpiece that is in focus within the system DOF of best focus. Fig. 8A The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0073] Figure 8B is a defect-free image of a workpiece that is defocused from best focus by four times the system DOF. For example, if the defocus threshold for an acceptable image is 4 times the DOF, the amount of defocus in this image is within the allowed tolerance, so no user warning should be provided. Therefore, the image data monitor and anomaly detector subsystem can be compared with Figure 8B The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0074] Figure 8C is a defect-free image of a workpiece defocused from best focus by eight times the system DOF. Figure 8CAs a result, the image data monitor and anomaly detector subsystem issues a user warning and / or takes another action (e.g., adjusting a parameter or position of a component of the camera system 260 that produces an acceptable DOF) with respect to the image that is not deemed suitable for the defect detection process.
[0075] Fig.8D is a defect image of the workpiece having a focus within the system DOF of best focus. According to this embodiment, the image data monitor and anomaly detector subsystem can Fig.8D The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0076] Fig. 8E is a defect-free image of a workpiece defocused from best focus by four times the system DOF. Fig. 8E The image data corresponding to the image is classified as "normal" or "non-abnormal".
[0077] Figure 8F is a defect-free image of a workpiece defocused from best focus by eight times the system DOF. Figure 8F The image data corresponding to the image is classified as "abnormal" or "abnormal".
[0078] As mentioned above, with and Figures 7A-7J and Figures 8A-8F Image data corresponding to images similar to the images shown may correspond to operating mode images classified as normal / non-abnormal or abnormal / abnormal by the image data monitor and anomaly detector subsystem. In various embodiments, such images may also represent training images that may be included in a training image set for training the image data monitor and anomaly detector subsystem of the defect detection portion 140dp. Such images may be based on image data that is enhanced to have a specific quality useful in training the image data monitor and anomaly detector subsystem in some embodiments. For example, image data corresponding to an enhanced image set may be generated based on parameters indicating an enhancement limit. In various embodiments, parameters indicating an increase limit may be automatically defined.
[0079] Additionally, parameters indicating enhancement limits may be defined under user control (eg, for generating training images). Fig. 9 is an embodiment of an image enhancement graphical user interface 900 that can be used to define parameters indicative of enhancement limits. The image enhancement graphical user interface 900 displays a selected baseline image 902 .
[0080] The image enhancement graphical user interface 900 includes a check box 904 that, when checked, enables brightness enhancement functionality. The brightness enhancement functionality enables a user to specify a parameter indicating a lower brightness limit and a parameter indicating a higher brightness limit for generating an enhanced training image. More specifically, the graphical user interface 900 includes a slider 906 having a handle 908 that can be selected and moved with a user input device (e.g., a mouse) to set a value 910 of a parameter indicating a lower brightness limit (e.g., 5), which is displayed. A corresponding image 912 is displayed, which is a copy of the baseline image 902 modified to have a lower brightness based on the parameter indicating the lower brightness limit. Similarly, the graphical user interface 900 includes a slider 914 having a handle 916 that can be moved to set a value 918 of a parameter indicating a higher brightness limit (e.g., 7). A corresponding image 920 is displayed, which is a copy of the baseline image 902 modified to have a higher brightness based on the parameter indicating the higher brightness limit.
[0081] The image enhancement graphical user interface 900 also includes a check box 922 that, when checked, enables focus enhancement functionality. The focus enhancement functionality enables a user to specify parameters indicating a range of acceptable focus for generating an enhanced training image. More specifically, the graphical user interface 900 includes a slider 924 having a handle 926 that can be selected and moved to set a value 928 for a parameter indicating a range of acceptable focus with respect to a best focus distance as a factor of the system depth of focus (e.g., 3 times DOF). A corresponding image 930 is displayed that is a copy of the baseline image 902 that has been modified to be defocused based on the parameters indicating the depth of focus.
[0082] The image enhancement graphical user interface 900 also includes a check box 932 that, when checked, enables noise enhancement functionality. The noise enhancement functionality enables a user to specify a parameter indicating an amount of noise for generating an enhanced training image. More specifically, the graphical user interface 900 includes a slider 934 having a handle 936 that can be selected and moved to set a value 938 for a parameter indicating a maximum amount of digital noise (e.g., 10) above which robust defect detection accuracy is desired. A corresponding image 940 is displayed that is a copy of the baseline image 902 modified to include additional noise based on the parameter indicating the amount of noise.
[0083] The various parameters set using the image enhancement graphical user interface 900 can be used to generate additional image data for the enhanced image set. In various exemplary embodiments, the same values (e.g., 910, 918, 928, 938) of the various enhancement parameters used to generate image data corresponding to images 912, 920, 930, and 940 based on the baseline image 902 can be applied to all or a selected subset of the image data of the training image set to generate additional image data for the enhanced image set.
[0084] Fig.10 and Fig.11 shows the operation for Figure 1 1. The method 1000 begins at 1010. At 1010, a plurality of training images are acquired. As a specific example, in one embodiment, the plurality of training images may be acquired. Figures 3A-3F , Figures 4A-4E and Figures 5A-5D 140dp and use them as training images. It should be appreciated that, although a limited number of example training images are illustrated in the current figure, a different number of training images (e.g., 50, 100, 150, etc.) may be utilized in various embodiments. In various embodiments, at least some of the plurality of training images are utilized to train the defect detection portion. For example, the training images may include original images of a workpiece acquired with the camera system 260 and input to the defect detection portion 140dp when it is operating in a training mode (e.g., such as Figures 3A-3F , Figures 4A-4E and Figures 5A-5D In addition, the plurality of training images may include, for example, Fig. 9 The method 1000 then proceeds to 1020 to create an enhanced image set using the image enhancement graphical user interface 900 .
[0085] At 1020, a plurality of training image features are determined from the plurality of training images acquired at 1010. In various embodiments, the training image features (e.g., as determined automatically and / or at least in part by a system designer) may include, for example, average image brightness, standard deviation of brightness, grayscale histogram, dominant frequency (e.g., obtained by performing a fast Fourier transform (FFT)), and average contrast measure. Method 1000 then proceeds to 1030.
[0086] At 1030, a plurality of anomaly detector classification characteristics are determined based at least in part on the plurality of training image features determined at 1020. In various exemplary embodiments, an automated process and / or a system designer may determine anomaly detector classification characteristics based at least in part on the determined training image features. For example, it may be determined that image data whose average image brightness is not within ±15% of the baseline image brightness will be classified as abnormal, and / or image data indicating a defocus from best focus greater than four times the system depth of focus (DOF) will be classified as abnormal; otherwise, the image data will be classified as normal or non-abnormal. In various embodiments, the training image features are automatically extracted (e.g., by a pre-trained convolutional neural network, etc.), and to this end, a generalized model of a "normal" training data set may be developed (e.g., defined, trained, fitted, etc.) based on the extracted features (e.g., different types of models for this may include: manually defined thresholds on selected features; thresholds based on vector similarity (e.g., as may be manually tuned); single-class support vector machine (anomaly) detector); or autoencoder (anomaly detector, etc.)). In various alternative embodiments, synthetic data sets may also be created in which various features may be altered / changed / interfered (or not altered / changed / interfered) to simulate abnormal images and normal images, and for this purpose, two-class classifiers may be trained to be able to distinguish between normal and abnormal images in the synthetic data set. In various exemplary embodiments, two-class classifiers may be used to classify features, and may be based on logistic regression analysis, nearest neighbor analysis, discriminant analysis, naive Bayesian classifiers, decision trees, support vector machines (SVMs), neural networks, ensembles of classifiers, or convolutional networks trained on ImageNet, such as described by Zeiler and Fergus in "Visualizing and Understanding Convolutional Networks", arXiv:1311.2901v3[cs.CV]28Nov 2013, which is hereby incorporated by reference in its entirety. In conjunction with such processes, features that make an image abnormal (e.g., causes of abnormality, etc. may be identified). In various embodiments, when the training images include a set of enhanced images, features may be extracted from the enhanced images and to this end at least some of the features may be defined at least in part by the type of enhancement utilized (e.g., average image brightness, standard deviation of brightness, grayscale histogram, dominant frequency (FFT)), average contrast measures, image noise measures, etc., and to this end in some cases a relatively simple limit-based anomaly detector may be created that considers features within the range of the set of enhanced images to be "normal" and features outside of these ranges to be "abnormal" (e.g., such as may also allow for relatively simple identification of features (i.e., causes) that caused an image to be classified as abnormal. Method 1000 then proceeds to 1040.
[0087] At 1040 , an operational mode image of the workpiece is acquired during the operational mode. For example, the same camera system used to acquire the training image at 1010 is used to acquire the operational mode image of the workpiece. The method 1000 then proceeds to 1050 .
[0088] At 1050 , a plurality of features from the run mode image are determined. For example, the features from the run mode image may include a value corresponding to an average brightness of the run mode image and a value corresponding to a defocus level. Method 1000 then proceeds to 1060 .
[0089] At 1060, the operating mode image acquired at 1040 is classified as non-anomalous (e.g., acceptable for accurate defect detection) or abnormal (e.g., unacceptable for accurate defect detection) based on the plurality of features of the operating mode image determined at 1050 using the anomaly detector classification characteristic determined at 1030. For example, if at 1050 a determination is made that the average brightness of the operating mode image has a value that has an abnormal characteristic and the operating mode image has a focus that corresponds to the baseline DOF, then the operating mode image is classified as abnormal based on the anomaly detector classification characteristic determined at 1030. As an example, if a determination is made that the operating mode image is classified as abnormal (e.g., as unacceptable for defect detection) or non-anomalous, the image may be labeled as abnormal or non-anomalous, respectively, accordingly (e.g., for which the labeled condition may be stored or otherwise indicated in association with the image data, etc.). Method 1000 then proceeds to 1070.
[0090] At 1070, a determination is made as to whether the operational mode image is classified as abnormal. For example, a determination is made as to whether a file including image data corresponding to the operational mode image is marked as abnormal or non-abnormal, such as indicating a corresponding abnormal classification or non-abnormal classification. If it is determined that the operational mode image is classified as abnormal, the method 1000 proceeds to 1080. If it is not determined that the operational mode image is classified as abnormal, the method 1000 proceeds to 1090.
[0091] At 1080, a process for an abnormal image is performed. For example, a designed action may be performed (e.g., a user may be alerted and / or the system may be adjusted, etc.). In various embodiments, the cause of the abnormality may be identified (e.g., one or more responsible for the image abnormality may be identified), and in some cases, a message or other indication may be provided to the user about why the image is abnormal and / or the system may be adjusted accordingly, etc. In various embodiments, adjustments to the system may include adjusting imaging conditions, lighting conditions, focusing conditions, and / or workpiece conditions. Examples of such adjustments may include adjusting the brightness of a light source, adjusting the position of an objective lens, and / or adjusting the position of a workpiece, etc.
[0092] Optionally at 1090 (or as may be performed at other times and / or in parallel with portions of method 1000), a determination is made using a defect detection portion to determine whether the operational mode image is to be classified as a defect image. For example, image data corresponding to the operational mode image may be input to a defect detector (e.g., as part of defect detection portion 140dp). In an example embodiment, the defect detector may be based on multi-label image classification with a neural network implemented by defect detection portion 140dp and trained to detect defects using a training image set. In response, the defect detection portion may identify a plurality of pixels within the workpiece image that are characteristic of a defect, and may accordingly determine that the operational mode image is to be classified as a defect image. Method 1000 then proceeds to 1095.
[0093] At 1095, a determination is made as to whether additional operating mode images are to be acquired. For example, in response to continuing to acquire multiple images and / or a user pressing a button or selecting an icon using a graphical user interface to indicate that another workpiece image is to be inspected, a determination is made that additional operating mode images are to be acquired. If additional operating mode images are to be acquired, method 1000 returns to 1040. If additional operating mode images are not to be acquired, method 1000 ends.
[0094] Although the preferred embodiments of the present disclosure have been shown and described, based on this disclosure, those skilled in the art will appreciate many variations of the arrangement of features shown and described and the sequence of operations. Various alternative forms may be used to implement the principles disclosed herein.
[0095] All U.S. patents mentioned in this specification are hereby incorporated by reference in their entirety. If it is necessary to adopt the concepts of various patents and applications to provide additional embodiments, various aspects of the embodiments may be modified. These and other changes may be made to the embodiments according to the above detailed description. Generally, in the following claims, the terms used should not be interpreted as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be interpreted as including all possible embodiments and the full range of equivalents to which such claims are authorized.
Claims
1. A workpiece inspection and defect detection system comprising: light source; a lens that inputs image light caused by the surface of the workpiece illuminated by the light source and transmits the image light along an imaging light path; a camera that receives imaging light transmitted along the imaging light path and provides an image of the workpiece; one or more processors; as well as a memory coupled to the one or more processors and storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least: acquiring a plurality of training images, wherein at least some of the plurality of training images are used to train a defect detection portion; determining a plurality of training image features from the plurality of training images; determining a plurality of anomaly detector classification characteristics based at least in part on the determined training image features; Using the camera to acquire a first operating mode image of a workpiece; determining a plurality of features from a first operating mode image of the workpiece; as well as utilizing the anomaly detector classification characteristic to determine whether the first operating mode image of the workpiece is to be classified as anomalous and therefore unsuitable for a defect detection process based at least in part on features determined from the first operating mode image of the workpiece; utilizing the anomaly detector classification characteristic to determine that a second operating mode image of the workpiece is to be classified as non-anomalous and therefore suitable for a defect detection process; and The second operation mode image is analyzed using the defect detection portion to determine whether the second operation mode image of the workpiece is to be classified as a defect image including a defect of the workpiece.
2. The system of claim 1, wherein the first operating mode image of the workpiece is classified as abnormal and therefore not analyzed by the defect detection portion in accordance with the utilization of the anomaly detector classification characteristic.
3. The system of claim 2, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: A message indicating that at least one of an imaging condition, a lighting condition, or a workpiece condition is problematic is generated based at least in part on classifying the first operational mode image of the workpiece as abnormal.
4. The system of claim 2, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: A reason why the first operating mode image of the workpiece was classified as abnormal is determined.
5. The system of claim 4, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: A message is generated indicating the reason why the first operation mode image of the workpiece is classified as abnormal.
6. The system of claim 4, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: After determining the reason why the first operating mode image is classified as abnormal, at least one of the brightness of the light source, or the position of the light source, or the position of the lens or lens element relative to at least one of the camera or workpiece is adjusted.
7. A method of operating a workpiece inspection and defect detection system, the method comprising: acquiring a plurality of training images using a light source, a lens, and a camera of the workpiece inspection and defect detection system, wherein at least some of the plurality of training images are used to train a defect detection portion; determining a plurality of training image features from the training image; determining a plurality of anomaly detector classification characteristics based at least in part on the training image features; Acquire a first operation mode image of the workpiece; determining a plurality of features from a first operating mode image of the workpiece; as well as utilizing the anomaly detector classification characteristic to determine whether the first operating mode image of the workpiece is to be classified as anomalous and therefore unsuitable for a defect detection process based at least in part on features determined from the first operating mode image of the workpiece; utilizing the anomaly detector classification characteristic to determine that a second operating mode image of the workpiece is to be classified as non-anomalous and therefore suitable for a defect detection process; and The second operation mode image is analyzed using the defect detection portion to determine whether the second operation mode image of the workpiece is to be classified as a defect image including a defect of the workpiece.
8. The method of claim 7, wherein the first operating mode image of the workpiece is classified as anomalous and therefore not analyzed by the defect detection portion in accordance with utilizing the anomaly detector classification characteristic.
9. The method according to claim 8, wherein: The method also includes generating a message indicating a problem with an imaging condition, a lighting condition, or a workpiece condition based at least in part on classifying the first operating mode image of the workpiece as abnormal.
10. The method according to claim 8, wherein: The method also includes determining a reason why the first operating mode image was classified as abnormal.
11. The method according to claim 10, further comprising: A message is generated indicating the reason why the first operation mode image of the workpiece is classified as abnormal.
12. The method according to claim 10, further comprising: After determining the reason why the first operating mode image is classified as abnormal, at least one of the brightness of the light source, or the position of the light source, or the position of the lens or lens element relative to at least one of the camera or workpiece is adjusted.
13. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to at least: acquiring a plurality of training images using a light source, a lens, and a camera of a workpiece inspection and defect detection system, wherein at least some of the plurality of training images are used to train a defect detection portion; determining a plurality of training image features from the plurality of training images; determining a plurality of anomaly detector classification characteristics based at least in part on the determined training image features; Using the camera to acquire a first operating mode image of a workpiece; determining a plurality of features from a first operating mode image of the workpiece; as well as utilizing the anomaly detector classification characteristic to determine whether the first operating mode image of the workpiece is to be classified as anomalous and therefore unsuitable for a defect detection process based at least in part on features determined from the first operating mode image of the workpiece; utilizing the anomaly detector classification characteristic to determine that a second operating mode image of the workpiece is to be classified as non-anomalous and therefore suitable for a defect detection process; and The second operation mode image is analyzed using the defect detection portion to determine whether the second operation mode image is a defect image including a defect of the workpiece.
14. The non-transitory computer readable medium of claim 13, wherein the first operational mode image of the workpiece is classified as anomalous and thus not analyzed by the defect detection portion pursuant to utilizing the anomaly detector classification characteristic.
15. The non-transitory computer readable medium of claim 14, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: A message indicating that at least one of an imaging condition, a lighting condition, or a workpiece condition is problematic is generated based at least in part on classifying the first operational mode image of the workpiece as abnormal.
16. The non-transitory computer readable medium of claim 14, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: determining a reason why the first operating mode image of the workpiece was classified as abnormal; and A message is generated indicating the reason why the first operation mode image of the workpiece is classified as abnormal.
17. The non-transitory computer readable medium of claim 16, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: After determining the reason why the first operating mode image of the workpiece was classified as abnormal, at least one of the brightness of the light source, or the position of the light source, or the position of the lens or lens element relative to at least one of the camera or the workpiece is adjusted.
18. A method of operating a workpiece inspection and defect detection system, the method comprising: acquiring a plurality of training images using a light source, a lens, and a camera of the workpiece inspection and defect detection system, wherein at least some of the plurality of training images are used to train a defect detection portion; determining a plurality of training image features from the training image; Determining a plurality of anomaly detector classification features based at least in part on the training image features, wherein determining the plurality of anomaly detector classification features based at least in part on the training image features comprises: creating a synthetic dataset in which a plurality of said training image features are altered to simulate anomalous images but are not altered to simulate non-anomalous images; and training a two-class classifier capable of distinguishing between anomalous images and non-anomalous images in the synthetic dataset, wherein the anomalous images are not considered acceptable for a defect detection process and the non-anomalous images are considered acceptable for a fault detection process; Acquire a first operation mode image of the workpiece; determining a plurality of features from a first operational mode image of the workpiece; and Based at least in part on features determined from the first operating mode image of the workpiece, an anomaly detector classification characteristic is utilized to determine whether the first operating mode image of the workpiece is classified as an anomaly and therefore unsuitable for a defect detection process.
19. The method of claim 18, wherein: classifying the first operating mode image of the workpiece as abnormal based on utilizing the classification characteristics of the anomaly detector; and The method also includes generating a message indicating a problem with an imaging condition, a lighting condition, or a workpiece condition based at least in part on classifying the first operating mode image of the workpiece as abnormal.
20. The method of claim 18, wherein: classifying the first operating mode image of the workpiece as abnormal based on utilizing the classification characteristics of the anomaly detector; and The method further includes: determining a reason why the first operating mode image is classified as abnormal; and A message is generated indicating why the first operation mode image of the workpiece is classified as abnormal.
21. The method of claim 18, wherein: classifying the first operating mode image of the workpiece as abnormal based on utilizing the classification characteristics of the anomaly detector; and The method further includes: determining a reason why the first operating mode image is classified as abnormal; and Based at least in part on the determined reason why the first operating mode image is classified as abnormal, at least one of the brightness of the light source or the position of the light source, or the position of the lens or lens element relative to at least one of the camera or the workpiece is adjusted.
