Workpiece inspection and defect detection system showing number of defect images for training

Abstract

To provide a precision machine vision system for inspecting a workpiece to detect a defect.SOLUTION: A workpiece inspection and a defect detection system 100 includes: light sources 220, 230, 240, and 300; lenses 286 and 288 for inputting image light emitted from a surface of a workpiece 20; and a camera 260 for receiving imaging light transmitted along an imaging optical path. The system uses images of the workpiece acquired by the camera as training images, trains a defect detection section 140dp so as to detect a defect image including a defective workpiece, and determines performance accuracy of the defect detection section trained by the training images. The system provides an instruction on whether to provide an additional defect image for training on the basis of the performance accuracy of the defect detection section. After the training is completed, the system acquires new images of the workpiece using the camera, analyzes the new images, and determines the defect image including the defective workpiece.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to workpiece inspection systems, and more particularly to precision machine vision systems for inspecting workpieces and detecting defects. [Background technology]

[0002] Precision non-contact workpiece inspection systems, such as machine vision inspection systems (or simply "vision systems"), can be used to acquire images of workpieces for inspection. Such systems can be used for various types of applications, such as general workpiece inspection, metrology applications for determining precise dimensional measurements of the workpiece, and the like. Such systems typically include a computer, a camera, and an optical system. Some configurations may include a motion mechanism (e.g., a precision stage, conveyor, etc.) that moves to enable traversal and inspection of the workpiece. One exemplary prior art machine vision inspection system is the QUICK VISION® series PC-based vision system and QVPAK® software available from Mitutoyo America Corporation (MAC), located in Aurora, Illinois. The functionality and operation of the QUICK VISION® series vision system and QVPAK® software are generally described, for example, in the QVPAK 3D CNC Vision Measuring Machine User's Guide, published in January 2003. This type of system utilizes a microscope-type optical system and a moving stage to provide inspection images of the workpiece. Summary of the Invention [Problem to be solved by the invention]

[0003] Such machine vision inspection systems typically face various types of challenges in inspecting workpieces (e.g., due to variations in the type or surface of the workpiece being inspected, changes in inspection conditions, etc.) Systems that can provide improvements to such challenges are desirable for some types of inspection operations (e.g., for workpiece defect detection and / or to improve the accuracy of defect detection).

[0004] This summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. [Means for solving the problem]

[0005] A workpiece inspection and defect detection system is provided that includes a light source, a lens, a camera, one or more processors, and a memory. The lens inputs image light emanating from a surface of the workpiece illuminated by the light source and transmits the image light along an imaging optical path. The camera receives the imaging light transmitted along the imaging optical path and provides an image of the workpiece.

[0006] The memory is coupled to the one or more processors and stores program instructions that, when executed by the one or more processors, cause the one or more processors to at least: train a defect detector to detect defect images including defective workpieces using images of the workpieces acquired by the camera, including both defect images and non-defect images, as training images; determine a performance accuracy of the defect detector trained with the training images; provide an indication of whether to provide additional defect images for use as training images for training the defect detector based at least in part on the performance accuracy of the defect detector trained with the training images; and, if the additional images are not used as training images for training the defect detector, acquire a plurality of new images of the workpieces using the camera during the run mode and analyze the new images using the defect detector to determine the defect images including the defective workpieces.

[0007] In various embodiments, the provided instructions include a message indicating that additional defect images should be provided, and the defect detector is trained with the provided additional defect images in response. The performance accuracy of the defect detector trained with the additional defect images is determined, and an instruction is provided as to whether to provide additional defect images to use as training images for training the defect detector based at least in part on the performance accuracy of the defect detector trained with the additional defect images. In various embodiments, the performance of the defect detector trained with the additional defect images may correspond to a portion of a performance accuracy curve having a smaller slope than a portion of the performance curve corresponding to the performance of the defect detector before training the defect detector with the additional defect images. Also, if no additional images are used to train the defect detector, the slope of the portion of the performance accuracy curve corresponding to the performance of the defect detector trained with the current number of defect images may be approximately flat relative to an earlier portion of the performance curve.

[0008] In various embodiments, the determination of the performance accuracy of the defect detector may be based at least in part on a determination of a number of defective pixels or defective images that are correctly classified as defective pixels or defective images by the defect detector. In various embodiments, the determination of the performance accuracy of the defect detector may additionally or alternatively be based at least in part on a determination of a number of non-defective pixels or non-defective images that are correctly classified as non-defective pixels or non-defective images by the defect detector.

[0009] In various embodiments, one or more new images of the workpiece acquired during the execution mode can be used to perform one or more metrology operations (e.g., to measure the dimensions of a defect on the workpiece).

[0010] In various embodiments, the training images may include a first plurality of training images and a second plurality of training images. Determining the performance accuracy of the defect detector may include: testing the defect detector trained with the first plurality of training images to determine a first accuracy performance metric using a first plurality of test or validation images of the workpiece acquired by the camera that are not included in the first plurality of training images; testing the defect detector trained with the first and second plurality of training images to determine a second performance metric using a second plurality of test or validation images acquired by the camera that are not included in either the first or second plurality of training images; and determining a recommended number of defect images for training the defect detector based at least in part on the first and second performance criteria. In various embodiments, providing the instructions may include generating a message including a recommended number of defect images for training the defect detector. The recommended number of defect images indicates whether additional defect images should be provided relative to the current number of defect images already used for training.

[0011] In various embodiments, the defect detector may be trained using augmented data (e.g., in addition to other training images). Based at least in part on the performance accuracy of the defect detector trained with and without the augmented data, at least one of whether to provide additional defect images to use as training images for training the defect detector or a recommended number of defect images for training the defect detector may be determined (e.g., the recommended number of defect images may indicate whether to provide additional defect images relative to the current number of defect images already used for training).

[0012] In various embodiments, a method may be provided that is executed by a workpiece inspection and defect detection system (e.g., a computer-implemented method operating under the control of one or more computing systems configured with executable instructions) including at least a light source, a lens, a camera, and a defect detection component, and includes: training the defect detection component to detect defect images including defective workpieces using images of the workpiece acquired by the camera, including both defect images and non-defect images, as training images; determining a performance accuracy of the defect detection component trained with the training images; providing an indication of whether to provide additional defect images for use as training images for training the defect detection component based at least in part on the performance accuracy of the defect detection component trained with the training images; and if the additional images are not used as training images for training the defect detection component, acquiring a plurality of new images of the workpiece using the camera during a run mode, and analyzing the new images using the defect detection component to determine the defect images including the defective workpiece.

[0013] In various embodiments, a non-transitory computer-readable medium may be provided that stores program instructions that, when executed by one or more processors, cause the one or more processors to at least: train a defect detector to detect defect images that include defective workpieces using images of the workpiece acquired by a camera of the workpiece inspection and defect detection system, including both defect images and non-defect images, as training images; determine a performance accuracy of the defect detector trained with the training images; provide an indication of whether to provide additional defect images for use as training images for training the defect detector based at least in part on the performance accuracy of the defect detector trained with the training images; and, if the additional images are not used as training images for training the defect detector, acquire a plurality of new images of the workpiece using the camera during a run mode and analyze the new images using the defect detector to determine the defect images that include the defective workpieces. [Brief explanation of the drawings]

[0014] The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

[0015] [Figure 1] FIG. 1 illustrates various typical components of a general-purpose precision machine vision inspection system. [Figure 2] 2 is a block diagram similar to that of FIG. 1 of a control system portion and vision component portion of a machine vision inspection system including certain features disclosed in the present invention. [Figure 3A] 2 is an example of an image of a defect-free workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 3B] 2 is an example of an image of a defect-free workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 3C]2 is an example of an image of a defect-free workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 3D] 2 is an example of an image of a defect-free workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 3E] 2 is an example of an image of a defect-free workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 3F] 2 is an example of an image of a defect-free workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 4A] 2 is an example of an image of a workpiece portion containing a defect that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 4B] 2 is an example of an image of a workpiece portion containing a defect that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 4C] 2 is an example of an image of a workpiece portion containing a defect that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 4D] 2 is an example of an image of a workpiece portion containing a defect that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 4E] 2 is an example of an image of a workpiece portion containing a defect that may be acquired using a machine vision inspection system similar to that of FIG. 1. [Figure 5A] 5B through 5D together are defect-free examples showing images of a workpiece portion that may be acquired using a machine vision inspection system similar to that of FIG. [Figure 5B] 5A, 5C, and 5D together show examples of images of workpiece portions with defects that may be acquired using a machine vision inspection system similar to that of FIG. [Figure 5C]5A, 5B, and 5D together illustrate examples of defective workpiece portion images that may be acquired using a machine vision inspection system similar to that of FIG. [Figure 5D] 5A-5C together show examples of images of workpiece portions with defects that may be acquired using a machine vision inspection system similar to that of FIG. [Figure 6] FIG. 1 illustrates the use of one or more video tools to perform metrology operations on an image of a workpiece containing a defect. [Figure 7] 10 is a graph showing an example of a performance curve of a defect detection unit. [Figure 8A] 2 shows examples of messages that may be generated by a machine vision inspection system similar to that of FIG. 1. [Figure 8B] 2 shows examples of messages that may be generated by a machine vision inspection system similar to that of FIG. 1. [Figure 9] 2 shows a flow diagram of a method for operating a machine vision inspection system similar to that of FIG. 1; [Figure 10] 2 shows a flow diagram of a method for operating a machine vision inspection system similar to that of FIG. 1; [Figure 11] 2 shows a flow diagram of a method for operating a machine vision inspection system similar to that of FIG. 1; [Figure 12] 2 shows a flow diagram of a method for operating a machine vision inspection system similar to that of FIG. 1; [Figure 13] 10 is a graph showing an example of a performance curve of a defect detection unit. [Figure 14] 2 shows a flow diagram of a method for operating a machine vision inspection system similar to that of FIG. 1; [Figure 15] 2 shows a flow diagram of a method for operating a machine vision inspection system similar to that of FIG. 1; DETAILED DESCRIPTION OF THE INVENTION

[0016] 1 is a block diagram of an exemplary machine vision inspection system 10 that can be used as an imaging system in accordance with the methods described herein. The machine vision inspection system 10 includes a vision inspection machine 12 operably connected to exchange data and control signals with a control computer system 14. The control computer system 14 is further operably connected to exchange data and control signals with a monitor or display 16, a printer 18, a joystick 22, a keyboard 24, and a mouse 26. The monitor or display 16 can display a user interface suitable for controlling and / or programming the operation of the machine vision inspection system 10. It will be appreciated that in various exemplary embodiments, the functionality of any or all of elements 14, 16, 22, 24, and 26 can be substituted for and / or provided redundantly by a touchscreen tablet and / or similar device, or the like.

[0017] Those skilled in the art will recognize that the control computer system 14 and / or other control systems described herein generally may be implemented using any suitable computing system or device, including distributed or networked computing environments, etc. Such computing systems or devices may include one or more general-purpose or special-purpose processors (e.g., non-custom or custom devices) that execute software to perform the functions described herein. The software may be stored in memory, such as random access memory (RAM), read-only memory (ROM), flash memory, or a combination of such components. The software may also be stored in one or more storage devices, such as optical disks, flash memory devices, or any other type of non-volatile storage medium for storing data. The software may include one or more program modules that include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In a distributed computing environment, the functionality of the program modules may be combined or distributed across multiple computing systems or devices, in either wired or wireless configurations, and accessed via service calls.

[0018] The vision inspection machine 12 includes a movable workpiece stage 32 and an optical imaging system 34, which may include a zoom lens or interchangeable lenses that generally provide different magnifications for the images acquired by the optical imaging system 34.

[0019] 2 is a block diagram of a control system portion 120 and a vision components portion 200 of a machine vision inspection system 100 similar to the machine vision inspection system of FIG. 1 and including several features described herein. The control system portion 120 is used to control the vision components portion 200, as described in more detail below. The control system portion 120 can be configured to exchange data and control signals with the vision components portion 200. The vision components portion 200 includes an optical assembly portion 205, light sources 220, 230, 240, 300, and a workpiece stage 210 having a central transparent portion 212. The workpiece stage 210 can be controllably moved along x- and y-axes that lie in a plane generally parallel to the surface of the stage on which a workpiece 20 can be positioned.

[0020] The optical assembly 205 includes a camera system 260 and an interchangeable objective lens 250. In some embodiments, the optical assembly 205 may optionally include a variable focal length (VFL) lens, such as, for example, a tunable acoustic gradient (TAG) lens.

[0021] In various exemplary embodiments, optical assembly portion 205 may further include turret lens assembly 280 having lenses 286 and 288. Instead of a turret lens assembly, various exemplary embodiments may include fixed or manually interchangeable magnification-altering lenses, zoom lens configurations, etc. In various exemplary embodiments, interchangeable objective lens 250 may be selected from a set of fixed magnification objective lenses (e.g., a set of objective lenses corresponding to magnifications of 0.5x, 1x, 2x or 2.5x, 5x, 10x, 20x or 25x, 50x, 100x, etc.) included as part of the variable magnification lens portion.

[0022] The optical assembly portion 205 can be controllably moved along a z-axis, which is generally orthogonal to the x- and y-axes, using a controllable motor 294. The controllable motor 294 drives an actuator to move the optical assembly portion 205 along the z-axis to change the focus of an image of the workpiece 20. The controllable motor 294 is connected to the input / output interface 130 via a signal line 296 to change the focus of the image over a specific range. The workpiece 20 can be positioned on a workpiece stage 210. The workpiece stage 210 is controlled to move relative to the optical assembly portion 205 such that the field of view of the interchangeable objective lens 250 can be moved between multiple positions on the workpiece 20 and / or between multiple workpieces 20.

[0023] One or more of the transmitted illumination light source 220, the epi-illumination light source 230, and the oblique illumination light source 240 (e.g., a ring illumination) can emit source light 222, 232, and / or 242, respectively, to illuminate one or more workpieces 20. For example, during an image exposure, the epi-illumination light source 230 can emit source light 232 along a path that includes a beam splitter 290 (e.g., a partial mirror). The source light 232 is reflected or transmitted as workpiece image light 255, which is used for imaging and passes through an interchangeable objective lens 250 and a turret lens assembly 280 before being collected by a camera system 260. The workpiece image exposure, which includes an image of one or more workpieces 20, is captured by the camera system 260 and output on a signal line 262 to the control system portion 120.

[0024] The various light sources (e.g., light sources 220, 230, 240, 300) can be connected to the lighting control interface 133 of the control system section 120 via associated signal lines (e.g., buses 221, 231, 241, 331, respectively). The control system section 120 can control the selection of one of the turret lenses by rotating the turret lens assembly 280 along axis 284 via signal line or bus 281 to change the magnification of the image.

[0025] 2, in various exemplary embodiments, control system section 120 includes a controller 125, an input / output interface 130, a memory 140, a workpiece program generator and executor 170, and a power supply section 190. Each of these components, and additional components described below, may be interconnected by one or more data / control buses and / or application programming interfaces, or by direct connections between the various elements. Input / output interface 130 includes an imaging control interface 131, a motion control interface 132, a lighting control interface 133, and a lens control interface 134.

[0026] Illumination control interface 133 can include illumination control elements 133a-133n, which control, for example, the selection, power, and on / off switching of various corresponding light sources in machine vision inspection system 100. Illumination control interface 133 also includes illumination control element 133sip, which, in the illustrated embodiment, operates in conjunction with structured illumination pattern (SIP) generator 300 to provide structured illumination during image acquisition. In various examples, a projection pattern can be output from SIP generator 300 and input to beam splitter 290, which is directed as epi-illumination light through objective lens 250 to provide SIP structured light 232′ that illuminates the field of view.

[0027] Memory 140 may include an image file memory portion 141, a defect detection portion 140dp, a workpiece program memory portion 142 that may include one or more part programs, etc., and a video tool portion 143. Video tool portion 143 includes video tool portion 143a and other video tool portions (e.g., 143n) that determine the GUI, image processing operations, etc. for each corresponding video tool, as well as a region of interest (ROI) generator 143roi. Region of interest generator 143roi supports automatic, semi-automatic, and / or manual operation of defining various ROIs operable for the various video tools included within video tool portion 143.

[0028] The video tools section 143 also includes an autofocus video tool 143af that determines GUIs for focus height measurement operations, image processing operations, and the like. In various exemplary embodiments, the autofocus video tool 143af can also include a fast-focus height tool that can be utilized to measure focus heights quickly using hardware. In various exemplary embodiments, the fast-focus height tool can be a special mode of the autofocus video tool 143af that otherwise operates according to conventional methods for autofocus video tools, or the operation of the autofocus video tool 143af can include only the operation of the fast-focus height tool. Fast autofocus and / or focus position determination for one or more image regions of interest can be based on analyzing the image to determine corresponding quantitative contrast metrics of various regions according to known methods.

[0029] In the context of this disclosure, as known to those skilled in the art, the term "video tool" generally refers to a relatively complex automated or programmed set of operations that can be performed by a machine vision user through a relatively simple user interface. For example, a video tool may include a complex set of pre-programmed image processing operations and calculations that can be adapted and customized for a particular instance by adjusting a small number of variables or parameters that define these operations and calculations. In addition to the underlying operations and calculations, a video tool also includes a user interface that allows a user to adjust those parameters for a particular instance of the video tool. It should be noted that in some cases, visible user interface features are referred to as video tools, and the underlying operations are implicitly included.

[0030] One or more display devices 136 (e.g., display 16 of FIG. 1 ) and one or more input devices 138 (e.g., joystick 22, keyboard 24, and mouse 26 of FIG. 1 ) may be connected to input / output interface 130. Display device 136 and input device 138 may be used to display a user interface, which may include various graphical user interface (GUI) functions that may be used to perform inspection operations and / or create and / or modify part programs, view images captured by camera system 260, and / or directly control vision components portion 200.

[0031] In various illustrative embodiments, when a user uses machine vision inspection system 100 to generate a part program for workpiece 20, the user generates part program instructions by operating machine vision inspection system 100 in learn mode and providing a desired image acquisition training sequence. For example, the training sequence may include placing specific workpiece features of a representative workpiece within the field of view (FOV), setting illumination levels, focusing or autofocusing, acquiring images, and providing an inspection training sequence to be applied to the images (e.g., using one instance of a video tool on the workpiece features). Learn mode operates such that this sequence or sequences are captured or recorded and converted into corresponding part program instructions. When the part program is executed, these instructions cause the machine vision inspection system to replicate the trained image acquisition and perform inspection operations to automatically inspect specific workpiece features (i.e., corresponding features at corresponding locations) on one or more workpieces in run mode that match the representative workpieces used to generate the part program. In various exemplary embodiments, several types of training modes may additionally or alternatively be utilized (e.g., a training mode to train a defect detector to detect defects, a training mode to train an anomaly detector to detect anomalous images that are not acceptable to the defect detection process, etc.).

[0032] The video tools portion 143 also includes a Z-height measurement tools portion 143z that provides various operations and functions related to Z-height measurement operations. In one embodiment, the Z-height measurement tools portion 143z can include a Z-height tool 143zt. The Z-height tools 143zt can include, for example, an autofocus tool 143af and a multipoint autofocus video tool 143maf. The Z-height tools 143zt can manage some aspects of image stack acquisition and related structured light pattern generation operations in conjunction with a Z-height tool configured in a mode for determining a best-focus height and / or Z-height measurements. In general, the Z-height measurement tools portion 143z can perform at least some operations similar to known Z-height measurement tools, such as operating in a learn mode and / or run mode or other mode to generate all or part of a focus curve and finding its peak as the best-focus position.

[0033] The defect detector 140dp performs various defect detection operations, as described in more detail below. In various embodiments, the defect detector 140dp utilizes models that require training data. In various exemplary embodiments, these models can be supervised (e.g., artificial intelligence (AI) models). The defect detector 140dp processes image data corresponding to user-labeled defect images to train a classification model, which in various embodiments is an AI classification model. The number of images required to achieve accurate model predictions for test images can vary widely (e.g., from about 25 to 300), depending on factors such as the complexity, variability, and visual distinctiveness of the defects and background. The defect detector 140dp analyzes the set of labeled defect images provided by the user and provides instructions to the user on whether to provide additional defect images for training. As an example, in various embodiments, the user can be provided with a recommendation (e.g., a yes or no recommendation) regarding whether additional images would improve performance. As another example, an estimate of the number of defect images required to train a classification model to perform highly accurate classification may be provided. As disclosed herein, such determinations and recommendations are specific to the images provided by the user, the defect characteristics, and the particular model being used, rather than a general recommendation of a predetermined number of images (e.g., 100 images). Such guidance may enable a user to provide a sufficient number of defect images to achieve a model with highly accurate classification capabilities, without burdening the user with providing additional images when the additional images are expected to provide little or no improvement in model accuracy.

[0034] In various exemplary embodiments, the defect detector 140dp can perform a defect detection process that is performed in conjunction with a metrology process. In various embodiments, it may be desirable to include the defect detector 140dp in a machine vision inspection system similar to the machine vision inspection system 100 of FIG. 1 configured to perform a metrology process because the system can generate image data that is input into the defect detection process performed by the defect detector 140dp. Thus, a single machine can be configured to perform both the metrology process and the defect detection process, providing advantages over conventional metrology systems. For example, if a defect is detected in a workpiece during a defect detection process, there is no reason to perform a metrology process on the workpiece, thereby saving time. More specifically, if a defect is detected in a workpiece during a defect detection process, it may not be necessary to measure the obviously defective portion. Therefore, it may be advantageous to perform the defect detection process before the start of the metrology process.

[0035] Additionally, some defects may require further measurement or inspection to determine additional defect parameters. For example, a 2D image allows for quick recognition of a potential defect and the defect's XY location and approximate XY area. If the 3D characteristics of the potential defect are important, the defect detector 140dp can perform additional processing (e.g., metrology operations) to determine whether the potential defect is an actual defect. For example, if a scratch on the surface of the workpiece must be deeper than a certain threshold to be considered a defect, the defect detector 140dp can acquire a more time-consuming 3D point cloud of the affected area (e.g., using the z-height measurement tool 143z) to determine whether the scratch is deep enough to reject the part. In various embodiments, various actions can be performed (e.g., can be programmed to be performed automatically) as a result of the initial defect classification, such as: For example, (1) continue with the standard metrology process, (2) stop or pause the defect detection process and perform a metrology process that includes more informative measures of potential defects (e.g., 3D measurements, measurements with different lighting, touch probe measurements, such as surface roughness), (3) dispose of the workpiece (e.g., scrap or recycle the workpiece), (4) send the workpiece for additional manual inspection, or (5) provide feedback to the production line indicating that there is something wrong with the machine, process, etc.

[0036] In various exemplary embodiments, the defect detection portion 140dp of the memory 140 stores model data and program instructions for various tools and algorithms of the defect detection system that can be used to infer whether various types of defects are present in an image of a workpiece (e.g., to indicate defects in the workpiece surface contained within the image). While the machine vision inspection system is operating in a training or learn mode, the defect detection system uses a training image set including defective and non-defective images to train the defect detection portion 140dp. The training image set is captured using specific (e.g., current) imaging, lighting, and workpiece conditions. After the defect detection portion 140dp is initially trained, it is operated in a run mode to infer whether new, unseen workpiece images contain defects and classify each workpiece image as defective or non-defective accordingly.

[0037] More specifically, in various embodiments, the defect detection unit 140dp can utilize a model (e.g., a machine learning model, an artificial intelligence (AI) model, etc.) trained to recognize defects in images, which often requires many example defect images. The number of training images depicting defects required to achieve good results varies (e.g., in a manner that is difficult to explain to a user) depending on the particular workpiece being inspected and the type of defects in the workpiece. Because the number of training images required for good results is unknown, some defect detection software recommends providing a predetermined number (e.g., 100) of defect images, with additional defect images being provided if available. Depending on the defects and the context of the workpiece, fewer (e.g., 50) defect images than the predetermined number (e.g., 100) may be required to achieve good results, potentially wasting valuable user time by providing and marking more defect images than necessary. For more visually challenging defects and workpiece backgrounds, a larger number (e.g., 200) of defect images than a given number (e.g., 100) may be required to generate optimal model performance (e.g., few false negative and false positive detections). In this case, once a user has provided a commonly recommended default number of defect images (e.g., 100 defect images), they must go to the trouble of collecting and marking the images, further training the model, and assessing performance to see if providing additional defect images could potentially improve performance. This wastes time and effort and may provide little or no benefit to model performance. While providing more defect training images can sometimes improve model performance, at some number of images specific to each defect / background dataset, the expected performance improvement may be nominally negligible. Therefore, it may be desirable to provide dataset-specific guidance regarding the point at which increasing the number of images no longer results in significant performance improvement.To this end, the defect detection unit 140dp is configured to automatically assess the performance of the model and provide an indication of whether additional defect images should be provided for training (e.g., as to whether training with additional defect images may result in significant performance improvement or whether such improvement may be nominally negligible).

[0038] As described in more detail below, FIGS. 3A-3F, 4A-4E, and 5A-5D show some example workpiece images that may be acquired by machine vision inspection system 100, which in some embodiments may be used (e.g., as training images and / or test images) to train defect detector 140dp during training mode (and / or some of these images may be example workpiece images later acquired by machine vision inspection system 100 during run mode and analyzed by trained defect detector 140dp). Specifically, the example workpiece images in FIGS. 3A-3F and 4A-4E are of various sections (i.e., various XY locations) on a machined aluminum plate. As described in more detail below, FIGS. 3A-3F show example non-defect images, and FIGS. 4A-4E show example defect images (e.g., a training image set typically includes a number of both defective and non-defective workpiece images). The differences between these images help explain one reason why it is desirable to use a large number of training images to train defect detector 140dp. More specifically, because different features may be observed in different workpiece images acquired to inspect a certain type of workpiece (e.g., as shown by the differences between the workpiece images in FIGS. 3A-3F and 4A-4E for inspecting a certain type of machined aluminum plate), improving the accuracy of defect detection by defect detector 140dp can be achieved by utilizing a wide variety of workpiece images for training that are similar to and / or enable superior defect detection in a wide variety of images that may be acquired later in run mode. For example, particularly with respect to the examples of Figures 3A to 3F, 4A to 4E, and 5A to 5D, such training can help the defect detection unit 140dp distinguish between defects (which, for example, in the illustrated example, may include a variety of different types of scratch defects, etc.) and elements of a normal workpiece surface (which, for example, in the illustrated example, may include a variety of different types of machining marks formed on the plate surface, which typically appear as a hashed texture that varies across the surface, etc.).

[0039] As noted above, Figures 3A-3F and 4A-4E are example workpiece images of a section of a workpiece that is a relatively "flat" machined aluminum plate. Each image is taken from the same angular viewpoint (e.g., directly above at a 90-degree angle relative to the plate), but at a different XY location on the plate. Each image shows an approximately 2.5 mm by 1.9 mm (XY) view of the plate surface. Thus, each image shows a magnified view of a portion of the plate surface. Machining marks formed on the plate surface typically produce a hash texture that varies across the surface. In these images, at such a magnified scale, the relatively flat plate may not appear flat. In this example, the height of some of the hash machining mark protrusions may be approximately 5 micrometers or less.

[0040] The sections of the machined aluminum plate shown in Figures 3A through 3F do not contain defects. In other words, Figures 3A through 3F show example images of sections of a machined aluminum plate that are "defect-free." In contrast, Figures 4A through 4E show example images of sections of a workpiece that contain defects. The images shown in Figures 4A through 4E are similar to those shown in Figures 3A through 3F, except that the images in Figures 4A through 4E include a defect 402 formed on the surface of the machined aluminum plate. In these examples, the defect 402 is a scratch formed on the surface of the machined aluminum plate. More specifically, Figure 4A shows scratch defects 402A1 and 402A2, Figure 4B shows scratch defects 402B1 and 402B2, Figure 4C shows scratch defects 402C1 and 402C2, Figure 4D shows scratch defect 402D, and Figure 4E shows scratch defects 402E1 and 402E2. As noted above, the wide variety of defect and workpiece surface features shown in Figures 3A to 3F and Figures 4A to 4E helps explain one reason why it is desirable to use a large number of training images to train the defect detection unit 140dp (e.g., a training image set typically includes a large number of defective and non-defective workpiece images).

[0041] 5A through 5D are example images of similar portions of a workpiece with and without defects that may be acquired using a machine vision inspection system similar to that of FIG. 1. FIG. 5A shows an example section of a plate that may be classified as "non-defective." FIGS. 5B through 5D show example images of a section of a plate that may be classified as "defective," each image including a scratch defect 502 (e.g., including each of scratch defects 502B, 502C, and 502D). In the examples of FIGS. 5A through 5D, the images are of similar portions of the workpiece (e.g., the types of machining marks formed on the surfaces of these portions are similar or nominally identical in each image, with the primary difference between these images being the characteristics of each of scratch defects 502B, 502C, and 502D). In one example embodiment, the images of FIGS. 5A through 5C may be included as part of a training image set for training defect detector 140dp.

[0042] In one exemplary embodiment, the image of FIG. 5D may be an example of a run-mode image that may be analyzed by defect detector 140dp to determine whether it should be classified as a defect image or a non-defect image. In various embodiments, defect detector 140dp may be sufficiently trained to properly classify the image of FIG. 5D as a defect image (e.g., trained with the training images of FIGS. 5A-5C, which include scratch defects 502B and 502C that include similar workpiece portions and have several characteristics similar to scratch defect 502D). In various embodiments, several additional processes may be performed on exemplary scratch defect 502D. For example, one or more metrology processes may be performed in conjunction with the defect detection process to determine various dimensions or other characteristics of exemplary scratch defect 502D, as described in more detail below in connection with FIG. 6.

[0043] FIG. 6 illustrates the use of one or more video tools to perform metrology operations on an image of a workpiece containing a defect (e.g., to determine the defect's dimensions, etc.). As shown, for an image 601 (which may be similar or identical, e.g., to the image of FIG. 5D ) containing a scratch defect 602 (which may be similar or identical, e.g., to scratch defect 502D), a video box tool 606 includes scan lines 608 (which may additionally or alternatively represent a video point tool, etc.) that are used to determine the edge location, dimensions, and / or other aspects of the scratch defect 602. In various exemplary embodiments, the video box tool 606 can be sized, positioned, and rotated until it indicates or defines a region of interest (e.g., the area within the box tool 606). Arrows (representing, e.g., scan lines, point tools, etc.) shown in FIG. 6 can be used to determine one or more edges of the scratch defect 602. In various exemplary embodiments, the video box tool 606 can generally use one or more conventional edge gradients along one or more edges of the defect 602 within the region of interest and can determine one or more edges of the defect 602 based on the local magnitude of one or more edge gradients along various scan lines 608, etc.

[0044] In various exemplary embodiments, such metrology operations may also include performing certain morphological or other filtering (e.g., to distinguish the edge of a scratch from the machining pattern of the workpiece). As shown in FIG. 6 , a box tool 606 is used in conjunction with scan lines 608 in a viewing area contained within the image to determine the edge location (e.g., outer edge or perimeter) of scratch defect 602. Based on such determination, video tools and / or other metrology operations may include determining a dimension D1 of the scratch defect (e.g., corresponding to the length or other dimension of scratch defect 602). In various exemplary embodiments, box tool 606, scan lines 608, and / or other video tools and / or metrology operations may also be utilized to determine other dimensions of scratch defect 602 (e.g., width, depth, etc.). For example, as described above, video tool portion 143 may include Z-height measurement tool portion 143z, and a corresponding video tool or operation may be used to determine the Z-height dimension of the scratch defect (e.g., including determining the depth of the scratch relative to other portions or features of the workpiece surface, etc.).

[0045] As part of the general operation of the defect detector, some detected defects may require further measurement or inspection to determine additional defect parameters. For example, as described above, various types of analysis and / or processing of defect images, including scratch defect 602 (e.g., using the video tools and / or other operations described above), may allow the XY location and approximate XY area and / or other dimensions of defect 602 to be determined. If the 3D characteristics of the potential defect are important (e.g., if the scratch must be deeper than a certain value to be considered a defect), defect detector 140dp may initiate a process to utilize a Z-height measurement tool (e.g., of Z-height measurement tool unit 143z) or other 3D detection process (e.g., to obtain a 3D point cloud of the affected area to determine the scratch depth, etc.).

[0046] As described above, the defect detector 140dp is configured to automatically assess the performance of the model and provide an indication of whether training images containing additional defects would likely improve the model's performance. In various exemplary embodiments, the defect detector 140dp is configured to divide the set of defect images available for training into several groups (e.g., 3-10 groups). In various embodiments, the defect detector 140dp trains the model using increasingly more defect images (e.g., increasing groups) over multiple training cycles, and obtains two performance results after each training cycle (e.g., a performance result using the image set used for training and a performance result using defect images not used for training).

[0047] As a simplified example, the images of Figures 5A-5D can be divided so that the images of Figures 5A and 5B are included with other images as part of a first group, the image of Figure 5C is included with other images as part of a second group, and the image of Figure 5D is included with other images as part of a third group. A first training cycle performed by defect detector 140dp can include using the images of the first group (including the images of Figures 5A and 5B) as training images and using one or both of the images of the second and third groups (including the images of Figures 5C and 5D) as test images. A second training cycle can use the images of the first and second groups (including the images of Figures 5A, 5B, and 5C) as test images and the images of the third group (including the image of Figure 5D) as test images. Additional training cycles can be performed using additional groups of images. This simplified example illustrates part of a process that can be used to determine a recommended number of defect images for training defect detector 140dp. This is explained in more detail below with reference to Figures 7 to 12.

[0048] FIG. 7 is a graph illustrating example performance curves 702 and 704 for a model used by the defect detection unit 140dp. In various embodiments, several typical trends can be observed as a function of the number of defect training images provided to the model. The performance accuracy of the training image set may be high with a small number of training images (e.g., the model memorizes the training image set). As the number of training images provided to the model increases, the performance accuracy of the training image set may decrease slightly and then stabilize. Additionally, the performance accuracy of the test image set relative to the training images may increase as the number of training images provided to the model increases and then stabilize. The performance accuracy of the training image set and the test image set may be very similar at a certain large number of training images. In some embodiments, the performance accuracy using the augmented data and the original training image set may quickly approach the minimum number of images required for training. If the number of defect images used to train the model is such that both the training image set and the test image set provide stable and similar performance, then in various embodiments, the model can be considered to derive no significant performance advantage from adding more defect training images.

[0049] In various exemplary embodiments, to help determine the recommended number of defect images to achieve good model results, defect detector 140dp is configured to fit both the training performance and the test performance to the number of defect images used for training (i.e., shown as curve fits 702 and 704, respectively). As shown in Figure 7, the value on the x-axis where these two curve fits approximately intersect within some tolerance can be defined as the recommended number of defect images to provide for training. Alternatively, defect detector 140dp can be configured to fit only the test image performance and look for a region where the slope approaches nominally zero within some tolerance.

[0050] Additionally, the defect detector 140dp can be configured to train a model using the augmented data to obtain some measure of how much improvement would occur if additional images were provided for training. The algorithm implemented by the defect detector 140dp can make the best decisions by training the model using the augmented data. In various exemplary embodiments, the defect detector 140dp can be configured to train a simple machine learning algorithm to predict the recommended number of training images from these data, rather than using curve fitting and algorithms for different cases.

[0051] A typical workflow for using the above-described model training functionality of the defect detection unit 140dp will now be described. A user imports image data corresponding to multiple images of a part into the defect detection model training process. The user then classifies each image as non-defective or defective. The user then marks pixels corresponding to defects in each defect image (e.g., using a paint tool to mark pixels in a transparent layer overlaid on the original image to produce an image with colored pixels corresponding to matching defect pixels in the original image). The user then selects a model training option that provides recommendations for improving performance. Alternatively, the model training option may be the default option. Finally, the user initiates model training (e.g., presses a button). In various embodiments, all subsequent defect detection model training and analysis functions may be automatic (e.g., no further user input is required).

[0052] The defect detector 140dp then divides the available non-defect and defect images into a number of groups (e.g., three or more) such that the number of defect and non-defect images is similar in each group. For example, if a user provides 80 defect images and 64 non-defect images, the defect detector 140dp generates four groups of images labeled A, B, C, and D. Each group has a unique set of 20 randomly selected defect images and 16 non-defect images.

[0053] Defect detection unit 140dp then trains the model using only the images in group A, obtains a model performance metric (e.g., accuracy) using the images in group A (training image performance for training with 20 defect images), and obtains a model performance metric using the images in group D or using groups B, C, and D together. In either case, this provides a test image performance for training with 20 defect images. In various exemplary embodiments, the performance metric includes the percentage of defect images correctly classified as defect images (e.g., the number of defect images correctly classified as defect images divided by the total number of defect images), the percentage of non-defect images correctly classified as non-defect images (e.g., the number of non-defect images correctly classified as non-defect images divided by the total number of non-defect images), or a combination of the two. Additionally or alternatively, the performance criteria may consider the rate of false positive detections (e.g., the number of non-defective images incorrectly classified as defective images divided by the total number of non-defective images), the rate of false negative detections (e.g., the number of defective images incorrectly classified as non-defective images divided by the total number of defective images), or a combination of the two.

[0054] Defect detection unit 140dp then trains the model using images from groups A and B, then obtains a model performance metric (e.g., accuracy) using images from groups A and B (training image performance for training with 40 defect images), and then obtains a model performance metric using images from group D or using groups C and D together. In either case, this provides a test image performance for training with 40 defect images.

[0055] After this, the defect detection unit 140dp trains the model using images from groups A, B, and C, then obtains a model performance metric (e.g., accuracy) using images from groups A, B, and C (training image performance for training with 60 defect images), and further obtains a model performance metric using images from group D to provide test image performance for training with 60 defect images.

[0056] The defect detection unit 140dp then trains the model using the images of groups A, B, C, and D, and obtains the model accuracy (performance) using the images of groups A, B, C, and D. In other words, it obtains the training image performance of the training using all 80 defect images.

[0057] The defect detector 140dp then fits an appropriate curve to the training image performance versus the number of defect images used for the model training results, so that the fitted training curve is extrapolated to a number of defect images greater than the number used to train the model.

[0058] The defect detector 140dp then fits an appropriate curve to the test image performance versus the number of defect images used to train the model, so that the fitted test curve is extrapolated to a larger number of defect images than were used to train the model.

[0059] The defect detection unit 140dp then determines (e.g., using a convergence algorithm) the number of defect images used to train the model at which the fitted test curve and the fitted training curve converge to one another. The convergence algorithm can use the distance between the curves, the intersection location, and the slope (e.g., approaching zero) as criteria for determining convergence. The convergence algorithm determines the number of defect images used to train the model at which maximum convergence is estimated or predicted. In various embodiments, this value can be determined as the recommended number of defect images (RNDI) for model training. In some alternative embodiments (e.g., as described in more detail below in connection with FIG. 13 ), a convergence algorithm is not used, and other techniques can be used to determine the RNDI and / or other general indications of when a sufficient number of defect images have been provided for training. For example, as shown in FIG. 7 , it can be observed that as the number of defect images used for training increases, the performance curve 704 generally becomes less steep. The RNDI corresponds to the portion where adding additional defect images for training results in a relatively small performance improvement. Following such principles, a sufficient number of defect images for RNDI and / or other training can be determined. This number may correspond, for example, to a portion of a performance curve where the slope tapers off to a particular level where adding additional defect images for training results in a relatively small (e.g., nominally negligible) performance improvement. In various embodiments, following such techniques, specific data corresponding to only one performance curve (e.g., data corresponding to performance curve 704) may be required / used.

[0060] In various embodiments, it may be desirable to have available additional defect images that are not used to train the model in order to reserve some defect images for model validation and testing. Accordingly, in some embodiments, defect detector 140dp may be configured to notify the user to provide a predetermined amount (e.g., 15%) more defect images than the RNDI (recommended number of defect images) for training the model. In other words, the number of defect images to use for training may be recommended by multiplying the determined RNDI by a predetermined amount (e.g., 1.15).

[0061] If the user provides more than the recommended number of defect images, the defect detector 140dp can be configured to generate a message informing the user that providing additional defect images for training is unlikely to result in a significant improvement in model accuracy / performance. If the user provides fewer than the recommended number of defect images, the defect detector 140dp can be configured to generate a message informing the user that they should continue to provide additional defect images for training. In some cases where an RNDI is determined, such a message can indicate how many additional defect images are recommended to provide for training to achieve good model accuracy / performance.

[0062] Thus, the defect detector 140dp can inform the user with some degree of certainty whether providing additional training images is likely to significantly improve model performance. Such a determination is based on model training performance using the specific image set provided by the user, rather than a generic recommendation that may be inaccurate. In some embodiments (e.g., when determining RNDI, etc.), the defect detector 140dp can provide the user with an estimate of how many defect images are desirable in the user's specific image data set to achieve desired performance (e.g., approach the maximum performance that a particular model can provide). In various embodiments, the defect detector 140dp may be unable to reliably estimate a recommended minimum number of defect images required if the minimum number of defect images is significantly greater than the total number of defect images currently provided. In such cases, the defect detector 140dp can provide a message or other indication that the user should provide a predetermined number of additional images. This number of additional images reliably improves the model's defect detection prediction accuracy, allowing for a reliable estimate of how many images are needed to achieve maximum accuracy. Moreover, the defect detector can perform such assessment automatically without user assistance other than providing labeled defect images for analysis.

[0063] 8A and 8B show examples of some types of messages that may be generated and displayed to a user in connection with RNDI. The message shown in FIG. 8A indicates that the number of defect images the user has supplied for the current model is 115, and that the minimum number of defect images recommended for best results is 125. Therefore, the user is notified that the current model should be trained with 10 (i.e., 125 - 115) additional defect images. The user must provide these additional defect images to improve model performance.

[0064] The message shown in FIG. 8B indicates that the user has supplied 40 defect images for the current model, that a reliable recommendation for the minimum number of defect images required for best results cannot be provided, that the user must supply at least 25 additional defect images, and that the model must be maintained in an analysis mode (e.g., training or learning mode) with the 25 additional defect images. The user is therefore informed that additional defect training images could significantly improve the current model's accuracy performance, and is advised not to use the currently trained model for defect detection until the model has been maintained with at least 25 additional defect images. A reliable recommendation for the minimum number of defect images required for best results can then be provided.

[0065] Figure 9 shows a flow diagram of a method 900 for operating a machine vision inspection system similar to that of Figure 1. Method 900 begins at 910, where multiple images of one or more workpieces are acquired. For example, camera system 260 acquires multiple images of workpiece 20. Method 900 then proceeds to 920.

[0066] At 920, each of the images of the workpiece acquired at 910 is classified as non-defective or defective. For example, a user visually inspects each image acquired at 910. In various embodiments, if an image is determined to not contain a defect, a predetermined field included in the header of the file containing image data corresponding to the image is set to a predetermined value (e.g., zero) indicating that the image is a non-defective image. If an image is determined to contain a defect, a predetermined field included in the header of the file containing image data corresponding to the image is set to a predetermined value (e.g., one) indicating that the image is a defective image. Method 900 then proceeds to 930.

[0067] At 930, a number of pixels corresponding to defects are determined in each image of the workpiece classified as defective at 920. For example, a user may visually inspect each image classified as defective and then use a software tool to set the color value (e.g., grayscale value) of each pixel in the overlaid transparent image corresponding to each defect to a predetermined value (e.g., 255). Method 900 then proceeds to 940.

[0068] At 940, a model training is determined. For example, the defect detector 140dp may divide the images acquired at 910 into groups based on how the images were classified at 920, similar to that described above in connection with FIG. 7. A user may also use a graphical user interface to select a particular one of the models included in the defect detector 140dp. Method 900 then proceeds to 950.

[0069] At 950, model training is initiated based on the model training determined at 940. For example, this can be done automatically, or the user can press a button or select an icon using a graphical user interface to initiate model training based on the model training determined at 940. Method 900 then ends.

[0070] Figure 10 shows a flow diagram of a method 1000 for operating a machine vision inspection system similar to that of Figure 1. Method 1000 begins at 1010. At 1010, a defect detector (e.g., defect detector 140dp) is trained to detect defect images, including defective workpieces, using a first plurality of training images of one or more workpieces acquired by a camera (e.g., camera system 260). Method 1000 then proceeds to 1020.

[0071] At 1020, the defect detector 140dp, trained using the first set of training images, is tested using a first plurality of test images of the workpiece acquired by the camera that are not included in the first set of training images. A first test image performance metric is determined from this testing. For example, the performance metric may be calculated by dividing the number of training images correctly classified as defective images by the total number of defective images in the first plurality of test images. Method 1000 then proceeds to 1030.

[0072] At 1030, the defect detector 140dp is trained to detect defect images using the first plurality of training images and the second plurality of training images acquired by the camera.

[0073] At 1040, the defect detector 140dp, trained using the first and second plurality of training images, is tested using a second set of test images acquired by the camera that are not included in either the first or second plurality of training images. A second test image performance metric is determined from this testing. For example, the performance metric may correspond to the accuracy of the defect detector 140dp. This accuracy may be calculated by dividing the number of defect images correctly classified as defect images by the total number of defect images included in the second test image set. Method 1000 then proceeds to 1050.

[0074] At 1050, the defect detector 140dp determines a recommended number of defect images for training the defect detector 140dp based at least in part on the first and second test image performance criteria. For example, the defect detector 140dp fits a suitable curve to the training image performance versus the number of defect images used for the model training results, fits a suitable curve to the test image performance versus the number of defect images used for the model training results, and uses a convergence algorithm to find a value for the number of defect images used to train the model at which the fitted test curves and the fitted training curves converge to each other, similar to that described above in connection with FIG. 7. Method 1000 then proceeds to 1060.

[0075] At 1060, after training defect detector 140dp with at least the recommended number of defect images, new images of the workpiece are acquired using the camera while in run mode, and defect detector 140dp is used to analyze the new images to determine defect images that include the defective workpiece (e.g., including classifying the new images as defect images and non-defect images). Method 1000 then ends.

[0076] FIG. 11 shows a flow diagram of a method 1100 for operating a machine vision inspection system similar to that of FIG. 1. For example, similar to that described above in connection with FIG. 7, defect detector 140dp is trained using multiple groups of training images. Each training image group includes a predetermined number of defect images. More specifically, defect detector 140dp is trained during a series of training cycles. During a first training cycle, defect detector 140dp is trained with the first group of training images. During a second training cycle, defect detector 140dp is trained with the first group of training images along with the second group of training images. During a third training cycle, defect detector 140dp is trained with the first and second groups of training images along with the third group of training images. During a fourth training cycle, defect detector 140dp is trained with the first, second, and third groups of training images along with the fourth group of training images. After each training cycle, the accuracy of the defect detection unit 140dp is determined using two image sets, including a training image set used to train the defect detection unit 140dp in the immediately preceding training cycle and a test image set not used to train the defect detection unit 140dp in the immediately preceding cycle.

[0077] Method 1100 begins at 1110. In various embodiments, method 1100 of FIG. 11 may be an example of a particular process performed as part of 1050 of FIG. 10. As shown in FIG. 11, at 1110, a corresponding curve is fitted to the training image performance versus the number of defect images used for the model training results. For example, after each training cycle, the accuracy of defect detector 140dp determined using the training image set used to train defect detector 140dp in the immediately preceding training cycle is fitted to a corresponding curve using conventional curve fitting techniques. Method 1100 then proceeds to 1120.

[0078] At 1120, the test image performance versus the number of defect images used for the model training results is fitted with a corresponding curve. For example, after each training cycle, the accuracy of the defect detector 140dp determined using a test image set not used to train the defect detector 140dp in the immediately preceding training cycle is fitted with a corresponding curve using conventional curve fitting techniques. Method 1100 then proceeds to 1130.

[0079] At 1130, an algorithm is used to find the number of defect images used to train the model at which the fitted test curve and the fitted training curve converge to each other. For example, similar to that described above in connection with FIG. 7, the algorithm used by defect detector 140dp considers the distance between the fitted test curve and the fitted training curve and obtains the number of defect images used to train the model at which the fitted test curve and the fitted training curve converge to each other such that the distance between the curves is less than or equal to a predetermined value. Method 1100 then proceeds to 1140.

[0080] At 1140, it is determined whether additional defect images are recommended for training. For example, defect detector 140dp compares the number of defect images currently being used to train defect detector 140dp with the number of defect images used to train the model found in 1130 at which the fitted test curves and fitted training curves converge to each other. If defect detector 140dp determines that the number of defect images currently being used to train defect detector 140dp is equal to or greater than the number of defect images used to train the model found in 1130 at which the fitted test curves and fitted training curves converge to each other, defect detector 140dp determines not to recommend additional defect images for training. Otherwise, defect detector 140dp determines to recommend additional defect images for training. Method 1100 then ends.

[0081] Figure 12 shows a flow diagram of a method 1200 for operating a machine vision inspection system similar to that of Figure 1. Method 1200 begins at 1210. At 1210, a location of a defect within a defect image is determined. For example, defect detector 140dp determines the location of defect 502D shown in Figure 5D. Method 1200 then proceeds to 1220.

[0082] At 1220, one or more metrology operations to perform on the defect may be determined. For example, it may be determined which dimension / feature of the defect to determine / measure, such as depth, length, width, etc. (e.g., as described above with respect to FIG. 6 ). In one exemplary embodiment, defect detector 140dp may determine to perform a metrology operation to obtain a 3D point cloud of an area containing defect 502D and to determine the maximum depth of the defect using the 3D point cloud. Method 1200 then proceeds to 1230.

[0083] At 1230, one or more metrology operations determined at 1220 are performed on defect 502D. For example, video tools and / or other metrology processes may be used to determine one or more dimensions / characteristics of defect 502D. In one exemplary embodiment, defect detector 140dp may perform metrology operations to obtain a 3D point cloud of an area containing defect 502D and determine a maximum depth of the defect using the 3D point cloud. Method 1200 then proceeds to 1240.

[0084] At 1240, a determination is made as to whether to perform an action based on the dimensions and / or characteristics determined at 1230. For example, if it is determined that the dimensions of defect 502D exceed a threshold, an action may be performed (e.g., issuing an alert, discarding the workpiece, etc.). In one exemplary embodiment, defect detector 140dp may determine whether the maximum depth of defect 502D is greater than or equal to a predetermined threshold. If defect detector 140dp determines that the maximum depth of defect 502D is greater than or equal to a predetermined threshold, it may decide to generate or display a message (e.g., indicating information about the defect and / or indicating that the workpiece should be discarded, etc.). If the dimensions or other characteristics (e.g., of defect 502D) do not exceed a threshold, defect detector 140dp may decide not to perform any action. Method 1000 then ends.

[0085] 13 is a graph illustrating an example performance curve 1304 for the defect detector 140dp. As discussed above and in more detail below in connection with FIG. 13, in various embodiments, it may be desirable to train the defect detector 140dp until it achieves a particular accuracy (e.g., approaching the maximum potential accuracy of the defect detector 140dp in identifying / classifying defects and / or defective pixels in a given image defect / background data set). With respect to such training, it may be desirable to determine how many defect images may be required for training to achieve a particular accuracy (e.g., which may correspond to an accuracy where utilizing additional defect images for training is not expected to result in any significant further improvement in accuracy). In various embodiments, such accuracy may be specified as corresponding to at least some percentage (e.g., 90%, 95%, 97.5%, etc.) of a maximum value (e.g., maximum IoU value) that the fitted performance curve (e.g., performance curve 1304) is expected to reach, and providing a large number (e.g., 500) or an infinite number of additional defect images for training is expected to result in only a relatively small percentage improvement in accuracy (e.g., 10%, 5%, 2.5%, etc.) (e.g., relative to the maximum value). With regard to the corresponding asymptote of the fitted performance curve (e.g., performance curve 1304), various embodiments may utilize different techniques for such a determination. For example, such techniques may include determining the slope of the curve, examining the error with the model at the top of the curve, and / or defining the asymptotic portion of the curve (e.g., according to a maximum value of negative curvature) and examining whether the current data may exceed that point (e.g., or other points mentioned above) of the fitted curve (or examining the data itself or a combination of the two). As some other examples, a DNN, or SVM, or other machine learning tool may be trained to provide corresponding instructions, etc. In some embodiments, following such a determination, the number of defect images required for training may vary (e.g., depending on the particular image dataset; in some example embodiments, this number may vary from 20 defect images to 300 defect images). This number may depend on various factors (e.g., variability in the visual characteristics of the defects and background, image quality, etc.), as discussed in more detail below.

[0086] In the graph of FIG. 13 , the x-axis corresponds to the number of defect images used for training, and the y-axis corresponds to the Intersection over Union (IoU) score (i.e., one of several commonly used accuracy metrics used to assess the correctness of semantic segmentation of objects using neural networks). In various embodiments, the IoU pixel-level accuracy score ranges from 0 to 1, where 1 indicates perfect classification of all pixels in the image as normal or defective. A performance curve 1304 is fitted to the determined data points (e.g., a particular number of defect images used for training is shown to produce a particular IoU score) and correspondingly shows the performance of the defect detector 140dp for a unique image dataset. In various embodiments, lines drawn above and below the performance curve 1304 can indicate the confidence level of the curve fitting. In various embodiments, the IoU score starts at the origin and generally increases as more defect images are used for training, up to a certain value, after which little improvement is observed. In various embodiments, the IoU score may not approach 1 (which corresponds to perfect classification; imperfect labeling of defects and / or defective pixels, such as when defect images used for training are manually labeled, may generally keep the score below a value of 1). In various embodiments, other metrics similar to IoU may also be effectively used as a measure of performance accuracy in this method, such as the F1 score (Sørensen-Dice coefficient), which is well known to those skilled in the art of neural network classification models.

[0087] In various embodiments, the defect detector performance curve corresponding to a unique image data set may have a shape similar to an S-shaped curve (e.g., a logistic growth curve). In such examples, there may generally be a point (i.e., corresponding to a particular number of defect images) where utilizing additional defect images for training may result in a relatively small performance improvement. Such a number of defect images used for training is typically M TR(and in some cases may approximately represent a desired minimum number of defect images used to train the defect detector 140dp). TR corresponds to approximately 37 defect images used for training. As noted above, in various embodiments, such analysis can provide a user with an indication as to whether to continue providing additional defect images to train defect detector 140dp (e.g., relative to the current number of defect images already used for training).

[0088] In various embodiments, since the performance curve is expected to be approximately an S-shaped type curve, this information can be useful in determining when a good fit of the performance curve (e.g., performance curve 1304) has been achieved (e.g., corresponding to a good fit to the data points indicative of the performance of defect detection portion 140dp). For example, in various embodiments, a good fit can be considered to have been achieved when there are several (e.g., three or more) data points that are shown to correspond to the "upper" portion of the S-shaped performance curve (e.g., in some examples, the M obtained from the fit). Tr 13, performance curve 1304 is shown to have a lower portion 1310, a rising slope portion 1320, and an upper portion 1330. In this example, upper portion 1330 is shown to include six data points, corresponding to the minimum number M of defect images required for training to achieve the desired performance. TrConfidence in estimating σ can be relatively high. In various embodiments, fewer data points may correspond to the upper portion of the performance curve (e.g., in some examples, there may be as few as 2, 3, or 4 data points depending on the spacing / difference in the number of defect images corresponding to each data point). In various embodiments, the data points at the upper portion of the performance curve may correspond to a relatively flat slope (e.g., approaching zero) and / or the slope may be a small percentage (e.g., less than 5%) of the maximum slope occurring on the rising slope of the performance curve (e.g., in some examples, the slope at each point may be considered to correspond to, e.g., the amount of performance improvement with each additional defect image used for training).

[0089] It will be appreciated that, following such analysis, various indications may be provided (e.g., to a user) indicating whether or not additional defect images should be provided for training. Providing such indications to a user informs the user whether or not they have provided enough training defect images to achieve a desired performance result (e.g., approximately the best possible performance result) for defect detector 140dp for a particular defect image data set. For example, once it is determined that the current number of defect images provided by the user is sufficient (e.g., determined to correspond to the top of the performance curve), an indication (e.g., display of a message, indicator, etc.) may be provided to the user indicating that no more defect images need be provided for training. As another example, once it is determined that the current number of defect images provided by the user is approximately sufficient (e.g., determined to approach and / or approximately correspond to the top of the performance curve), an indication (e.g., display of a message, indicator, etc.) may be provided to the user indicating this (e.g., indicating that the user only needs to provide a small number or some specified number of additional defect images for training, and / or that the performance of defect detector 140dp is deemed / estimated to be approximately at a desired level, etc.). 7, in some examples, a particular analysis may be used to determine a recommended number of defect images for training (e.g., which may correspond to when the top of an estimated performance curve occurs and / or may correspond to approximate convergence with another curve or value set, etc.). Following such a determination, in some examples, a user may be notified in advance of the recommended number of defect images for training (e.g., an indication may be provided as to whether or not to continue providing additional defect images for training relative to the current number of defect images used for training).

[0090] As noted above, various factors may affect the number of defect images required for training in each unique image dataset. One such factor may relate to how visually distinct the defects are from the background in the image. For example, with respect to the example images of FIGS. 3 through 5 , it can be seen that scratch defects 402 and 502 may be difficult to visually distinguish from a background of machining marks (e.g., formed on a plate surface that appears as a hashed texture). With such example image datasets, which may include the images of FIGS. 3 through 5 , a greater number of defect images for training may generally be required to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal, etc.) than would be required for an image dataset including images in which the defects are more visually distinct from the background. Conversely, a smaller number of defect images for training may generally be required to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal, etc.) than would be required for an image dataset including images in which the defects are less visually distinct from the background.

[0091] Another factor may relate to the amount of variability in the visual characteristics of the defects. For example, scratch defects 402A1, 402A2, 402B1, 402B2, 402C1, 402C2, 402D, 402E1, 402E2, 502B, 502C, and 502D in Figures 4A-4E and / or 5B-5D can each be seen to have some amount of variance in certain visual characteristics (e.g., shape, size, etc.). For an example image dataset that may include such defect images, generally, more defect images may be required for training to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal, etc.) than would be required for an image dataset including defect images with less variance in the visual characteristics of the defects. As a specific example, in an image dataset in which all defect images contain approximately the same scratch defect (e.g., similar or identical to scratch defect 502D), fewer training defect images may be required to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal) than would be required in an image dataset containing images with greater variation in the visual characteristics of the defects, such as those shown in the images of Figures 4A to 4E and Figures 5B to 5D.

[0092] Another factor may relate to the amount of variation in the visual features of the image background. For example, the image backgrounds of FIGS. 3A-3F and / or 4A-4E can each be seen to have a certain amount of variation in certain visual features (e.g., due to the wide variety of types of machining marks formed on the plate surface, typically seen as a hash texture that varies across the plate's surface). For such an example image dataset, which may include the images of FIGS. 3A-3F and 4A-4E, a greater number of training defect images may generally be required to achieve a particular level of accuracy than would be required for an image dataset including images with less variation in the background visual features (e.g., assuming other factors are approximately equal, etc.). As a specific example, in the images of FIGS. 5A-5D, the backgrounds are more similar (i.e., the types of machining marks formed on the surface of the plate portions are similar or substantially identical in each image, with the primary difference between these images being the characteristics of each scratch defect 502B, 502C, and 502D). For such an example image dataset, which may include the images of Figures 5A-5D (and other images in this set may include similar backgrounds), generally fewer defect images for training may be required to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal, etc.) than would be required for an image dataset including images with greater variation in background visual features, such as those shown in the images of Figures 3A-3F and 4A-4E.

[0093] Another factor may relate to image quality (e.g., whether image quality is consistent and good with respect to focus, lighting, etc., and / or may vary among different images within an image dataset). For example, it can be seen that the images in FIGS. 3A-F, 4A-4E, and 5A-5D each appear to have relatively good focus, etc. For example, for an example image dataset that may include such images, fewer training defect images may generally be required to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal, etc.) than the number of defect images required for an image dataset that includes images with lower image quality. As a specific example, an image dataset having similar images in which all images are poorly focused (e.g., which may correspondingly make it more difficult to accurately determine features such as scratch defects and the background and / or distinguish scratch defects from the background, etc.) may generally require more training defect images to achieve a particular level of accuracy (e.g., assuming other factors are approximately equal, etc.).

[0094] With respect to these factors, as discussed above, it may be desirable to determine the performance of defect detector 140dp (e.g., to characterize the predictive accuracy of the model used) with each unique image defect / background data set (e.g., as a function of the number of defect images provided for training). Determining such performance, in accordance with the principles disclosed herein, can provide an indication as to whether to continue to provide additional defect images for training (e.g., relative to the current number of defect images already used for training) to achieve a particular accuracy (e.g., an accuracy where providing additional defect images for training is not expected to provide any significant further improvement).

[0095] FIG. 14 shows a flow diagram of a method 1400 for operating a machine vision inspection system similar to that of FIG. 1. At 1410, images of workpieces acquired by a camera of the workpiece inspection and defect detection system are used as training images to train a defect detector 140dp to detect defect images, including defective workpieces. In various embodiments, the training images include both defect and non-defect images. At 1420, the performance of the defect detector 140dp trained with the training images is determined. For example, with respect to the process described above in connection with FIG. 13, performance curve 1304 and / or all or a portion of the associated performance criteria may be determined for the defect detector 140dp. As another example, with respect to the process described above in connection with FIG. 7, performance curves 702 and / or 704 and / or all or a portion of the associated performance criteria may be determined. As noted above, in some embodiments, performance curves 702 and 704 may correspond to a training image set and a test image set, respectively. In various embodiments, some images (and / or corresponding sets and / or data) may be referred to as "validation" images (and / or corresponding sets and / or data). As used herein, such images, sets, and / or data are considered to be "test" images, sets, and / or data of a particular type.

[0096] At 1430, an indication is provided as to whether additional defect images should be provided to be used as training images to train defect detector 140dp, based at least in part on the performance of defect detector 140dp trained with the training images. For example, with respect to the process described above in connection with FIG. 13 , a determination as to whether additional defect images should be provided may be made (e.g., based on the slope of performance curve 1304 and / or related performance criteria). An indication may be provided (e.g., a message may be generated, etc.) regarding whether additional defect images should be provided. As another example, with respect to the process described above in connection with FIG. 7 , a determination as to whether additional defect images should be provided may be made (e.g., based on using an algorithm or other technique to find a value where performance curves 702 and 704 for the number of defect images used to train the defect detector converge to each other, and / or based on the slope of performance curve 704 (e.g., approaching zero), and / or based on other related performance criteria). In response to such a determination, an instruction may be provided (e.g., indicating a recommended number of defect images for training the defect detector, which may indicate whether additional defect images should be provided relative to the current number of defect images already used). In 1440, if additional images are not used as training images for training the defect detector (e.g., providing a message indicating that additional defect images are not needed and / or if the indicated recommended number of defect images have already been used for training), multiple new images of the workpiece may be acquired using the system's camera during run mode, and these new images may be analyzed using the defect detector to determine defect images that include the defective workpiece. Other operations may also be performed, such as the metrology operations described above with respect to FIGS. 6 and 12.

[0097] FIG. 15 illustrates a flow diagram of a method 1500 for operating a machine vision inspection system similar to that of FIG. 1. In various embodiments, method 1500 may represent more specific and / or more detailed example implementations of certain operations 1410-1430 of FIG. 14, as described in more detail below. At 1510, provided images (e.g., including defect images in which defects and / or defect pixels corresponding to the defects have been marked by a user) are used to train a defect detector. At 1520, a performance curve and / or associated performance criteria are determined (e.g., as described above with reference to performance curve 1304 of FIG. 13 or performance curve 704 of FIG. 7, etc.). At 1530, a determination is made as to whether additional defect images should be provided to train defect detector 140dp (e.g., based on the approximate slope, intersection, or one or more other characteristics of the performance curve and / or one or more associated performance criteria, as described above with reference to FIGS. 7 and 13). If additional images are provided at 1540, method 1500 proceeds to 1550 and provides an indication (e.g., a message) indicating that additional images should be provided to train defect detector 140dp. At 1560, additional images are received (e.g., provided by a user, etc.). From 1560, method 1500 returns to 1510 (e.g., uses the additional images for further training). If additional images are not provided at 1540, method 1500 proceeds to 1570 and provides an indication (e.g., a message) indicating that additional images are not required / recommended, etc., to train defect detector 140dp. Method 1500 then ends.

[0098] As shown by the sequence of operations 1510-1560, in various embodiments, the training process may proceed through multiple iterations. Following these iterations, after the user provides a new set of defect images for training, various embodiments may continue to notify the user whether the current number of defect images is sufficient to achieve a desired level of performance for the defect detector 140dp, or whether they should continue to provide additional images. By continuing to notify the user, the user may stop providing defect images when the additional images no longer significantly improve the performance of the defect detector 140dp, thereby saving time and resources. It will be appreciated that, without utilizing a process such as that disclosed herein, the user would need to go to the trouble of collecting and marking defect images (e.g., by marking the defect images or otherwise marking the defects) and further training and assessing performance of the defect detector 140dp before they would know whether providing additional defect images could potentially improve performance. As noted above, for a certain number of images specific to each image defect / background data set, the performance improvement expected from providing additional defect images for training may be substantially negligible. Therefore, it may be desirable to provide dataset-specific guidance as to when a sufficient number of defect images have been provided to achieve the desired performance results (which may, for example, correspond to a portion of the performance curve that approaches zero slope and / or is relatively flat).

[0099] While preferred embodiments of the present disclosure have been shown and described, numerous variations in the arrangement of elements and sequence of operation shown and described will be apparent to those skilled in the art based on this disclosure. Various alternative forms can be used to implement the principles disclosed herein.

[0100] Aspects of the above-described embodiments can be modified as necessary to provide further embodiments employing concepts from various patents and applications herein. These and other modifications can be made to the embodiments in light of the description set forth above. In general, in the following claims, the terms used should not be construed as limiting the scope of the claims to the specific embodiments disclosed in the specification and claims, but rather as embracing all possible embodiments along with the full range of equivalents to which such claims are entitled.

Claims

1. A light source and a lens that receives image light emanating from a surface of a workpiece illuminated by the light source and transmits the image light along an imaging optical path; a camera that receives imaging light transmitted along the imaging optical path and provides an image of the workpiece; one or more processors; a memory coupled to the one or more processors and configured to store program instructions; wherein the program instructions, when executed by the one or more processors, at least: first training the defect detection unit to detect defect images, including defective workpieces, using images of the workpieces acquired by the camera, including both defect images and non-defect images, as training images; determining a performance accuracy of the defect detection unit that has undergone first training using the training images; providing instructions to the defect detector as to whether to provide additional defect images to be used as training images for a second training, which is training of a model of the defect detector used in a run mode, based at least in part on the performance accuracy of the defect detector that has performed a first training with the training images; if no additional images are used as training images for training the defect detector, acquiring new images of the workpiece using the camera during a run mode and analyzing the new images using the defect detector to determine defect images including defective workpieces; on the one or more processors; the training images include a first plurality of training images and a second plurality of training images; performing a first training on the defect detection unit, training the defect detector using the first plurality of training images; training the defect detector using the first and second plurality of training images; Including, Determining the performance accuracy of the defect detection unit includes: using the defect detector trained using the first plurality of training images to determine a first test performance criterion for a first plurality of test or verification images acquired by the camera that are not included in the first plurality of training images; determining a first training performance criterion for the first plurality of training images using the defect detector trained using the first plurality of training images; using the defect detector trained using the first and second plurality of training images to determine a second test performance criterion for a second plurality of test or verification images acquired by the camera that are not included in either the first or second plurality of training images; determining a second training performance criterion for the first and second plurality of training images using the defect detector trained using the first and second plurality of training images; Including, Providing the instructions comprises: determining a recommended number of defect images for the second training of the defect detector based at least in part on a first performance curve generated based at least on the first and second testing performance criteria and a second performance curve generated based at least on the first and second training performance criteria. Workpiece inspection and defect detection systems.

2. 10. The system of claim 1, wherein determining the performance accuracy of the defect detector is based at least in part on determining a number of defective pixels or defective images that are correctly classified as defective pixels or defective images by the defect detector.

3. 2. The system of claim 1, wherein the determining the performance accuracy of the defect detector is based at least in part on determining a number of non-defective pixels or non-defective images that are correctly classified as non-defective pixels or non-defective images by the defect detector.

4. The program instructions, when executed by the one or more processors: performing one or more metrology operations using one or more of the new images of the workpiece acquired during the run mode; The system of claim 1 , further causing the one or more processors to execute:

5. 2. The system of claim 1, wherein providing the instructions includes generating a message including a recommended number of defect images for performing a second training on the defect detection unit, the recommended number of defect images indicating whether additional defect images should be provided to the defect detection unit relative to the current number of defect images already used to perform a first training on the defect detection unit.

6. 1. A method performed by a workpiece inspection and defect detection system including at least a light source, a lens, a camera, and a defect detection component, comprising: first training the defect detection unit to detect defect images, including defective workpieces, using images of the workpieces acquired by the camera as training images, the images including both defect images and non-defect images; determining a performance accuracy of the defect detection unit that has undergone first training using the training images; providing instructions to the defect detector as to whether to provide additional defect images to be used as training images for a second training, which is training of a model of the defect detector used in a run mode, based at least in part on the performance accuracy of the defect detector that has performed a first training with the training images; if no additional images are used as training images for the second training of the defect detection unit, acquiring new images of the workpiece using the camera during a run mode, and analyzing the new images using the defect detection unit to determine defect images including defective workpieces; Including, the training images include a first plurality of training images and a second plurality of training images; performing the first training on the defect detection unit, training the defect detector using the first plurality of training images; training the defect detector using the first and second plurality of training images; Including, Determining the performance accuracy of the defect detection unit includes: using the defect detector trained using the first plurality of training images to determine a first test performance criterion for a first plurality of test or verification images acquired by the camera that are not included in the first plurality of training images; determining a first training performance criterion for the first plurality of training images using the defect detector trained using the first plurality of training images; using the defect detector trained using the first and second plurality of training images to determine a second test performance criterion for a second plurality of test or verification images acquired by the camera that are not included in either the first or second plurality of training images; determining a second training performance criterion for the first and second plurality of training images using the defect detector trained using the first and second plurality of training images; Including, Providing the instructions comprises: determining a recommended number of defect images for the second training of the defect detector based at least in part on a first performance curve generated based at least on the first and second testing performance criteria and a second performance curve generated based at least on the first and second training performance criteria; generating a message to the defect detector indicating a recommended number of defect images for second training; the recommended number of defect images indicates whether additional defect images should be provided to the defect detector in addition to the current number of defect images already used in performing the first training. method.

7. Determining the performance accuracy of the defect detection unit includes: determining a number of defective pixels or defective images that are correctly classified as defective pixels or defective images by the defect detection unit; or determining a number of non-defective pixels or non-defective images that are correctly classified as non-defective pixels or non-defective images by the defect detection unit; The method of claim 6 , based at least in part on at least one of:

8. performing one or more metrology operations using one or more of the new images of the workpiece acquired during the run mode; The method of claim 6 further comprising:

9. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, at least: first training the defect detection unit to detect defect images, including defective workpieces, using images of the workpieces acquired by a camera of the workpiece inspection and defect detection system, including both defect images and non-defect images, as training images; determining a performance accuracy of the defect detection unit that has undergone first training using the training images; providing instructions to the defect detector as to whether to provide additional defect images to be used as training images for a second training, which is training of a model of the defect detector used in a run mode, based at least in part on the performance accuracy of the defect detector that has performed a first training with the training images; if no additional images are used as training images for training the defect detector, acquiring new images of the workpiece using the camera during a run mode and analyzing the new images using the defect detector to determine defect images including defective workpieces; on the one or more processors; the training images include a first plurality of training images and a second plurality of training images; performing the first training on the defect detection unit, training the defect detector using the first plurality of training images; training the defect detector using the first and second plurality of training images; Including, Determining the performance accuracy of the defect detection unit includes: using the defect detector trained using the first plurality of training images to determine a first test performance criterion for a first plurality of test or verification images acquired by the camera that are not included in the first plurality of training images; determining a first training performance criterion for the first plurality of training images using the defect detector trained using the first plurality of training images; using the defect detector trained using the first and second plurality of training images to determine a second test performance criterion for a second plurality of test or verification images acquired by the camera that are not included in either the first or second plurality of training images; determining a second training performance criterion for the first and second plurality of training images using the defect detector trained using the first and second plurality of training images; Including, Providing the instructions comprises: determining a recommended number of defect images for conducting the second training based at least in part on a first performance curve generated based at least on the first and second testing performance criteria and a second performance curve generated based at least on the first and second training performance criteria; generating a message indicating a recommended number of defect images for performing said second training; the recommended number of defect images indicates whether additional defect images should be provided relative to the current number of defect images already used in performing the first training. Non-transitory computer-readable medium.

10. Determining the performance accuracy of the defect detection unit includes: determining a number of defective pixels or defective images that are correctly classified as defective pixels or defective images by the defect detection unit; or determining a number of non-defective pixels or non-defective images that are correctly classified as non-defective pixels or non-defective images by the defect detection unit; 10. The non-transitory computer-readable medium of claim 9, based at least in part on at least one of:

11. The program instructions, when executed by the one or more processors: performing one or more metrology operations using one or more of the new images of the workpiece acquired during the run mode; 10. The non-transitory computer-readable medium of claim 9, further causing the one or more processors to execute:

Images