Methods for inspecting defects on surfaces

Abstract

A model for defect identification on a surface to be inspected includes: a first supervised machine learning classifier configured to classify image units of an image into at least defective image units and non-defective image units; a plurality of convolutional filter sets performing feature extraction, wherein each such set includes one or more convolutional filters defined by a separate set of filter parameters; applying the convolutional filters to one or more labeled reference images of the surface to obtain multiple sets of feature maps; obtaining multiple trained first machine learning classifiers using the feature maps; providing an input test set for each trained first machine learning classifier and obtaining respective quality parameters; and applying an optimized set of convolutional filters to one or more labeled training images to train a copy of the first machine learning classifier.

Description

Technical Field

[0001] The present invention relates to a method for inspecting surface defects, and more particularly to a method for inspecting a photomask cassette used in photolithography for storing photomasks or photomasks. Background Technology

[0002] To achieve smaller feature sizes, photolithography techniques used in semiconductor manufacturing are constantly evolving towards shorter wavelengths. Currently, wavelengths in the extreme ultraviolet (EUV) range, from 125 nm down to 10 nm, are being increasingly employed. EUV lithography is extremely sensitive to contamination and defects, particularly any defects or contamination in the photomask used. Various types of protective housings and storage containers have been developed for storing, handling, and transporting EUV photomasks. A widely used solution provides a dual-cassette carrier for photomasks, comprising two nested housing elements that form an inner EIP (EUV inner cassette) and an outer EOP (EUV outer cassette). This protects the photomask within the dual cassette from contamination by the external atmosphere. To ensure that photomasks are always handled safely and cleanly, EUV cassette contamination and defects (such as scratches or material fatigue) must be inspected. For example, cassette inspections can be performed periodically or after a certain number of processing operations or cycles. However, cassette materials can be made from a variety of materials with different surface properties (such as highly reflective and matte surfaces). The surfaces to be inspected will typically have a large number of very small defects, potentially thousands of defects in the micrometer range, and highly reflective surfaces can result in a lot of noise in the surface images. Therefore, it is necessary to identify defects with high accuracy and very fast speed. Summary of the Invention

[0003] A method for constructing a model for defect identification on a surface to be inspected, as defined in the appended independent claims, and a method for inspecting surface defects are proposed. The dependent claims define further exemplary embodiments.

[0004] This disclosure relates to a first method for constructing a model for defect identification on a surface to be inspected, wherein the model includes a first supervised machine learning classifier configured to classify image units of an image into at least two categories, the categories including at least a defective image unit category and a non-defective image unit category. The method includes defining a plurality of convolutional filter sets for feature extraction from the image, wherein each of the plurality of convolutional filter sets includes one or more convolutional filters defined by a set of filter parameters, each convolutional filter set differing from another convolutional filter set in at least one filter parameter; applying the convolutional filters of each of the plurality of convolutional filter sets to one or more labeled reference images of the surface to obtain a plurality of feature maps, each set of feature maps corresponding to one of the plurality of convolutional filter sets; and obtaining a plurality of trained first machine learning classifiers by training a plurality of untrained copies of the first machine learning classifier using the plurality of feature maps as a training set, wherein each copy of the first machine learning classifier is provided with a set of feature maps from the plurality of feature maps as input, such that each of the trained first machine learning classifiers is coupled to the plurality of convolutional filter sets. A set of convolutional filters is associated with each of the trained first machine learning classifiers; for each of the trained classifiers, an input test set is provided to the trained classifier, wherein the input test set comprises a set of test feature maps obtained by applying the set of convolutional filters associated with the trained classifier to at least one predefined test image and obtaining quality parameters indicative of the quality of the classification result of each trained machine learning classifier; based on the obtained quality parameters associated with each of the plurality of convolutional filter sets, an optimized set of convolutional filters is determined from the plurality of convolutional filter sets; the optimized set of convolutional filters is applied to one or more labeled training images of the surface to obtain a set of optimized training feature maps; and a copy of the first machine classifier is trained using the set of optimized training feature maps as training data to obtain a first trained machine learning classifier. An image unit may, for example, be defined in a conventional manner as an image pixel.

[0005] In this way, the step of optimizing feature extraction via convolutional filters can be separated from the classification step, resulting in a less complex machine learning classifier. For example, a multilayer perceptron or a mixture of Gaussian models can be used to implement the machine learning classifier. As a result, model training and, particularly, classification during online (live) surface inspection using such a model, require considerably less processing power and are faster than defect identification in existing techniques.

[0006] Filter parameters used to define convolutional filters can include one or more of the following: filter type, kernel size, stride, padding size, activation flags of the convolutional filters, or the number of convolutional filters in a group. Using parameterized filters in this way allows testing any desired number of different sets of convolutional filters to find the set that yields optimized classification results. Multiple sets of convolutional filters can be determined, for example, by randomly assigning values ​​from a given range to one or more of the filter parameters for each set.

[0007] Test images for testing the results of model training can be generated by: identifying at least one region of a reference image labeled as defect-free; constructing a background image by tiling multiple copies of the defect-free region of the reference image; and adding one or more simulated defective image regions to the background image, wherein the defective regions are defined by predetermined and / or random image parameters for each image unit of the simulated defective region. For example, the test image may include multiple simulated defects of different sizes, contrasts, shapes, and locations, and the trained model can be tested on a single image to determine the quality of the model and filters.

[0008] In another variation of the method for building the model, the model may include at least a second machine learning classifier configured to classify defective image regions in an image into one of a plurality of defect type categories. The method may then further include displaying a plurality of defective image regions to a user, each defective image region comprising one or more defective image units; assigning at least one label indicating a defect type category to each said defective image region based on user input; and training said at least one second machine learning classifier using said labeled defective image regions as training data. That is, the classifier for identifying defects in the image is trained separately from the second classifier for determining the defect type, which provides faster and more robust recognition in the final model. In addition to or as an alternative to manually labeled training datasets, defective regions may be at least partially labeled automatically, for example, using threshold-based classification methods.

[0009] Furthermore, a method for inspecting surface defects is proposed, wherein at least one inspection image of the surface to be inspected is obtained, and a predefined set of optimized convolutional filters is applied to the at least one inspection image to obtain a set of inspection image feature maps. The obtained set of inspection image feature maps is then used as input to a model, which includes at least a first trained machine learning classifier, and as output of the model, the model obtains classification results for image units of the at least one image. The classification results at least indicate image units of the image that are classified as defective image units. In this case, feature extraction using optimized convolutional filters is not included in the backpropagation machine learning network but is performed as a separate step, thereby accelerating both the network training and classification processes.

[0010] The method may further include the following steps: providing parameters of the defective image unit as input to at least one second trained machine learning classifier of the model, wherein the at least one second trained machine learning classifier is configured to classify defective image regions in an image that include one or more defective image units into one of a plurality of defect type categories; obtaining a classification result for the defective image region, the classification result indicating, for each defective image unit, the associated defect type category as the output of the at least one second trained machine learning classifier. Because the defect type classifier (in both training and classification) is separate from the actual defect pixel detection in the image, the training database and classifier of the labeled defective image regions can be at least partially reused for other backgrounds, other surfaces, or other devices.

[0011] The output can be used for various further steps; for example, one method may include visually displaying a defect image unit in the at least one inspected image to a user. In this way, a user (e.g., an operator of the inspection equipment) can easily identify defects in the surface image, correct or re-examine any potential identification errors, or initiate further steps to address the defects.

[0012] It is also possible to retrieve stored parameters of defect image units and / or defect type categories from at least one earlier inspection image obtained at an earlier time point, compare the retrieved parameters of the defect region and / or defect image units from the earlier inspection image with the classification results obtained in the current inspection image, and define the defect image region as a changed defect if the difference (discrepancy) between the defect region and / or defect image units in the comparison exceeds a predefined threshold. Using comparisons with previous image inspections aids in the lifecycle monitoring of surfaces, such as observing changes in defects, identifying severe degradation of components, or checking the success of measures such as surface cleaning processes.

[0013] A method for inspecting a surface may further include the following steps: providing at least a first parametric classifier configured to classify image units into defective image units and non-defective image units based on predefined parameter thresholds for image units in an image; providing at least a second parametric classifier configured to classify a defective image region in an image, comprising one or more defective image units, into one of a plurality of defect type categories based on predefined parameter thresholds for the defective image region; selecting a first classifier from the first trained machine learning classifier and the first parametric classifier for classifying the defective image units, and using the selected classifier to determine the defective image units; and selecting at least a second classifier from the at least one second trained machine learning classifier and the second parametric classifier, and using the selected second classifier to classify the defective region into a defect type category; wherein the selection of the classifier is based on one of: user input, image characteristics, or surface characteristics of the imaged surface. In this way, the user (or automated process) can choose between computationally intensive machine learning classification and a simpler parametric method. It is also conceivable that, in some cases, parametric methods provide better results for defect identification, such as on certain surface structures or when a fully trained model is not yet available.

[0014] Defect type categories can be defined based on one or more parameters of the defective image region, wherein the parameters include at least one of the following: an indication of whether the defect is acceptable; the size of the defective image region; the contrast level of the defective image region; a parameter indicating the isoaxiality of the defective image region; the relative size of the defective image region; and the defect type. These categories can be used for machine learning classification and parameter-based classification, or different parameters can be used for each classifier. Using such parameters allows for differentiation of, for example, scratches, large low-contrast defects that may indicate surface contamination, defects of different shapes that may be identified as shallow defects or critical volume defects, and others. For different surfaces, different imaging techniques, or different classification models, the defect types and parameters, and their interpretation, can vary.

[0015] The second machine learning classifier that classifies regions into defect types can be implemented in various ways; for example, the second classifier may include at least a first sub-classifier and one or more second sub-classifiers in succession (cascade), wherein the first sub-classifier is configured to classify defect image regions as acceptable defects or unacceptable defects, and wherein one or more second sub-classifiers are configured to classify the acceptable defects or the unacceptable defects as defect type categories.

[0016] Images used at any stage (i.e., training images, reference images, test images, or inspection images) can be preprocessed in various ways before applying a set of convolutional filters. For example, image preprocessing may include one or more of the following steps: enhancing contrast values ​​in the image; applying coordinate transformations to the image registration based on reference coordinates; and applying image filters to the image. These and other image processing steps can be used to enhance the identification of potential defects on the surface or to reduce unwanted noise in the image that may lead to false defects in the results. Furthermore, images may be downsampled, for example, to both reduce data size and improve recognition results. Image registration and transformation can be used to ensure consistent coordinate references or to combine several images. Several preprocessing steps can be combined or performed sequentially. Additionally, several images can be processed together to form a final reference image, inspection image, or training image, for example, by constructing a difference image or convolution of images. It is also possible to combine several images of the surface captured using different cameras, different fields of view, different regions of the surface, different imaging techniques, different lighting conditions, and / or different wavelengths into a single image.

[0017] As an example, preprocessing may include obtaining a dark-field image and a bright-field image captured with the same field of view; obtaining a contrast-enhanced image by enhancing the local contrast in each of the dark-field and bright-field images; determining a defect-free image background in each of the dark-field and bright-field images by calculating an average image value; obtaining a difference image (difference image) of each bright-field and dark-field image by calculating the difference between the image background and each of the contrast-enhanced images; and combining the difference images by selecting the maximum value for each image unit from the bright-field and dark-field images.

[0018] For each image region corresponding to a surface region with specified different surface properties, a separate model can be trained, including a separate trained classifier and a separate optimized filter set. The untrained machine learning classifier itself can be the same or different for different surface regions. Similarly, a separate trained model can be provided for each surface region in the final trained inspection system. For example, surfaces can include regions of different materials or surface structures, different focal planes, regions where imaging is geometrically constrained, or others.

[0019] The described method may also include defining a continuous image region as a defective image region, the continuous image region comprising multiple defective image units located adjacent to each other or at least at a predefined maximum distance from each other. For example, an image region may be determined by defective image units (pixels) whose distance from the next defective image unit is less than a predefined distance. It is also possible to distinguish between image units directly adjacent to another defective image unit and image units formed by clusters of independent defective image units.

[0020] The various functions, steps, and methods described herein can be implemented or supported by one or more computer programs, each computer program being formed by computer-readable program code and embodied in a computer-readable medium. A computer program may include one or more software components, instruction sets, programs, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in appropriate computer-readable program code. The phrase "computer-readable program code" can include any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" can include any type of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital video disc (DVD), or any other type of storage. Computer-readable media includes media that can permanently store data and media that can store data and be rewritten later, such as rewritable optical discs or erasable storage devices.

[0021] This disclosure covers several components that can be combined or used together, or can operate as a standalone solution. Attached Figure Description

[0022] In the following description, exemplary embodiments will be illustrated in more detail with reference to the accompanying drawings, wherein...

[0023] Figure 1 This is an overview of the various software functions that perform the steps of the method according to the exemplary embodiments.

[0024] Figure 2a An exemplary setup for imaging in a box inspection system is described.

[0025] Figure 2b A side view of an exemplary setup for imaging in a box inspection system is shown;

[0026] Figure 2c A plan view of an exemplary EUV mask inner box is shown.

[0027] Figure 3 A schematic flowchart illustrating an exemplary training phase of a pixel defect classifier is shown.

[0028] Figure 4 This is a flowchart illustrating the steps involved in optimizing the convolutional filter for feature extraction;

[0029] Figure 5 A schematic flowchart illustrating an exemplary training phase for a defect type classifier is shown; and

[0030] Figure 6 This is a flowchart of surface inspection using a trained model. Detailed Implementation

[0031] While the following description of several embodiments uses an internal mask cassette for photomasks or photomasks as an example, it should be noted that the invention is generally applicable to defect inspection of any surface, particularly any kind of internal or external mask cassette, mask, and photomask. All these objects face similar surface inspection challenges, which are particularly pronounced in EUV mask cassettes due to the materials used.

[0032] To optimally identify surface defects, a two-stage classification process is proposed. In the first stage (hereinafter referred to as the pixel classification stage in the description), the image can be processed to determine all regions or pixels of the image that can be identified as parts of any kind of defect. In the second stage (hereinafter referred to as the defect classification stage or defect classification module in the description), these regions can be classified into different defect groups or types, such as scratches, objects on the surface, watermarks, surface problems, or others. Defects can also be classified by certain characteristics, such as size, width, and / or contrast. In the following, any kind of surface variation to be detected is represented as a defect, i.e., damage to the surface or bulk material on the surface, and contamination that is directly visible or alters the optical properties of the surface to be inspected. The size of common defects of interest on the surface of a box typically ranges from a few micrometers to several hundred micrometers. Of course, defects or undesirable surface features can generally have any size, and the size of defects that can be detected using the following methods can depend on the field of view and the image resolution of the imaging sensor used.

[0033] Various image processing methods can be employed in both stages of defect identification. Machine learning (particularly feedforward neural networks such as multilayer perceptrons) can be used as a classification mechanism. Therefore, the image filters applied for feature extraction and the training data for the classifier model are particularly important. In this context, machine learning models involve feeding training data to a machine learning algorithm to generate a model artifact as the output of the training process. This artifact can be a fully or partially trained machine learning model, a file storing the model's parameters (such as the model's trained weights), or something else. As is the case for real-time defect identification, at least two machine learning models are built and trained separately for pixel classification and defect type classification.

[0034] In addition to machine learning, this method can provide further options for classifying pixels or defects, such as simple thresholding, which will be explained in more detail below.

[0035] Figure 1An exemplary overview of various software functions or modules that can be used to implement the method steps described below is shown. First module 110 and second module 120 are used to construct a surface inspection model and can be performed offline, i.e., outside of real-time surface inspection. First module 110 can be used to implement a process for constructing a pixel defect classification model that classifies pixels (or image units in general) of an image into defective pixels and non-defective pixels. Second module 120 can implement another process for constructing a defect type classifier that classifies pixels or regions identified as defective pixels into defined pixel type categories. Using the surface inspection model constructed in these steps as a combination of the pixel defect classifier and the defect type classifier facilitates real-time defect identification 150 implemented by surface inspection module 130, and the results of surface inspection module 130 can be used to separately implement further applications, such as module 140 for identifying defect variations (e.g., for online cyclic monitoring 160).

[0036] In the following text, it is assumed that a surface image of the surface to be inspected has been captured, preferably a planar view of the surface or at least a portion thereof. In the same manner, such surface images can be obtained to construct models and define various parameters of the inspection method. Preferably, the imaging parameters of the images used to construct and train the classification model should be substantially the same as the imaging parameters used in the subsequent actual surface inspection procedure.

[0037] Exemplary methods for obtaining images for the classification and inspection methods described herein will be described in detail below. For example, images can be obtained using illumination of one or more wavelengths or continuous wavelength spectra, dark field and / or bright field illumination, different types of imaging sensors, and methods such as line scan imaging or area scan imaging.

[0038] Figure 2a Elements of an exemplary setup for obtaining surface images of an EUV box for defect inspection are depicted. Inspection station 210 can be arranged to process EUV box 20 in a partially or fully automated process, for example by providing loading modules 222, 224 adapted to open the outer EUV box 201 to access the inner EUV box 200. The same or additional modules can be provided to process the EUV box assembly, such as separating the base member of box 202 from the cover member 204, removing the mask stored within box 20, rotating or otherwise aligning the box, or, for example, using a processing robot 242 to transfer the mask box or mask assembly to imaging positions 262, 264. For the imaging step, the box or box assembly can, for example, be placed in a container or frame on a holding surface, which optionally allows for precise positioning of the placed component. Figure 2bA schematic diagram of an exemplary inspection unit 262 is shown, which is used to image EUV cassette members 202, 204 held in an insert or component holder 270. For example, a multi-axis stage 275 can be used, which can facilitate translation and / or rotation in several axial directions to allow precise positioning of the cassette members relative to one or more imaging sensors and to image all surfaces of the object to be inspected. The positioning of the cassette can be performed automatically or based on user input. Such a stage 275 can also be used to focus a pre-arranged surface to be inspected onto one or more imaging devices, for example, by adjusting the position of a camera located above the surface along the z-axis.

[0039] Imaging systems 262, 264 may include one or more imaging devices 262a, 262b, such as line scan or area scan cameras. Line scan cameras typically include an imaging sensor with a single row (or a narrow array of very few rows) of sensor elements, allowing imaging of a linear segment at a time. To obtain an image of a two-dimensional region by scanning, the surface to be captured or the line scan camera needs to be moved, for example using the stage 275 as described above. This movement should be synchronized with the imaging process so that images of the two-dimensional region are generated as a sequence of line images. It will be understood that the positioning elements described above can be used to bring the box surface to the desired position for imaging, or to provide scanning motion in a predefined direction in conjunction with the line scan camera. In contrast, area scan cameras include an array of sensor elements suitable for imaging a two-dimensional region at a time. While it is possible to use sensors of sufficient size and resolution to provide an image of the entire surface to be inspected, it is also conceivable to move the camera and / or object to image different surface areas, and optionally combine these into a single image if desired. If image stitching is required, an overlapping area of ​​an appropriate size (e.g., 100 pixels) can be considered for the imaging location. The amount of overlap can vary and can depend on factors such as image sensor size and positioning accuracy.

[0040] The imaging system may have a single camera, or it may include several line-scan and / or area-scan cameras, such as cameras 262a and 262b, which have different fields of view and / or different optical characteristics, such as wavelength sensitivity or resolution, on the objects 202 and 204 to be inspected. Images from several cameras can then be processed individually or in combination. If several cameras are used for imaging, these cameras can be set up in a single location or in separate imaging stations, allowing the mask cassette or cassette assembly to be moved from one imaging station to another and to continuously capture different types of images.

[0041] Especially in mask box imaging or similar surfaces with a wide variety of different properties, it may be appropriate to use line scan imaging for some parts of the surface and area scan imaging for others. Figure 2c A plan view of the EUV inner casing member 204 is shown as an example. It is to be understood that the methods described herein can generally be used to examine other casing designs and other surfaces. As shown, the surface to be examined will typically have several edges raised or recessed from the horizontal plane and may have areas with very different optical properties, such as matte opaque surfaces, structured surfaces such as filtering areas, or highly reflective surfaces. Therefore, some areas of the surface may be out of focus in the plan view, may be outside the field of view during imaging, or may be substantially invisible or lack sufficient contrast for a certain type of imaging sensor or method. For example, although... Figure 2c Most of the surfaces shown can be inspected using line scan and / or area scan imaging, but other areas, such as region 204a, which forms an operational ridge for gripping or positioning the lid component, cannot be easily scanned in this manner. Other areas, such as edge region 204b, can be inspected using line scan or area scan imaging, but may only provide inspection images of limited quality. Furthermore, in this example, region 204c (which includes filters and / or openings for gas exchange between the interior of the inner box and the volume between the inner and outer boxes) can be conveniently inspected by means of area scan, and region 204d, covering the largest portion of the lid surface, can be conveniently inspected by means of line scan.

[0042] In addition, common components of imaging systems, such as one or more illumination devices (not shown), can be provided in the inspection system. Illumination can be provided using light of one or more wavelengths in the visible or invisible spectrum, and the illumination devices (e.g., lamps, lasers, optionally equipped with filters) can be arranged around the object to be imaged in any desired manner. The illumination of the surface to be inspected can, for example, include dark-field illumination and / or bright-field illumination. In dark-field illumination, essentially only light scattered from the object or surface is detected on the imaging sensor. In bright-field illumination, the light source is arranged such that light passes through the object (e.g., for translucent materials) and attenuated light is detected at the imaging sensor. Illumination conditions can have a significant impact on the detectability of surface defects in the image. For example, scratches on the surface of an EUV cartridge assembly are reliably detectable by means of dark-field imaging, while contamination particles on the surface of an EUV cartridge assembly are easily detected by bright-light imaging. By combining these two imaging techniques, a broad spectrum of surface defects and contaminants can be detected. It is also possible to use both dark-field and bright-field illumination simultaneously during imaging, for example, by using light of different wavelengths / colors, which can be separated by image processing after detection, or by using detectors with appropriate wavelength sensitivity to detect them separately. Preferably, the components of the imaging system (such as imaging sensors or cameras, illumination devices, and similar parts) can be arranged in a manner that is at least partially fixed, such as being fixedly positioned within a frame or housing, to ensure consistent imaging and illumination conditions during measurement.

[0043] Data captured by one or more imaging devices 262a, 262b, 264 can be obtained by a suitable processing unit 202, such as one or more computers, a central controller, several microcontrollers or microprocessors for different functions, and can be temporarily or permanently stored, processed and / or forwarded to other components and devices in any desired manner. Data from other components of the imaging system (such as data from the positioning system, illumination device or various sensor elements) can be processed and / or stored together with the image data. The same or another processing unit can be configured to control elements such as the motion stage 275, the cartridge processor 242, the loading modules 222, 224 or any other controllable element of the system.

[0044] This system can also be a combined system for inspecting both the mask cassette and the mask itself, for inspecting the inner and / or outer casing, and / or for cleaning the surface of the inspected casing using various cleaning processes if contamination is found. Therefore, corresponding system components not shown or described herein may be present, and / or components described herein may be omitted from inspection systems employing the methods described below.

[0045] It should be understood that the above setup is mentioned only by way of example, and a wide variety of other imaging systems, devices and methods can be used to obtain surface images for defect identification.

[0046] The surface image of the surface to be inspected can be processed in the first step and used to prepare the inspection model and define various parameters of the method. However, the images used in actual surface inspection should be processed in essentially the same way as any image data used for model building and training to ensure correct classification.

[0047] Image registration processes can be used to ensure a consistent reference coordinate system is used for surface classification, defect identification, defect labeling, and any other steps requiring information about the surface's properties relative to other elements. Furthermore, image registration processes can be used to combine several image datasets, for example, from different sensors, different fields of view, or different imaging depths, into a single image.

[0048] Reference data for image registration can be provided, such as CAD (Computer-Aided Design) images, which are typically readily available from the manufacture of the box. It should be understood that any other image format or other data element used to achieve image registration can also be used. For example, instead of a complete reference image, the coordinates of reference points can be used. The characteristic structure of the surface to be inspected (such as holes, edges, fasteners with known shapes, or any other structure) can be used for image registration and to determine any transformations required.

[0049] As another part of the image data preprocessing, various methods for enhancing image contrast can be applied, such as using lookup tables to map pixel values ​​to enhancement values. For example, if an image of a surface region has been captured as both a bright-field image and a dark-field image, these images can be used to increase the contrast of defects by combining them. To do this, contrast can be enhanced separately in the bright-field and corresponding dark-field images, and then the contrast-enhanced images can be combined into an enhanced composite image. Optionally, further steps can be taken, such as inverting one of the dark-field and bright-field images. This enhanced composite image can then be used as input for any further steps.

[0050] In another variation, images can be combined into a defect image by first determining the background of the two images via calculating the mean, and then obtaining separate defect images of the dark and bright fields by calculating the difference between the background and the enhanced image. The maximum value of the defective dark and bright field images can then be determined and used to construct a combined image showing the maximum defect contrast.

[0051] Alternatively or additionally, other steps can be used to alter, for example, the contrast or color in the image. It is also possible to use more than two different images to generate combined defect images, such as images at different wavelengths. Typically, the original image can be processed in a way that produces an enhanced image that emphasizes the defect and thus allows for better identification. Optimizing the steps and methods used for preprocessing (such as determining values ​​for a lookup table or combinations of certain images) can be based on the final results of experimental defect classification.

[0052] As a further optional step, the image resolution can be reduced to improve the performance of the detection algorithm. Care should be taken not to lose defect information due to reduced resolution; on the other hand, reducing resolution can also reduce spurious effects in defect identification and thus lead to better classification results. For example, the Gaussian pyramid method can be applied to both bright-field and dark-field images, but the interpolation parameters can be chosen differently for bright-field and dark-field images. Similarly, other downsampling and / or filtering methods can be employed.

[0053] All the preceding steps have been described only as an example process for obtaining a surface image suitable for defect classification. It should be understood that other methods may be used, some steps may be omitted or modified, or other steps not described herein may be added. The following steps for identifying and classifying surface defects in both stages can also be used in any method using image data that has been preprocessed in a different way or not preprocessed at all.

[0054] Any parameters used in image preprocessing and model building can be stored in a suitable manner, such as in a data file, so that the same parameters can be applied during the surface inspection process. Alternatively, parameters can be modified by user input or by using values ​​obtained through other methods.

[0055] This method employs a two-stage classification approach to detect surface defects in images. In the first stage, defective pixels (regions) on the surface can be identified, and in the second stage, these defective pixels can be classified into various types of defects, such as scratches, small defects, pseudo-defects, defect clusters, or others. Based on this classification, any defect on the surface can be further analyzed regarding its impact on the functionality of the object under inspection; for example, some defects may appear on the surface of a box but may not pose a risk to the integrity of the box, while other defects may cause serious damage to the box and therefore to the masking film used for packaging.

[0056] Therefore, the goal of the first classification stage is to detect defective regions in the image, i.e., to determine which image sub-units should be defined as "good" or without defects, and which are parts of defective regions. Image sub-units classified in this way can be, for example, image pixels as commonly defined, or can be specified in any other way. In the example below, pixels are used as the basic sub-units of the image. The number and size of pixels depend on the image resolution used and can vary, for example, according to the desired sensitivity of defect identification. Obviously, the resolution may also be limited by the type of imaging sensor used. Since the surface to be inspected typically does not present a homogeneous image but will have some inherent structure (both regular and / or irregular), the challenge is to distinguish true defects from this structure and any other surface characteristics, such as reflection.

[0057] Reference images can be used for different stages of model building (i.e., defining parameters, training the classifier, and other steps to obtain the final classification model). Figure 3 This is a flowchart illustrating exemplary steps for constructing and training a pixel defect classifier, using a reference image 300 as input. As an exemplary embodiment, the reference image 300, generated from bright-field and dark-field images that have been processed as described above to generate the maximum defect image, can be used as training data for the pixel classification model. The reference images can be selected such that they provide one or more defect-free regions for all regions of interest on the surface. Typically, at this stage, a single reference image or a small number of reference images may be sufficient.

[0058] Therefore, in the first step, one or more background areas free of defects can be defined and marked on the reference image. Such areas can be manually selected by the user, i.e., by visually inspecting the image on the screen and using a user interface to mark or indicate defect-free areas. It is also conceivable that areas that can be used as background areas can be pre-selected, for example, based on contrast values ​​or homogeneous structure, and these pre-selected areas can be provided to the user for selection of defect-free background areas. Since the object to be inspected may have very different surface areas with different textures and structures (such as matte or polished surfaces), one or more associated reference background images can be selected for each of these areas. For example, for the EUV cartridge surface as described above, most of the surface will be a flat surface area free of interference. In addition, there are areas with uneven appearance (such as air filter areas), structured areas (such as steps and grooves on the cartridge surface or fasteners), and others. Different components may also be made of different materials. For each of these, at least one defect-free background area is selected. The number of background areas used in this step is generally unlimited, and more than one area can be used for each surface area of ​​interest.

[0059] A minimum size can be defined for the defect-free background to ensure that any inherent structure of the surface is fully reproduced in the selected area. For example, a 256×256 pixel background tile can be selected. Then, a full-size background image can be generated by tiling, that is, combining multiple copies of the selected background area together to obtain a large background image of the required size for training.

[0060] A simple way to classify pixels as 'good' or 'defective' is to use a threshold, such as a threshold for the grayscale values ​​of an image. Thus, all pixels with grayscale values ​​below a certain threshold can be considered 'good' pixels, while all pixels with grayscale values ​​above that threshold can be considered 'defective' pixels, or vice versa. However, even in enhanced images, many defects will have poor contrast and therefore cannot be detected by a simple threshold. Furthermore, there are numerous small defects and false defects or noise that should not be classified as defects.

[0061] To address these issues, thresholding can be combined with different lower and upper thresholds to obtain a hysteresis thresholding method. Let's consider the assumption that defects are displayed as bright areas, i.e., with large grayscale values. With hysteresis thresholding, all pixels in the input image with grayscale values ​​greater than or equal to the upper threshold "low" can be immediately accepted as explicit defect pixels. Conversely, all pixels with grayscale values ​​less than the lower threshold can be immediately rejected. When the upper and lower thresholds are chosen to be unequal, any pixel falling between these two thresholds can be considered a potential defect. Additional checks can then be used to determine whether these potential defect pixels are ultimately defined as good or defective. For example, an additional check on a potential defect pixel could include determining whether it is connected to an explicit defect pixel. A length threshold can be used in this check, allowing it to be checked whether a potential defect pixel is connected to one or more explicit defect pixels via a path of a potential defect pixel with a path length equal to or less than the length threshold. In this way, the area around the defect pixel is also considered. These steps can be repeated until all pixels in the image (or image region) to be checked have been classified. Similarly, thresholds can be defined differently, for example, such that any pixel with a grayscale value below the lower threshold is defined as "defective," while pixels with a grayscale value above the upper threshold are defined as "good."

[0062] Thresholding parameters can be defined using defect-free background regions already defined in a reference image. Because these regions are labeled as defect-free, any image parameters within these regions should be below an upper threshold (or conversely, above a threshold if a lower threshold is given for a parameter). However, it is also possible to define the parameters of the thresholding model in other ways and independently of the formation of any reference image.

[0063] While this hysteresis thresholding method allows for simple parameter definition and simple classification, certain types of defects (e.g., connected scratches or defect clusters) may not be detected, and checking the connection paths can become very time-consuming for a large number of small defects.

[0064] Therefore, a trained machine learning model can be used to implement a more sensitive option for classifying pixels / pixel regions as good or defective. In the final surface inspection method, these two methods can be used in parallel or alternately, or supplemented by further classification methods. Exemplary steps for defining and training such a machine learning model for pixel classification will be described in more detail below.

[0065] In contrast to thresholding methods that classify based solely on pixel values, digital filters can be used to include information from neighboring pixels in the classification. The final filtered image can be obtained by convolving the image with a filtering function or kernel and applying filters through further operations such as subtracting or adding to the original or convolved image. As a result of each convolution, a feature map is obtained, providing features for each image pixel. Convolving the input image with a set of convolutional filters will produce the same number of different feature maps. The feature maps obtained from the convolutional filters can be provided in a multi-channel image, where the number of channels corresponds to the number of filters applied.

[0066] Possible candidates for such filters include bandpass filters, Gaussian filters, or various Gaussian-based filters such as difference-of-Gaussian (subtracting two versions of an image blurred with different Gaussian kernels), Laplacian-Gaussian, Gabor filters, or others. However, it will depend on the characteristics of the surface being inspected (such as surface structure and material) to determine which filter(s) will provide the best defect identification results.

[0067] Therefore, a universal set of parameterized convolutional filters can be used, and different sets of parameters (corresponding to different filters) used to define each filter can be tested to find an optimized set of filters. Figure 4 This is a flowchart illustrating exemplary steps for optimizing a set of convolutional filters. At step 420, filter parameters for the convolutional filters are defined, where each set of filter parameters corresponds to a set of convolutional filters 425. The convolutional filter parameters can be defined randomly or based on predefined settings, or optionally pre-selected by the user or an automated optimization process. It is also possible to pre-define the number of convolutional filters used to provide a maximum and / or minimum number of filters, or to pre-select certain filter types and determine a random or optimized set of filters within these constraints. Defining the parameters for each filter in each set can determine both the type of convolutional filter and further filter characteristics, such as kernel size, filter derivative mode, filter standard deviation σ, and others. Optionally, binary parameters for each filter can also be used to enable or disable the filter, i.e., to control the number of convolutional filters in the group.

[0068] As an example, the model can be defined using up to seven convolutional filters as a filter set 425a. Each filter in this filter set can be defined by a set of filter parameters. The number of filters chosen here is arbitrary and can be replaced with any desired number of convolutional filters. As a result of applying these filters to the reference image, seven feature maps are obtained.

[0069] A model can be trained and tested using each parameterized filter set 425a, 425b, 425c, 425d (i.e., each set of filter parameters for all convolutional filters defining a filter set). In step 430, training data can be obtained by applying the selected filter set to one or more reference images, which, as described above, have labeled defect-free background regions. Then, in step 430, a first parameterized filter set 425a, defined by a set of filter parameters, can be applied to one or more reference images 300. The resulting feature maps provide training data including regions labeled "defect-free," which can then be used as input to a machine learning model in step 440, preferably a multilayer perceptron or another feedforward neural network. As a result, trained models 445a, 445b, 445c, 445d corresponding to specific filter sets 425a, 425b, 425c, 425d are obtained. At this stage, training is performed using only a small training dataset.

[0070] Then, in step 450, the initially trained model 445a is tested for model quality (i.e., the accuracy of defect pixel classification) based on the first filter set 425a.

[0071] One or more simulated defects 310 can be added to the defect-free background image as defined above to obtain simulated defect images as a test set, i.e., for testing the quality of the trained model in step 450. These defects can vary in size, angle, shape, pattern, cluster arrangement, and / or any other desired parameters to obtain a wide range of simulated defects. As an example, a simple simulated defect image can be used, which includes a large number of variations of different simulated defects. The parameters used to simulate defects can be determined randomly, optionally using limits, thresholds, or other boundary conditions of defect parameters, such as maximum width or minimum contrast. Alternatively, simulated defects can be generated in defined patterns of different defect types. These simulated defects can then be superimposed or added to the defect-free background image to create simulated defect images. The resulting simulated defect images can, for example, show patterns of linear and / or dotted defects of different widths, lengths, and positions across the background image. Since the defect characteristics are known, and therefore the desired classification as a good pixel or a defective pixel for each pixel of the simulated defect image is also known, these images can then be fed into each trained model, and the results can be analyzed to evaluate the model quality.

[0072] As a possible criterion for model error rate and model quality, the number of correctly and incorrectly classified pixels can be determined. For example, for each simulated defect pixel, it can be determined whether it has been correctly classified as a defect (true positive, TP) or has not yet been identified as a defect (false negative, FN), and similarly, for each background pixel without any simulated defects, it can be determined whether it has been correctly classified as a good pixel (true negative, TN) or has been incorrectly classified as a defect pixel (false positive, FP). These values ​​form a confusion matrix and can then be used directly as a measure of quality as needed. As a simple implementation, the model can be simply rated based on the total number of misclassified pixels, i.e., the sum of false negatives and false positives. Another option is to use the confusion matrix to determine values ​​for a particular model / filter set, such as sensitivity (or true positive rate), specificity (or true negative rate), and / or accuracy. Typically, any other values ​​derived from the confusion matrix can also be used.

[0073] As an example, the accuracy and sensitivity of a particular trained model can be calculated using the usual methods:

[0074] ,

[0075] In other words, sensitivity is the ratio of true positive results (TP) to the sum of true positive TP and false negatives (FN), i.e., the ratio of identified defective pixels to all actual defective pixels; and

[0076] ,

[0077] In other words, the accuracy of the model is the ratio of true positive results (TP) to the sum of true positive TP and false positive results (FP), that is, the ratio of true positive results to all positive results.

[0078] Of course, defining defects as "positive" and good pixels as "negative" is arbitrary and can also be reversed.

[0079] Therefore, the quality of the results obtained by a trained model using a specific set of convolutional filters to obtain input data is used as a measure of the quality of the convolutional filter set itself.

[0080] Similarly, a suitable number of different filter sets 425a, 425b, 425c, 425d can be tested, each filter set defined by its filter parameter set. For example, a first filter set 425a can be selected, comprising a given number of filters with random or predefined first parameters, and convolutional filters defined in this way can be applied to one or more reference images 300. The resulting feature map obtained as the output of the convolutional filter application 430 can be used as training data for an untrained machine learning model 440 for pixel classification. During this stage of filter selection training, only one or a few reference images can be used for training. This first trained model 425a can then be tested on a set of test data as described above, for example, on at least one simulated defect image, and a confusion matrix or values ​​derived from the confusion matrix can be output in step 450 as the result of the evaluation of the filter set used.

[0081] Then, a second filter set 425b with random second parameters can be selected and applied to the same reference image 300 in the same manner. Similarly, the resulting feature maps are provided as training data to the same untrained model, resulting in a different second trained model 445b. Comparatively, testing the second trained model 445b on the same test data (e.g., the same simulated defect image) allows for the evaluation of the quality of this second trained model, and thus indirectly, the evaluation of the quality of the second filter set.

[0082] In this way, more filter sets can be defined, tested, and evaluated based on the final results of each trained model. The number of filter sets to be evaluated is generally unlimited, but can be preset or limited to avoid long training times. Alternatively, a quality threshold can be defined so that more filter sets are defined and tested until a predetermined number of filter sets meet the quality threshold, i.e., sufficient for surface checking. In some cases, a predetermined number of filter sets can be used for training and testing results, and the optimal filter set 465 can be selected from the predetermined number of filter sets based on the aforementioned metrics in step 460. In this case, an optional quality threshold can also be defined such that if the number of misclassifications in the training results exceeds the quality threshold, the filter set will not be accepted as the optimal filter set. Conversely, it is conceivable that testing more filter sets would stop once at least one training result that meets the quality threshold 460 is found.

[0083] After finding the optimal set of convolutional filters 465 for the pixel classification model, this optimal filter set 465 (i.e., the optimal set of parameters defining this filter set) can be used to fully train the machine learning model to build the trained pixel classification model. The training data for this full training can be prepared from one or more reference images, for example, from the image with the largest defect containing labeled regions, using the defined optimal filter set 465. Applying the optimal convolutional filter set to each image in step 470 will again produce feature maps suitable as input training data for the model. Compared to the filter optimization phase, in this full training phase 480, a large number of reference images with labeled, defect-free background regions can be used. As a result, a trained pixel defect classifier 330 is obtained.

[0084] Optionally, test images with simulated defects can be used again as test data to control the quality of the final trained model and determine the required level of training, for example, to avoid overtraining. Of course, other test data, such as manually labeled images of real defects, can be used alternatively or in combination with simulated defect images. Simulated defect images can be generated as described above for testing the trained model during the filter optimization phase.

[0085] The above steps can be implemented in a software function or software module 110 used to construct a first machine learning model 330 as a pixel classifier. This function uses a reference box image 300 and simulated defective pixels 310 as inputs and allows user input, for example, to label defect-free background areas in the reference image in step 340, or to adjust parameters of the vision system, image registration, or image preprocessing in step 350.

[0086] The steps of defining a reference image, finding an optimized filter set, training a machine learning model using the optimized filter set, and evaluating the model using a pixel classifier can be performed individually for each type of background or region on the box, allowing different classification models and filter sets to be used for different regions of the surface to be inspected. The resulting trained models, processing parameters, optimized filter sets, and other data associated with each trained classifier can be stored for retrieval in further functions.

[0087] After a first machine learning model 330 for pixel classification has been found and trained in such a way to produce a pixel classifier model artifact and an optimized set of convolutional filters 465 for feature extraction, at least a second machine learning model can be trained to classify defect regions according to defect type. Similarly, defect type classification in subsequent surface inspection processes can typically be performed based on parameters, such as thresholds for various defect parameters like size, contrast, and isoaxiality, or can be implemented using at least one machine learning model with labeled defect regions as training data. The training data for the machine learning model used for defect type classification can be manually labeled and / or pre-classified using parameter thresholding methods.

[0088] Categories or defect types can be selected as needed. As an example, defect type classification can be divided into two consecutive classifiers. A first defect type classifier can be used to determine whether a defect area is considered acceptable ("qualified") or unacceptable ("unqualified"), while a second defect type classifier selects exactly one defect type for both acceptable and unacceptable defects, such as spots, small areas, large areas, scratches, and others. Alternatively, two separate second defect type classifiers can be used from the two categories obtained from the first classifier, such that one second classifier is used to further determine the defect type of "qualified" defect areas, while the other second classifier is used to further determine the defect type of "unqualified" defect areas. Optionally, if a defect area has already been classified as "qualified," the second classifier can be omitted, such that only defects classified as unacceptable are assigned a specific defect type.

[0089] Alternatively, a single classifier can be used, for example by defining each feasible combination of defect type and acceptable / unacceptable as a separate category, and a multi-class classifier can be used to accurately assign one of these defect type categories to each defect region. In other embodiments, each category may correspond to a certain characteristic, and a multi-label classifier may be used to combine characteristics, such as for classifying defect size, criticality (acceptable / unacceptable), and contrast.

[0090] During the training phase, implemented by the software module or software function 120 for building a trained defect type classifier (i.e., defect type classifier artifact), one or more classifiers (machine learning models) for defect type classification can be provided with labeled training data, i.e., image data including regions that have been defined as defects and labeled with one of the selected defect type categories. Figure 5 The diagram shows a schematic flowchart for constructing a trained defect type classifier.

[0091] The process 120 of labeling defect data used for training may include displaying an image or portion of an image 500 containing defective regions to a user, who then classifies each defect into a defect type category via user input 530. Defective regions can be obtained by applying the trained pixel classification model 330, as described above, to a reference image 500 of the surface, such that pixel regions classified as “defects” are defined as output. Categories for defect type classification may be predefined for user selection and / or provided by user input, allowing new or additional categories to be defined during the labeling process. Optionally, supporting features, such as visually annotating defective regions on the screen or labels proposed based on parameter thresholds, may be displayed to the user. If more than one defective region exists in the image, the user may select one or more defective regions via user input and assign a category as a label to that defective region. In response to the user's labeling, the defective regions may be stored in a database 510 along with the given label, specifically as part of the training dataset.

[0092] Once a sufficient number of defects have been labeled to form a complete training dataset 510, one or more defect type classifiers can be trained to construct a complete defect type classification model 520. Using labeled test data and, for example, a confusion matrix for evaluating the results, the trained defect type classification model 520 can be tested again.

[0093] As a simpler alternative to machine learning models, defect regions can be classified into defect types based on parameters such as the isoaxiality of the defect region, the clustering of several defects, absolute size (length and / or width of the defect region), relative size (e.g., thin defects), contrast, or any combination of these and other defect characteristics. An algorithm can be defined that optionally checks thresholds for one or more of these parameters in a given priority order and assigns a category to the defect region based on the result of each check. If more than one image type is used as input for classification, parameters can also be selected individually for each image, such as different contrast parameters and / or the final classification of dark-field and bright-field images. Similarly, parameter checks for more than one image type can be used as additional conditional checks for category selection; for example, a defect region can be classified into a certain defect type category only if it exceeds a threshold for a given parameter in both the dark-field and bright-field images of the defect region.

[0094] The models used for pixel classification and defect type classification can also be trained and parameterized independently of each other or in a different order. In this case, the defect type classification model 520 can use labeled defect regions as training data, which can be manually found or found using other methods, instead of using the trained pixel classification model 330 to define the defect regions.

[0095] It should be understood that the defect type categories and parameters given here are for illustrative purposes only, and any desired classification can be used. In particular, the choice of category will also depend on the type of surface to be inspected, for example, based on what action is intended to be taken in the case of a certain defect type, or based on the surface material.

[0096] The steps and stages described above can be used to obtain the final trained classification models for both pixel classification 330 and defect classification 520. These final models can then be saved and used for defect recognition on all boxes of the same type. References will be made below. Figure 6 An exemplary embodiment of surface inspection of an EUV box is described. Figure 6 An exemplary process for software function 130 for surface inspection is shown.

[0097] Again, for example, the image acquisition methods and systems detailed above can be used to obtain one or more surface images 600 of the surface of the cassette to be inspected. Image registration is particularly important if imaging is repeatedly performed on the same surface, such as during periodic inspections throughout the cassette's lifespan.

[0098] The preprocessing of the obtained images for inspection can also be performed in the same manner as described for the reference images and training data in the training phase. Preferably, the steps and parameters 610 of image preprocessing in the surface inspection phase correspond exactly to those steps and parameters in the model building phase to ensure the validity of the classification results. The preprocessing parameters 610 can be stored, for example, with the classification models 330, 520, or integrated as predefined parameters in the final inspection module 130.

[0099] After at least one image of the surface to be inspected is captured and preprocessed, it is used as input data for a trained model. Specifically, a defined set of optimized convolutional filters can be applied to the image to obtain a feature map, which is then fed into a trained pixel classification model 330. The defect regions obtained from this first step, as the output of the pixel classifier 330, can then be classified into defect type categories using a trained defect type classifier 520.

[0100] All these processing and classification steps for real-time inspection can be integrated into a single function 130, whereby the image is used as input to the function, and indications of defect regions along with associated defect types are received as output 630 of function 130. Intermediate steps (such as image preprocessing, feature maps, or indications of defect pixels from the first classification stage) may or may not be given as separate outputs, which can then be stored, displayed, or communicated to another unit. It will be understood that "real-time inspection" does not necessarily mean that image capture must be performed at a certain time or in any temporal relationship with the classification steps; it is also possible to capture and store surface images and perform further preprocessing and classification steps independently at a later time.

[0101] In some cases, changes in surface features over time are of interest. For example, EUV boxes are typically inspected periodically (e.g., daily or weekly) to ensure their integrity. Pixel classification and defect type classification, as described above, can be performed, and the results of the inspection and classification can be stored. In a subsequent surface inspection, the same surface can be imaged using the same settings and parameters for image capture and processing, and the new results of pixel classification and / or defect type classification can be compared with one or more previous classification results, allowing any changes in defects to be identified.

[0102] The data 630 obtained as a result of inspection 130 may include parameters of the defect region, such as the coordinates of the defect profile in a defined reference coordinate system, and information about the defect type obtained from the second classification stage. The stored data may also include one or more of the following: raw or partially processed image data; defect parameters (such as size, contrast level, isoaxiality, information about adjacent defects (clusters, distance to other defects), image data of the defect region cut out only from the background, parameters of the filter and classification model, and parameters of the transformations performed for image registration. Data comparisons used to determine defect variations may include all or only partially stored data. For example, if a defect at a certain location on the box is known and has significantly increased in size during the next surface inspection scan, this variation may be labeled, stored, and / or indicated to the user as severe surface degradation.

[0103] If a defect was detected in a previous surface inspection run, rather than at a later time, a notification can be stored or the user can be instructed to notify. At least a portion of the inspection process from imaging to classification can be repeated, optionally using different parameters (e.g., downsampling without image data), to determine whether the discovery was an error or that the defect no longer exists. In other cases, surface inspection can be performed, for example, before and after the box cleaning process; in this case, it is expected that at least some defects (such as objects on the surface) will not appear in subsequent scans. Defect identification can then be used to check the success of the cleaning process.

[0104] Any findings from the surface inspection run can be further analyzed, for example, by determining the total number of critical defects or a given defect type.

[0105] Since pixel classification models are primarily trained based on background regions in an image, these will be similar across different types of boxes, but with the same material and properties. In some cases, pixel classifiers can be used without retraining the model if small changes are made to the mask boxes, or different boxes are used in a setup with different types of masks (if the relevant features are the same). Test data as previously described (e.g., simulated defect pixels on a tiled background image), along with confusion matrices, can be used to validate the quality of existing trained classification models on another surface or object. A standalone defect type classification model can also be at least partially applied to other surfaces and surface regions, as it is trained based on parameters of the defect region without a background.

[0106] A surface inspection system or program module for surface inspection may include only machine learning models for two classification stages, or it may include additional classification methods for any stage. If more than one classification method is available for a given classification stage, the selection of the classification method can be performed automatically, for example, based on characteristics of the surface to be inspected or the imaging data, or it may be at least partially based on user selection. As an example, the system may include a trained machine learning model for pixel classification and a hysteresis thresholding method as described above. For the defect type classification stage, another machine learning model with one or more classifiers and a thresholding classification method based on defect characteristics may exist in the system. The user can then be provided with a query to select one of the available methods for each stage, or to select one of several predefined patterns corresponding to a specific combination of classification methods for each stage.

[0107] If a fully trained model is not yet available, for example due to a new box design or material, a parameter-based approach (such as a hysteresis threshold) can be chosen for all stages. Optionally, the classification data obtained in this way can then be used as labeled training or testing data for training a machine learning model on this box design. Optionally, the user can be informed of the missing trained model and can then initialize the training stages.

[0108] User input 620 can also be used during real-time classification to relabel defect data and thus improve the model. For example, if the user disagrees with the classification output, user input 620 indicating the correct defect type or defect image unit label can be obtained and used to relabel the corresponding defect image. The relabeled defects can then be used to retrain one or more machine learning classifiers.

[0109] For each region of the surface to be inspected with different characteristics, a separate library, a separate background image, a separate trained model, a separate filter set, or different thresholds and parameters (e.g., for hysteresis thresholding) can be identified and used. For example, for combining... Figure 1 For each of the different regions described in b (e.g., flat surface, edge, gas filter region), the steps described above for obtaining the final filter and model can be run. It is also possible that some or all parameters and models are determined to be similar or identical for certain regions. Optionally, some steps of the described method can be limited to certain regions of the surface to be inspected, or can be replaced by other steps; for example, different preprocessing steps can be performed based on the selected surface regions.

[0110] It should be understood that the above methods can be implemented in software, i.e., program code that can be executed on a suitable processing unit. Any processing unit with sufficient processing power to perform the desired steps can be used in this process, such as a microcontroller, microprocessor, single-core or multi-core processor, embedded system, or any other computing device capable of executing software commands. It is not required that all steps and sub-steps of the method be performed by the same unit or implemented in the same software program module; for example, the construction and training of the classifier model and the optimization of the filter set can be separated from the program module used for actual defect identification, and / or the preprocessing of image data can be at least partially separated from any further image processing. Data sets, parameters, image data, models, and / or executable program code can be stored in volatile and / or non-volatile storage elements, such as hard disks, flash memory, and others. Furthermore, any data, parameters, models, image data, and / or program code can be transmitted using wireless or wired communication channels between two or more local or remote processing units, for example, from a camera unit to the central control / processing unit of the inspection system, or from / to a remote server. Supporting software tools, such as machine vision software (e.g., HALCON) or various integrated development environments (IDEs), can be used to implement some parts of the method.

Claims

1. A method for constructing a model (130) for defect identification on a surface to be inspected, wherein, The model includes a first supervised machine learning classifier configured to classify image units of an image into at least two categories, the categories including at least a defective image unit category and a defect-free image unit category; The method includes: Definition (420) defines a plurality of convolutional filter sets (425a, 425b, 425c, 425d) for feature extraction from an image, wherein each of the plurality of convolutional filter sets (425a, 425b, 425c, 425d) includes one or more convolutional filters defined by a set of filter parameters, and each convolutional filter set differs from the other convolutional filter set in at least one filter parameter; The convolutional filter of each of the plurality of convolutional filter sets (425a, 425b, 425c, 425d) is applied to one or more labeled reference images on the (430) surface to obtain a plurality of feature maps, each feature map corresponding to one of the plurality of convolutional filter sets (425a, 425b, 425c, 425d); Multiple untrained copies of the first machine learning classifier are trained using the multiple sets of feature maps as training sets to obtain (440) multiple trained first machine learning classifiers (445a, 445b, 445c, 445d), wherein each copy of the first machine learning classifier is provided with one set of feature maps from the multiple sets of feature maps as input, such that each of the trained first machine learning classifiers is associated with one of the multiple sets of convolutional filter sets (425a, 425b, 425c, 425d); For each of the trained first machine learning classifiers (445a, 445b, 445c, 445d), an input test set is provided to the trained classifier, wherein the input test set comprises a set of test feature maps obtained by applying the set of convolutional filters associated with the trained classifier to at least one predefined test image and obtaining (450) a quality parameter indicating the quality of the classification result of each trained machine learning classifier; Based on the obtained quality parameters associated with each of the plurality of convolutional filter sets, an optimized convolutional filter set (465) is determined (460) from the plurality of convolutional filter sets (425a, 425b, 425c, 425d). The optimized set of convolutional filters is applied to one or more labeled training images on the (470) surface to obtain a set of optimized training feature maps; and The first machine learning classifier copy is trained (380) using the set of optimized training feature maps as training data to obtain the first trained machine learning classifier.

2. The method according to claim 1, wherein, The filter parameters used to define the convolutional filter include at least one of the following: filter type, kernel size, stride, padding size, activation flag of the convolutional filter, and number of convolutional filters in a group.

3. The method according to claim 1 or 2, wherein, The plurality of convolutional filter sets (425a, 425b, 425c, 425d) are determined by randomly assigning values ​​from a given range to one or more of the filter parameters of each convolutional filter set.

4. The method according to any one of the preceding claims, wherein, The test image was generated in the following way: Identify at least one region of a reference image that is marked as defect-free; A background image is constructed by tiling multiple copies of the defect-free region of the reference image; as well as One or more simulated defect image regions (310) are added to the background image, wherein the defect regions are defined by predetermined and / or random image parameters of each image unit of the simulated defect regions.

5. The method according to any one of the preceding claims, in, The model (130) includes at least a second machine learning classifier configured to classify defective image regions in an image into one of a plurality of defect type categories; The method further includes: Display multiple defect image regions to the user, each defect image region including one or more defect image units; Based on user input, at least one label indicating the defect type category is assigned to each of the defect image regions (530); and The at least one second machine learning classifier is trained using the labeled defective image region as training data (510) to obtain a trained second machine learning classifier (520).

6. A method for inspecting surface defects, the method comprising: Obtain at least one inspection image (600) of the surface to be inspected; A predefined set of optimized convolutional filters (465) is applied to the at least one inspection image to obtain a set of inspection image feature maps; The set of inspected image feature maps is provided as input to the model, which includes at least a first trained machine learning classifier (330). The classification result of the image unit of the at least one image is obtained as the output of the model (630), and the classification result indicates at least the image unit of the image that is classified as a defective image unit.

7. The method according to claim 6, further comprising: The parameters of the defective image unit are provided as input to at least one second trained machine learning classifier (520) of the model, wherein the at least one second trained machine learning classifier is configured to classify a defective image region in an image that includes one or more defective image units into one of a plurality of defect type categories; A classification result (630) is obtained for the defective image region, the classification result indicating the associated defect type category for each defective image unit as the output of the at least one second trained machine learning classifier.

8. The method according to claim 6 or 7, further comprising: Provide the user with a visual display of at least one defective image unit in the inspected image.

9. The method according to any one of claims 6 to 8, further comprising: Retrieve stored parameters of defect image units and / or defect type categories from at least one earlier inspection image obtained from a defect image region captured at an earlier time point; The retrieval parameters of the defective regions and / or defective image units in the earlier examined images are compared with the obtained classification results; as well as When the difference between the defective region and / or the defective image unit exceeds a predefined threshold in the comparison, the defective image region is defined as a changed defect.

10. The method according to any one of claims 6 to 9, further comprising: Provide at least a first parameter-based classifier, the first parameter-based classifier being configured to classify the image units of an image into defective image units and defect-free image units based on a predefined parameter threshold for the image unit; Provide at least a second parameter-based classifier, the second parameter-based classifier being configured to classify a defective image region in an image, which includes one or more defective image units, into one of a plurality of defect type categories based on a predefined parameter threshold for the defective image region; A first classifier is selected from the first trained machine learning classifier and the first parameter-based classifier to classify defective image units, and the selected classifier is used to determine the defective image units. as well as Select at least one second classifier from the at least one second trained machine learning classifier and the second parameter-based classifier, and use the selected second classifier to classify the defect region into a defect type category; The classifier is selected based on one of the following: user input, image characteristics, or surface characteristics of the imaging surface.

11. The method according to claim 5 or any one of claims 7 to 10, wherein, The defect type category is defined based on one or more parameters of the defective image region, the parameters including at least one of the following: an indication of whether the defect is acceptable; the size of the defective image region; the contrast level of the defective image region; a parameter indicating the isoaxiality of the defective image region; the relative size of the defective image region; and the defect type.

12. The method according to claim 5 or any one of claims 7 to 11, wherein, The second machine learning classifier (520) includes at least a continuous first sub-classifier and one or more second sub-classifiers, wherein the first sub-classifier is configured to classify defective image regions as acceptable defects or unacceptable defects, and wherein the one or more second sub-classifiers are configured to classify the acceptable defects or the unacceptable defects as defect type categories.

13. The method according to any one of the preceding claims further comprises preprocessing the image before applying the set of convolutional filters, wherein, The image is at least one of a training image, a reference image, a test image, or an inspection image. The preprocessing includes at least one of the following: Enhance the contrast value in the image; apply coordinate transformation to the image registration based on the reference coordinates; apply an image filter to the image.

14. The method according to claim 13, wherein, The preprocessing includes: Obtain dark-field and bright-field images taken using the same field of view. A contrast-enhanced image is obtained by enhancing the local contrast in each of the dark-field image and the bright-field image; The defect-free image background in each of the dark field image and the bright field image is determined by calculating the average image value; A difference image for each bright field and dark field is obtained by calculating the difference between the image background and each of the contrast-enhanced images; and The difference image is combined by selecting the maximum value for each image unit from the bright field image and the dark field image.

15. The method according to any one of the preceding claims, wherein, For each image region corresponding to a surface region with specified different surface properties, a separate model is trained.

16. The method according to any one of the preceding claims further comprises: A continuous image region is defined as a defective image region, which includes multiple defective image units located adjacent to each other or at least at a predefined maximum distance from each other.

17. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of any of the methods of the preceding claims.

Images