Method for defect inspection of surfaces

Abstract

A model for defect recognition on a surface under inspection comprises a first supervised machine learning classifier configured for classifying image units of an image into at least defective and defect-free image units; multiple sets of convolution filters perform feature extraction, wherein each such set comprises one or more convolution filters defined by a set of individual filter parameters; said convolution filters are applied to one or more labelled reference images of a surface to obtain multiple groups of feature maps; multiple trained first machine learning classifiers are obtained using said feature maps; each trained first machine learning classifier is provided with an input test set and a respective quality parameter is obtained; an optimized set of convolution filters is applied to one or more labelled training images in order to train a copy of said first machine learning classifier.

Description

[0001] Method for defect inspection of surfaces

[0002] Field of the invention

[0003] The present invention relates to a method for inspecting surfaces for defects, in particular a method for inspecting lithography reticle pods which are used for storing reticles or photomasks in lithography.

[0004] Background

[0005] Lithography for semi-conductor manufacturing is constantly changing to shorter wavelengths in order to be able to achieve smaller features. At present, wavelengths in the extreme ultraviolet range in the range of 125 down to 10 nm, so called EUV, are increasingly being implemented. EUV lithography is extremely sensitive to contamination and defects, in particular any defects or contaminations of the reticles (photomasks) used. For storing, handling and transporting EUV reticles, different kinds of protective housings and storage containers have been developed. A widely used solution provides a reticle dual pod carrier comprising two nested housing elements forming an inner pod, EIP (EUV inner pod) and an outer pod, EOP (EUV outer pod). A reticle within a dual pod is thus protected against contamination from the outside atmosphere. In order to ensure safe and clean handling of reticles at all times, EUV pods have to be inspected for contamination as well as defects such as scratches or material fatigue. Pod inspection may be performed e.g. at regular intervals, or after a certain amount of handling operations or cycles. However, pod materials may be made from various materials having different surface characteristics such as highly reflective and matte surfaces. A surface under inspection will typically have a large number of very small defects, i.e. potentially thousands of defects in a size range of micrometres, while the highly reflective surfaces may lead to abundant noise in surface images. Therefore, very fast recognition of defects with high accuracy is required.

[0006] Summary

[0007] A method is proposed for building a model for defect recognition on a surface under inspection, as well as a method for inspecting a surface for defects as defined by the appended independent claims. Further exemplary embodiments are defined by the dependent claims. The present disclosure relates to a first method for building a model for defect recognition on a surface under inspection, wherein the model comprises a first supervised machine learning classifier configured for classifying image units of an image into at least two classes, said classes comprising at least a class for defect image units and a class for defect-free image units. The method comprises defining multiple sets of convolution filters for feature extraction from an image, wherein each set of the multiple sets of convolution filters comprises one or more convolution filters defined by a set of filter parameters, each set of convolution filters differing from another set of convolution filters in at least one filter parameter; applying said convolution filters of each one of said multiple sets of convolution filters to one or more labelled reference images of a surface to obtain multiple groups of feature maps, each group of feature maps corresponding to one out of said multiple sets of convolution filters; obtaining multiple trained first machine learning classifiers by training multiple untrained copies of said first machine learning classifier using said multiple groups of feature maps as training set, wherein each copy of said first machine learning classifiers is provided with one group of feature maps out of said multiple groups as input, such that each trained first machine learning classifier is associated with one out of said multiple sets of convolution filters; for each one of said trained first machine learning classifiers, providing said trained classifier with an input test set, wherein said input test set comprises a group of test feature maps obtained by applying said set of convolution filters associated with said trained classifier to at least one predefined test image, and obtaining a quality parameter indicating a quality of the classification result for each trained machine learning classifier; determining an optimized set of convolution filters out of said multiple sets of convolution filters based on said obtained quality parameters associated with each of said sets of convolution filters; applying said optimized set of convolution filters to one or more labelled training images of a surface to obtain a group of optimized training feature maps; and training a copy of said first machine classifier using said group of optimized training feature maps as training data. Image units may for example be defined as image pixels in the usual way.

[0008] In this way, the step of optimizing feature extraction by convolution filters may be separated from the classification step, thus resulting in a less complex machine learning classifier. For example, the machine learning classifier may be implemented using a multilayer perceptron or a Gaussian model mixture. As a result, model training and in particular classification during a live surface inspection using such a model requires considerably less processing power and is faster than prior art defect recognition.

[0009] Filter parameters for defining the convolution filters may comprise one or more of a filter type, a kernel size, a stride length, a padding size, an activation flag for a convolution filter, or a number of convolution filters within one set. Using parametrized filters in this way allows for testing any desired number of different sets of convolution filters in order to find a filter set that brings optimized classification results. The multiple sets of convolution filters may for example be determined by randomly assigning values from a given range for one or more of said filter parameters for each set of convolution filters.

[0010] The test images for testing the result of model training may be generated by determining at least one region of a reference image which is labelled as defect-free, building a background image by tiling multiple copies of said defect-free region of said reference image; and adding one or more simulated defect image regions onto said background image, wherein said defect regions are defined by predetermined and / or random image parameters for each image unit of a simulated defect region. For example, a test image may comprise multiple simulated defects of different size, contrast, shape and location, and a trained model may be tested on a single image for determining model and filter quality.

[0011] In a further variation of a method for building a model, the model may comprise at least a second machine learning classifier configured for classifying defect image regions in an image into one of a plurality of defect type classes. The method may then further comprise displaying a plurality of defect image regions to a user, each defect image region comprising one or more defect image units; assigning at least one label indicating a defect type class to each of said defect image regions based on a user input; and training said at least one second machine learning classifier using said labelled defect image regions as training data. That is, the classifier for recognition of defects in an image is trained separately from a second classifier for determining a defect type, which provides a faster and more robust recognition in the finalized model. In addition or as an alternative to the manual labelling of training data sets, defect regions may at least in part be labelled automatically, for example based on threshold classifying methods.

[0012] Further, a method for inspecting a surface for defects is proposed, wherein at least one inspection image of a surface under inspection is obtained, and a predefined optimized set of convolution filters is applied to said at least one inspection image to obtain a group of inspection image feature maps. The obtained group of inspection image feature maps is then used as input to a model which comprises at least a first trained machine learning classifier, and as output of the model, one obtains a classification result for image units of said at least one image. The classification result indicates at least image units of the image which are classified as defect image units. In this case, the feature extraction using optimized convolution filters is not included in the backpropagating machine learning network, but is performed as a separate step, thus accelerating both the training and the classification process of the network.

[0013] The method may further comprise the steps of providing parameters of said defect image units as an input to at least one second trained machine learning classifier of said model, wherein said at least one second trained machine learning classifier is configured for classifying defect image regions including one or more defect image unit in an image into one of a plurality of defect type classes; obtaining a classification result for said defect image regions, said classification result indicating, for each defect image unit, an associated defect type class as an output of said at least one second trained machine learning classifier. As the defect type classifier is separated (both in training and in classification) from the actual defect pixel detection in an image, the classifier and the training database of labelled defect image regions may be reused at least in part for other backgrounds, other surfaces or other devices.

[0014] The output may be used in various further steps; for example, a method may include a visual display of defect image units in said at least one inspection image to a user. In this way, a user (e.g. an operator of the inspection facility) may easily identify defects in the surface image, may correct or re-examine any potential recognition errors or may initiate further steps for handling the defects.

[0015] It is also possible to retrieve stored parameters of defect image units and / or defect type classes of defect image regions obtained from at least one earlier inspection image that has been captured at an earlier point in time, to compare the retrieved parameters of defect regions and / or defect image units of the earlier inspection image to the obtained classification results in the current inspection image, and, if a difference in defect regions and / or defect image units in said comparison exceeds a predefined threshold, to define said defect image region as a changed defect. Using a comparison to previous image inspection facilitates a live-cycle monitoring of a surface, e.g. to observe changes in defects, to recognize critical deterioration of a component, or to check the success of measures such as a cleaning process of the surface.

[0016] A method for inspecting a surface may also include the steps of providing at least a first parameter-based classifier configured for classifying image units of an image into defect image units and defect-free image units based on predefined parameter thresholds for said image units; providing at least a second parameter-based classifier configured for classifying defect image regions including one or more defect image unit in an image into one of a plurality of defect type classes, said classifying being based on predefined parameter thresholds for said defect image regions; selecting a first classifier out of said first trained machine learning classifier and said first parameter-based classifier for classifying defect image units, and determining defect image units using the selected classifier; and selecting at least a second classifier out of said at least one second trained machine learning classifier and said second parameter-based classifier, and classifying defect regions into defect type classes using said selected second classifier; wherein said selecting of classifiers is based on one of: a user input, an image characteristic, a surface characteristic of the imaged surface. In this way, a user - or an automated process - may choose between the more computation-intensive machine learning classification and simpler parameter-based methods. It is also conceivable that in some situations parameter-based methods provide better results for defect recognition, e.g. on certain surface structures or when a fully trained model is not yet available.

[0017] The defect type classes may be defined based on one or more parameters of said defect image region, wherein the parameters comprise at least one of an indication of whether the defect is acceptable; a size of the defect image region; a contrast level of the defect image region; a parameter indicating an anisometry of the defect image region; a relative dimension of the defect image region; a defect type. These classes may be used for machine learning classification and parameter-based classification, or different parameters may be used for each classifier. Using such parameters allows to differentiate between e.g. scratches, large low- contrast defects which might indicate a surface contamination, different shapes of defects which might be recognized as either shallow defects or critical volume defects, and others. The defect types and parameters as well as their interpretation may vary for different surfaces, different imaging techniques, or different classification models.

[0018] The second machine learning classifier which classifies regions into defect types may be implemented in various different ways; for example, this second classifier may comprise at least a first sub-classifier and one or more second sub-classifiers in succession, wherein the first sub classifier is configured for classifying defect image regions as either an acceptable defect or a non-acceptable defect, and wherein the one or more second sub-classifiers are configured for classifying said acceptable defects or said non-acceptable defect into defect type classes.

[0019] The images used in any phase, i.e. for example a training image, a reference image a test image, or an inspection image, may be preprocessed in various way before the set of convolution filters is applied. For example, preprocessing images may comprise one or more of the following steps: enhancing a contrast value in said image; applying a coordinate transformation for image registration based on reference coordinates; applying an image filter to said image. Those and other image processing steps may be used to enhance the recognition of potential defects on the surface or to decrease undesired noise in the image which might lead to pseudo defects in the result. Also, images may for example be down-sampled, both to reduce data size and to improve the result of the recognition. Image registration and transformation may be used for ensuring a consistent coordinate reference or to combine several images.

[0020] Several preprocessing may be combined or performed in succession. Also, several images may be processed together to form the final reference image, inspection image or training image, for example by building difference images or convolutions of images. It is also possible to combine several images of a surface which have been captured using different cameras, different fields of views, different areas of a surface, different imaging techniques, different lighting conditions, and / or different wavelengths into a single image.

[0021] As an example, the preprocessing may comprise obtaining a darkfield and a brightfield image captured with a same field of view; obtaining contrast enhanced images by enhancing local contrast in each of the darkfield and brightfield images; determining an image background free of defects in each of the darkfield and brightfield images by calculating a mean image value; obtaining difference images for each brightfield and darkfield by calculating a difference of said image background and each of said contrast enhanced images; and combining said difference images by selecting a maximum for each image unit from said brightfield and darkfield images.

[0022] For each image region corresponding to a surface region having specified different surface characteristics, a separate model may be trained, including separate trained classifiers and separate optimized filter sets. The untrained machine learning classifier itself may be identical or different for different surface regions. Similarly, a separated trained model for each surface region may be provided in a final trained inspection system. For example, surfaces may include regions of different materials or different surface structure, different planes of focus, areas for which imaging is subject to geometrical limitations, or others.

[0023] The described methods may further comprise defining contiguous image regions which include multiple defect image units located adjacent to each other or at least at a predefined maximum distance to each other as defect image regions. For example, image regions may be determined by all defect image units (pixels) which have less than a predefined distance to the next defect image unit. It is also possible to differentiate between image units directly adjacent to another defect image unit, and image units which are formed by clusters of separate defect image units.

[0024] Various functions, steps and methods described herein can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. A computer program may comprise one or more software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code. The phrase "computer readable program code" may include any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" may include any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

[0025] The present disclosure covers several components which can be used in conjunction or in combination with one another, or can operate as standalone schemes.

[0026] Brief description of the drawings

[0027] In the following, exemplary embodiments will be described in more detail with reference to the accompanying drawings, where

[0028] Fig. 1 is an overview of various software functions performing steps of the methods according to exemplary embodiments,

[0029] Fig. 2a depicts an exemplary setup for imaging in a pod inspection system,

[0030] Fig. 2b shows a side view of an exemplary setup for imaging in a pod inspection system;

[0031] Fig. 2c shows a plan view of an exemplary EUV reticle inner pod,

[0032] Fig.3 shows a schematic flow diagram for an exemplary training phase of a pixel defect classifier,

[0033] Fig. 4 is a flow diagram showing steps of a convolution filter optimization for feature extraction;

[0034] Fig. 5 shows a schematic flow diagram for an exemplary training phase of a defect type classifier; and

[0035] Fig. 6 is a flow diagram of a surface inspection surface using a trained model. Detailed Description

[0036] While the following description for several embodiments refers to an inner reticle pod for a lithographic reticle or photomask as an example, it should be noted that the invention is generally applicable to the defect inspection of any surface, in particular inner or outer reticle pods, reticles and photomasks of any kind. All these objects share similar challenges for surface inspection, which are especially present in EUV reticle pods due to the materials used.

[0037] For optimum recognition of surface defects, it is proposed to use a two-stage classification process. In a first stage, which will be referred to as a pixel classification stage in the following description, an image may be processed to determine all regions or pixels of the image which may be identified as part of a defect of any kind. In a second stage, which will be referred to as a defect classification stage or defect classification module in the following description, these regions may be classified into different groups or types of defects, such as scratches, objects on the surface, water marks, surface issues or others. The defects may also be classified by certain characteristics, such as size, width and / or contrast. In the following, any kind of surface change to be detected is denoted as a defect, that is, e.g. damages to the surface or bulk material which show on the surface as well as contaminations which are directly visible or which change the optical properties of the surface under inspection. Common defects of interest on a pod surface may usually have a size in the range of only a few micrometers up to several hundred micrometers. Of course, a defect or undesired surface feature may generally have any size, and the defect dimensions which may detected using the below methods may depend on the field of view as well as the image resolution of the imaging sensor used.

[0038] In these two stages of defect recognition, various methods of image processing may be employed. Machine learning - in particular feed-forward neural networks such as multilayer perceptrons - may be employed as a mechanism of classification. The applied image filters for feature extraction and the training data used for the classifier model are therefore of particular importance. A machine learning model in this context involves providing a machine learning algorithm with training data to generate a model artifact as an output of the training process, wherein the model artifact may be a fully or partially trained machine learning model, a file storing parameters of the model such as the trained weights of a model, or others. Just as for the live defect recognition, the (at least two) machine learning models for pixel classification and defect type classification are built and trained separately. The method may offer further options for classifying pixels or defects besides machine learning, such as simple thresholding, which will also be explained in more detail below.

[0039] Figure 1 shows an exemplary overview of various software functions or modules which may be used to implement steps of the methods described below. The first and second module 110 and 120 are used for building a surface inspection model and may be performed offline, i.e. outside of live surface inspection. A first module 110 may be used to implement a process for building a pixel defect classification model that classifies pixels (or image units in general) of an image into defect and non-defect pixels. A second module 120 may implement another process for building a defect type classifier, which classifies pixels or regions determined as defect pixels into defined pixel type classes. Using the surface inspection model built in these steps as a combination of the pixel defect classifier and the defect type classifier, live defect recognition 150 implemented by a surface inspection module 130 is facilitated, and further applications such as a module 140 for identifying defect changes (e.g. for live cycle monitoring 160) may be implemented separately using the result of surface inspection module 130.

[0040] In the following, it is assumed that surface images of the surface under inspection are captured, preferably a plan view of the surface or at least of parts of the surface. In the same way, such surface images may be obtained for building the model and defining various parameters of the inspection method. Preferably, the imaging parameters of images used for building and training classification models should be essentially the same as the imaging parameters used in the later actual surface inspection procedure.

[0041] Exemplary methods of obtaining images for the classification and inspection methods described in here will be detailed below. For example, images may be obtained using illumination of one or more wavelengths or a continuous spectrum of wavelengths, darkfield and / or brightfield illumination, different types of imaging sensors and methods such as line scan imaging or area scan imaging.

[0042] Figure 2a depicts elements of an exemplary setup for obtaining surface images of a EUV pod for defect inspection. An inspection station 210 may be arranged to handle EUV pods 20 in a partially or fully automated process, e.g. by providing a loading module 222, 224 which is adapted to open an outer EUV pod 201 in order to access an inner EUV pod 200. The same or further modules may be provided for handling the EUV pod components, such as separating a base member of a pod 202 from a cover member 204, removing a reticle stored within a pod 20, rotating or otherwise aligning the pod, or transferring the reticle pod or reticle member to the place of imaging 262, 264, e.g. using a handling robot 242. For the imaging step, a pod or pod member may e.g. be placed in a receptacle or in a frame on a holding surface which optionally allows precise positioning of the placed part. Figure 2b shows a schematic view of an exemplary inspection unit 262 for imaging an EUV pod member 202, 204 that may be held in an insert or component holder 270. For example, a multi-axis table 275 may be used which may facilitate translation and / or rotation in several axial directions to allow for precise positioning of a pod member relative to one or more imaging sensors and for imaging all surfaces of an object under inspection. Positioning the pod may be performed automatically or based on user input. Such a table 275 may also be used to bring the prearranged surface under inspection into focus for one or several imaging devices, e.g. by adjusting a position along a z-axis for a camera positioned above the surface.

[0043] The imaging system 262, 264 may comprise one or more imaging devices 262a, 262b such as a line scan or area scan camera. Line scan cameras usually comprise an imaging sensor having a single line (or a narrow array of very few lines) of sensor elements, such that one linear section may be imaged at a time. To obtain an image of a two-dimensional area by scanning, either the surface to be captured or the line scan camera needs to be moved, e.g. using a table 275 as described above. The movement should be synchronized with the imaging process, so that an image of a two-dimensional area is generated as a sequence of line images. It will be understood that positioning elements as described above which may be used to bring a pod surface into a desired position for imaging may also be used to provide a scanning motion in a predefined direction, in combination with a line scan camera. An area scan camera, in contrast comprises an array of sensor elements which is adapted to image a two-dimensional area at once. While it is possible to use a sensor of sufficient size and resolution to provide an image of the complete surface under inspection, it is also conceivable to move the camera and / or the object to image different surface areas, and to optionally combine these into a single image if required. If stitching of images is desired, an overlap area of suitable size (e.g. 100 pixels) may be considered for the imaging positions. The amount of overlap may vary and may depend on factors such as image sensor size and positioning accuracy.

[0044] The imaging system may have a single camera or may also comprise several line scan and / or area scan cameras, for example cameras 262a, 262b having different fields of view onto the object 202, 204 under inspection and / or different optical properties such as wavelength sensitivity or resolution. Images from several cameras may then be processed separately or in combination. If several cameras are used for imaging, these may either be provided in a single place or in separate imaging stations, such that the reticle pod or pod component may be moved from one imaging station to another and different types of images may be captured in succession. Especially in reticle pod imaging or similar surfaces having a wide range of different properties, it may be suitable to use line scan imaging for some parts of the surface and area scan imaging for others. Figure 2c shows a plan view image of an EUV inner pod cover member 204 by way of example. It is understood that other pod designs and other surfaces in general may be used and inspected using the methods described in here. The surface under inspection will usually, as shown in the Figure, have several edges which are elevated or recessed from the level plane and may have areas having very different optical properties, such as matte opaque surfaces, structured surfaces such as filter areas, or highly reflective surfaces. Therefore, some areas of the surface may be out of focus in a plan view, out of the field of view during imaging, or may be essentially invisible or without sufficient contrast for a certain type of imaging sensor or method. For example, while most surfaces shown in Fig. 2c may be inspected using line scan and / or area scan imaging, other regions such as region 204a, which constitutes a handling ridge for gripping or positioning the pod cover member, cannot readily be scanned in this way. Other areas, such as edge area 204b may be inspected using line scan or area scan imaging, but may only provide inspection images of limited quality. Further, areas 204c in this example, which comprise filters and / or openings for gas exchange between the inside of the inner pod and the volume between inner and outer pod, may expediently be inspected by means of an area scan, and area 204d, which covers the largest part of the cover surface, may expediently be inspected by means of a line scan.

[0045] Furthermore, usual elements of an imaging system such as one or more lighting devices (not shown) may be provided in the inspection system. Lighting may be provided with one or more wavelengths of light in the visible or also non-visible spectrum, and lighting devices (e.g. lamps, lasers, optionally provided with optical filters) may be arranged in any desired way around the object to be imaged. Illumination of the surface under inspection may for example comprise darkfield illumination, where essentially only light scattered from the object or surface is detected on an imaging sensor, and / or brightfield illumination, where a light source is arranged such that light is transmitted through the object (e.g. for semiopaque materials) and the attenuated light is detected at the imaging sensor. Lighting conditions may have significant influence on the detectability of surface defects in an image. For example, by means of darkfield imaging, scratches on the surface of an EUV pod component are reliably detectable, while contamination particles on a surface of an EUV pod component are readily detectable by bright light imaging. By combining these two imaging techniques, a broad spectrum of surface defects and contaminations can be detected. It is also possible to use both darkfield and brightfield illumination at the same time during imaging, e.g. by using light of different wavelengths / colors, which may be separated by image processing after detection or may be detected separately using detectors of suitable wavelength sensitivity. Preferably, the components of the imaging system such as imaging sensors or cameras, lighting devices and similar parts may be provided in an at least partially fixed manner, such as fixedly positioned within a frame or housing, in order to ensure consistent imaging and lighting conditions during measurements.

[0046] The data captured by the one or more imaging devices 262a, 262b, 264 may be obtained by a suitable processing unit 202 such as one or more computers, a central controller, several microcontrollers or microprocessors provided for different functions, and may be stored temporarily or permanently, processed in any way desired, and / or forwarded to other elements and devices. Data from other parts of the imaging system, such as from a positioning system, a lighting arrangement, or various sensor elements may be processed and or / stored together with image data. The same or another processing unit may be configured for controlling elements such as the moving table 275, pod handler 242, loading modules 222, 224 or any other controllable element of the system.

[0047] The system may also be a combined system for inspecting both the reticle pod and the reticle itself, for inspecting inner pods and / or outer pods, and / or for cleaning the inspected pod surfaces using various cleaning processes if contaminations have been found. Corresponding system parts not shown or described here may therefore be present, and / or parts described here may be omitted from an inspection system which employs the method below.

[0048] It will be understood that the setup described above is only mentioned by way of example, and that a wide variety of other imaging systems, devices and methods may be used for obtaining surface images for defect recognition.

[0049] Surface images of the surface to be inspected may in a first step be processed and used for preparing an inspection model and defining various parameters of the methods. However, images used in actual surface inspection should be processed in essentially the same way as any image data used for model building and model training to ensure correct classification.

[0050] An image registration process may be used in order to ensure that a consistent system of reference coordinates is used for surface categorization, defect recognition, defect labelling, and any other steps which require information on any characteristic of the surface relative to other elements. Also, the image registration process may be used to combine several image data sets e.g. from different sensors, different fields of view or imaging depths into a single image. Reference data for image registration may be provided e.g. as a CAD (computer aided design) image, which is usually readily available from manufacturing of a pod. It is understood that any other image format or other data elements for achieving image registration may be used as well. For example, instead of full reference images, coordinates of reference points may be used.

[0051] Characteristic structures of the surface under inspection such as holes, edges, fasteners having a known shape or any other structure may be used for image registration and for determining any transformation required.

[0052] As another part of preprocessing image data, various methods for enhancing image contrast may be applied, e.g. using a lookup table for mapping pixel values to enhanced values. If, as an example, images of a surface region have been captured as both brightfield and darkfield images, these can be used to increase defect contrast by combination. To that end, contrast may be enhanced separately in a brightfield and a corresponding darkfield image, and then the contrast enhanced images may be combined into an enhanced combined image. Optionally, further steps such as inverting one of the darkfield and brightfield images may be taken. This enhanced combined image may then be used as input for any further steps.

[0053] In another variation, the images may be combined into a defect image by first determining a background of both images by calculating a mean, and then obtaining separate defect images for both darkfield and brightfield images by calculating a difference of background and enhanced images. Then, the maximum of the defect darkfield and brightfield images may be determined and used to build a combined image showing maximum defect contrast.

[0054] Other steps for changing e.g. contrast or colors in the images may be used alternatively or additionally. It is also possible to use more than two different images for producing combined defect images, such as images in different wavelengths. Generally, the raw images may be processed in a way that produces enhanced images which emphasize defects and therefore allow better recognition. Optimizing steps and methods for preprocessing, such as determining values for lookup tables or combinations of certain images, may be based on the final result of a trial defect classification.

[0055] As a further optional step, image resolution may be decreased to improve performance of the detection algorithms. Care should be taken that defect information is not lost due to the decreased resolution; on the other hand, decreasing resolution may also reduce pseudo-effects in the defect recognition and may therefore lead to better classification results. For example, a Gaussian Pyramid method may be applied to both brightfield and darkfield images, but interpolation parameters may be selected differently for brightfield and darkfield. Other downsampling and / or filtering methods may be employed similarly. All previous steps have only been described as an example process of obtaining surface images suitable for defect classification. It will be understood that other methods may be employed as well, that some steps may be omitted or altered, or other steps not described in here may be added. The following steps for recognizing and classifying surface defects in two stages may also be used in any methods using image data that has been preprocessed in a different way or not at all.

[0056] Any parameters used in image preprocessing and model building may be stored in a suitable way, such as in a data file, such that during the surface inspection process, the same parameters may be applied. Optionally, parameters may be amended by user input or by using values obtained in other methods.

[0057] The present method employs two stages of classification for detecting surface defects in an image. In a first stage, defect pixels (regions) on a surface may be identified, and in a second stage, these defect pixels may be classified into various types of defects, such as scratches, small defects, pseudo-defects, defect clusters or others. From this classification, any defects on a surface may be further analysed with regard to their impact on functionality of the object under inspection; for example, some defects may be present on a pod surface, but may not pose a risk for integrity of the pod, while others may lead to serious damage to the pod and therefore the encased reticle.

[0058] The aim of the first classification stage is therefore to detect defect regions in the image, that is, to determine which image subunits of the image are to be defined as “good” or without defects, and which are part of a defect region. Image subunits to be classified in this way may for example be image pixels as usually defined, or may be specified in any other way. In the following examples, pixels are used as basic subunits of an image. The number and size of pixels depends on the image resolution used and may vary, e.g. according to the desired sensitivity of defect recognition. Evidently, resolution may also be limited by the type of imaging sensor used. Since a surface under inspection will usually not show a homogenous image, but will have some inherent structure - both regular and / or irregular -, the challenge is to distinguish real defects from this structure and from any other surface characteristics such as reflections.

[0059] For different stages of model building (i.e. defining parameters, training classifiers, and other steps for obtaining a final classification model), reference images may be used. Figure 3 is a flow diagram showing exemplary steps for building and training a pixel defect classifier, which uses reference images 300 as an input. As an exemplary embodiment, reference images 300 generated from brightfield and darkfield images that have been processed as described to generate a maximum defect image above may be used as training data for a pixel classification model. The reference images may be selected such that they provide one or more defect free regions for all areas of interest on the surface. Usually, a single reference image or a low number of reference images may be sufficient in this stage.

[0060] In a first step, one or more background areas without any defects may therefore be defined and labelled on the reference image. Such areas may be selected manually by a user, that is, for example by a visual inspection of an image on a screen and marking or indicating defect-free areas using a user interface. It is also conceivable to preselect areas which may be used as background areas, e.g. based on contrast values or homogenous structures, and to offer these preselected areas to the users for selecting a background area without defects. Since the object under inspection may have very different surface regions with different textures and structures (such as matte or polished surfaces), one or more associated reference background images may be selected for each of these regions. For example, with an EUV pod surface as described, most of the surface will be flat surface regions without any disturbances. In addition, there are air filter areas with inhomogeneous appearance, structured areas such as steps and recesses of the pod surface or fastening elements, and others. Different parts may also be made from different materials. For each of these, at least one defect free background area is selected. The number of background areas used in this step is not limited in general, and more than one area may be used for each surface region of interest.

[0061] A minimum size may be defined for a defect-free background in order to ensure that any inherent structure of the surface is sufficiently reproduced in the selected area. For example, background tiles of 256x256 pixels may be selected. Then, a full-size background image may be generated by tiling, i.e. combining multiple copies of the selected background area together for obtaining a large background image of the required size for training.

[0062] A simple method of classifying pixels as ‘good’ or ‘defect’ pixels is using thresholds, e.g. threshold values for the grey values of an image. All pixels having grey values below a certain threshold may then be considered as ‘good’ pixels, while all pixels above the threshold may be considered ‘defect’ pixels, or vice versa. However, many of the defects will have poor contrast even in an enhanced image and would therefore not be detected by a simple threshold, while there is a large number of small defects and pseudo-defects or noise which should not be classified as defect.

[0063] To counter these issues, thresholding may be combined with different lower and upper thresholds in order to obtain a hysteresis threshold method. Let us consider that defects are assumed to show as light areas, i.e. having a large grey value. For a hysteresis threshold, all pixels in an input image having a grey value larger than or equal to an upper threshold “low” may immediately be accepted as clearly defect pixels. Conversely, all pixels having a grey value less than a lower threshold may be rejected immediately. When the upper threshold and the lower threshold are selected to be non-equal, any pixel falling in between these two threshold values may be considered potential defects. Additional checks may then be used to determine whether these potential defect pixels are ultimately to be defined as good or defect. For example, one additional check for a potential defect pixel may include determining whether it is connected to clearly defect pixels. A length threshold may be used in this check, such that it may be checked whether the potential defect pixel is connected to one or more clearly defect pixels by a path of potential defect pixels having a path length of equal or less the length threshold value. In this way, the surroundings of defect pixels are considered as well. These steps may be repeated until all pixels of an image (or an image region) to be inspected are classified. Again, thresholds may also be defined differently, e.g. such that any pixels having grey values below a lower threshold are defined as ‘defect’ while grey values above an upper threshold are defined as ‘good’.

[0064] The threshold parameters may be defined by using the defect-free background areas that have been defined in the reference image. Since these areas are labelled as defect-free, any image parameters in these areas should fall below the upper thresholds (or, vice versa, above thresholds if lower thresholds are given for a certain parameter). However, it is also possible to define parameters for a threshold model in other ways and independent of the formation of any reference images.

[0065] While this hysteresis threshold method allows for a simple definition of parameters and a simple classification, some types of defects (e.g. connected scratches or defect clusters) may not be detected, and with a large number of small defects, the check for connecting paths may become very time consuming.

[0066] A more sensitive option for classifying pixels into good or defect pixels / regions of pixels may therefore be implemented using a trained machine learning model. In a final surface inspection method, both methods may be employed in parallel or alternatively and may also be complemented with further classification methods. In the following, exemplary steps for defining and training such a machine learning model for pixel classification will be described in more detail. In contrast to thresholding methods which only classify based on pixel values, digital filters may be used such that information of neighbouring pixels is included in the classification. The filters may be applied by convolving the image with a filter function or kernel and by further operations, such as subtracting or adding original or convolved images in order to obtain the final filtered image. As a result of each convolution, a feature map is obtained, providing features for each image pixel. Using a set of convolution filters on an input image will result in the same amount of different feature maps. The feature maps obtained from convolution filters may be provided in a multi-channel image, with the number of channels corresponding to the number of filters applied.

[0067] Possible candidates for such filters are, for example, bandpass filters, Gaussian filters or various filters based on Gaussian filters, such as Difference of Gaussian (subtracting two versions of the image blurred with different Gaussian kernels), Laplace of Gaussian, Gabor filter or others. However, it will depend on characteristics of the surface (such as surface structure, material) under inspection which filter or filters will provide the best defect recognition results.

[0068] Therefore, a set of universal parameterized convolution filters may be used, and different sets of parameters for defining each filter (corresponding to different filters) may be tested for finding an optimized filter set. Figure 4 is a flow diagram showing exemplary steps for optimizing a set of convolution filters. In step 420, filter parameters for convolution filters are defined, with each set of filter parameters corresponding to one set of convolution filters 425. The convolution filter parameters may be defined randomly or based on predefined settings, or may optionally be preselected by a user or an automated optimization process. It is further possible to predefine the number of convolution filters used, to provide a maximum and / or minimum number of filters, or to preselect certain filter types, and to determine random filter sets or optimized filter sets within these constraints. The parameters defining each filter of a set may determine both the type of convolutional filter and the further filter characteristics, such as kernel size, derivative mode for filters, standard deviation o of a filter, and others. Optionally, binary parameters for each filter may also be used to enable or disable a filter, i.e. for controlling the number of convolution filters in the set.

[0069] As an example, a model could be defined to use up to seven convolution filters as one set of filters 425a. Each of the filters within this set of filters may be defined by a group of filter parameters. The number of filters chosen here is arbitrary and may be replaced by any desirable number of convolution filters. As a result of applying these filters to a reference image, seven feature maps are obtained. With each set 425a, 425b, 425c, 425d of parametrized filters, i.e. each set of filter parameters defining all convolution filters of one filter set, a model may be trained and tested. Training data may be obtained in step 430 by applying a selected filter set on one or more reference images as described above with labelled defect-free background areas. A first set 425a of parametrized filters defined by a set of filter parameters may then be applied in step 430 to the one or more reference images 300. The resulting feature maps provide training data including regions labelled as “defect free” which may then be used in step 440 as input for a machine learning model, preferably a multilayer perceptron or another feed-forward neural network. As a result, trained models 445a, 445b, 445c, 445d corresponding to specific filter sets 425a, 425b, 425c, 425d are obtained. In this phase, only a small training data set is used for training.

[0070] The preliminary trained model 445a, based on the first filter set 425a, is then in step 450 tested for model quality, i.e. accuracy of the defect pixel classification.

[0071] One or more simulated defects 310 may be added onto defect-free background images as defined above for obtaining simulated defect images as test set, i.e. for testing the quality of the trained models in step 450. These defects may be varied in size, angle, shape, pattern, cluster arrangement and / or any other desired parameter for obtaining a broad set of simulated defects. As an example, a simple simulated defect image may be used which includes a large variation of different simulated defects. The parameters for simulating defects may be determined at random, optionally using limits, thresholds or other boundary conditions for the defect parameters, such as a maximum width or a minimum contrast. Alternatively, simulated defects may be produced in a defined pattern of different defect types. Then, these simulated defects may be superimposed on or added to the defect-free background image for creating a simulated defect image. The resulting simulated defect image may for example show a pattern of line and / or spot defects of different widths, lengths, and positions across the background image. Since the defect characteristics are known and thus for each pixel of the simulated defect image, the desired classification as a good or defect pixel is known as well, these images may then be fed into each trained model and the result may be analysed in order to evaluate model quality.

[0072] As a possible criterion of model quality and error rate of the model, the number of correct and incorrect pixel classifications may be determined. For example, it may be determined for each simulated defect pixel whether it has been correctly classified as a defect (true positive, TP) or has not been recognized as a defect (false negative, FN), and similarly, it may be determined for each background pixel free of any simulated defects whether it has been correctly classified as a good pixel (true negative, TN) or erroneously classified as a defect pixel (false positive, FP). These values form a confusion matrix and may then be used directly as a measure of quality as desired. As a simple implementation, the model may simply be rated based on a total number of falsely classified pixel, i.e. the sum of false negatives and false positives. Another option is to use the confusion matrix for determining values such as sensitivity (or true positive rate), specificity (or true negative rate), and / or precision for the specific model / filter set. Generally, any other values derived from the confusion matrix may be used as well.

[0073] As an example, precision and sensitivity of the specific trained model may be calculated as usual:

[0074] TP

[0075] Sens = -

[0076] TP + FN that is, sensitivity is the ratio of true positive results TP and the sum of true positives TP and false negatives FN, i.e. the ratio of recognized defect pixels to all actually defect pixels; and

[0077] TP

[0078] Prec = -

[0079] TP + FP that is, the precision of the model is the ratio of true positive results TP and the sum of true positives TP and false positives FP, i.e. the ratio of true positive results from all positive results.

[0080] Of course, definitions of defects as “positive” and good pixels as “negative” are arbitrary and may also be reversed.

[0081] The resulting quality of the trained model using a specific convolution filter set for obtaining the input data is thus used as a measure of quality for the convolution filter set itself.

[0082] In the same way, a suitable number of different filter sets 425a, 425b, 425c, 425d may be tested, with each filter set being defined by its set of filter parameters. For example, a first filter set 425a including a given number of filters with random or predefined first parameters may be selected, and the convolution filters defined in this way may be applied to one or more reference images 300. The resulting feature maps obtained as output of the convolution filter application 430 may be used 440 as training data for an untrained machine learning model for pixel classification. Only one or a few reference images may be used for training in this stage of filter selection training. This first trained model 425a may then be tested on a set of test data as above, e.g. on at least one simulated defect image, and the confusion matrix or values derived from the confusion matrix may be output in step 450 as a result of evaluation of the filter set used.

[0083] Then, a second filter set 425b with random second parameters may be selected and applied to the same reference image 300 in the same way. Again, the resultant feature maps are provided to the same untrained model as training data, which leads to a second, different trained model 445b. Testing the second trained model 445b on the same test data (e.g. the same simulated defect image) allows to evaluate the quality of this second trained model, and thus indirectly the second filter set, in comparison.

[0084] In this way, multiple further filter sets may be defined, tested and evaluated based on the end result of each trained model. The number of filter sets to be evaluated is not limited in general, but may be preset or limited to avoid long training times. Alternatively, a quality threshold may be defined such that further filter sets are defined and tested until a predefined number of filter sets fulfils the quality threshold, i.e. is sufficient for use in a surface inspection. In some cases, a predetermined number of filter sets may be used for training and testing the results, and the optimum filter set 465 from the predetermined number of filter sets may be selected in step 460 based on the above indicators. Optional quality thresholds may be defined in this case as well, such that a filter set would not be accepted as best filter set if the number of false classifications in the trained result is higher than the quality threshold. Vice versa, it is conceivable to stop testing of further filter sets as soon as at least one trained result fulfilling the quality threshold 460 is found.

[0085] After an optimum convolution filter set 465 for the pixel classification model has been found, the machine learning model may be fully trained using this optimum filter set 465 (i.e. the optimum set of parameters defining this filter set) for building a trained pixel classification model. The training data for this full training may be prepared from one or more reference images using the defined optimum filter set 465, e.g. from maximum defect images with labelled regions. Applying the optimum set of convolution filters to each of the images in step 470 will again result in feature maps suitable as input training data for the model. At this full training stage 480, a larger number of reference images with labelled defect-free background areas may be used, compared to the filter optimization stage. As a result, a trained pixel defect classifier 330 is obtained.

[0086] Optionally, a test image with simulated defects may be used again as test data to control the quality of the final trained model and to determine the required extent of the training phase, e.g. to avoid overtraining. Of course, other test data such as manually labelled real defect images may be used alternatively or in combination with simulated defect images. A simulated defect image may be generated as described before for testing the trained model in the filter optimization phase. The above steps may be implemented in a software function or software module 110 for building a first machine learning model 330 as a pixel classifier. The function uses reference pod images 300 and simulated defect pixels 310 as input, and allows for user input e.g. to label the defect free-background regions in the reference image in step 340, or to adjust parameters of the vision system, image registration or image preprocessing in step 350.

[0087] The steps of defining reference images, finding optimized filter sets, training machine learning models using the optimized filter sets, and evaluation of the model using the pixel classifier building function 110 may be performed separately for each type of background or area on the pod, such that different classification models and filter sets may be used on different areas of the surface under inspection. The resulting trained models, processing parameters, optimized filter sets and other data relating to each trained classifier may be stored in order to be retrieved in further functions.

[0088] After a first machine learning model 330 for pixel classification has been found and trained in this way, resulting in a pixel classifier model artefact and a set of optimized convolution filters 465 for feature extraction, at least a second machine learning model may be trained for classifying the defect regions by defect type. Again, defect type classification in the later surface inspection process may generally either be performed parameter-based, e.g. based on thresholds for various defect parameters such as size, contrast, anisometry, or may be implemented as at least one machine learning model using labelled defect regions as training data. Training data for machine learning models for defect type classification may be labelled manually, and / or may also be preclassified using parameter threshold methods.

[0089] The categories or defect types may be selected as desired. As an example, defect type classification may be divided into two classifiers in succession. A first defect type classifier may be used to decide whether the defect region is considered an acceptable defect region (“okay”) or a not acceptable defect region (“not okay”), while the second defect type classifier selects exactly one class of-defect types for both acceptable and not acceptable defects, such as spots, small areas, large areas, scratches, and others. Alternatively, two separate second defect type classifier classifiers may be used for the two classes obtained from the first classifier, such that one second classifier is used for further determining the defect type of a “okay” defect region, while the other second classifier is used for further determining the defect type of a “not okay” defect region. Optionally, the second classifier may be omitted if a defect region has been classified as “okay”, such that only defects classified as not acceptable are assigned a specific defect type. Alternatively, a single classifier may be used, e.g. by defining each feasible combination of defect type and acceptable / non-acceptable as a separate class, and using a multi-class classifier which assigns exactly one of these defect type classes to each defect region. In other embodiments, each class may correspond to a certain characteristic, and a multi-label classifier may be used for combined characteristics, e.g. for classifying size, criticality (okay / not okay), and contrast of a defect.

[0090] In a training phase implemented by a software module or software function 120 for building a trained defect type classifier (i.e. a defect type classifier artifact), the one or more classifiers (machine learning models) used for defect type classification may be provided with labelled training data, i.e. image data comprising regions which have been defined as defect and have been labelled with one of the selected defect type classes. A schematic flow diagram for the building of a trained defect type classifier is shown in Figure 5.

[0091] The process 120 for labelling defect data for training may include images or partial images 500 with defect regions being displayed to a user who then sorts each of the defects into defect type classes by user input 530. The defect regions may be obtained by applying a trained pixel classification model 330 as above to reference images 500 of surfaces, such that pixel regions classified as “defect” are defined as an output. Classes for defect type classification may be predefined for selection by a user, and / or may be provided by user input, such that new or additional classes may be defined during a labelling process. Optionally, supporting features may be displayed for a user, such as visibly marking the defect region on a screen or proposed labels based on parameter thresholds. If more than one defect region is present in an image, a user may select one or more defect regions by user input and may assign a class as label for that defect region. In response to labelling by a user, the defect region may be saved together with the given label in a database 510, in particular as part of a training data set.

[0092] When a sufficient number of defects has been labelled to form a full training data set 510, the one or more defect type classifiers may be trained to build the complete defect type classification model 520. Testing the trained defect type classification model 520 may again be performed using labelled test data and e.g. a confusion matrix for evaluating the results.

[0093] As a simpler alternative to a machine learning model, defect regions may be classified into defect types based on parameters such as anisometry of a defect region, cluster formation of several defects, absolute size (length and / or width of a defect region, relative dimensions (e.g. thin defects), contrast, or any combination of these and other defect characteristics. An algorithm may be defined that checks thresholds for one or more of these parameters, optionally in a given order of priority, and assigns a class to a defect region depending on the result of each check. If more than one image type is used as input for classification, the parameters may also be selected separately for each of the images, e.g. different contrast parameters and / or resulting classifications for darkfield and brightfield images. Similarly, parameter checks for more than one image type may be used as additional conditional checks for selecting a class; for example, a defect region may be classified into a certain defect type class only if a threshold for a given parameter is exceeded in both darkfield and brightfield image of the defect region.

[0094] The models for pixel classification and defect type classification may also be trained and parametrized independently of each other or in another order. The defect type classification model 520 may in that case use labelled defect regions as training data which may have been found manually or using other methods, instead of using the trained pixel classification model 330 for defining the defect regions.

[0095] It will be understood that the defect type classes and parameters given here are only used by way of example and that any desirable categorization may be used. In particular, the selection of classes will also depend on the type of surface under inspection, e.g. based on what action is intended in case of a defect of a certain defect type, or based on surface materials.

[0096] The steps and phases as previously described may be used for obtaining finalized trained classification models for both pixel classification 330 and defect classification 520. These finalized models may then be saved and utilized in defect recognition on all pods of the same type. In the following, an exemplary embodiment of surface inspection for EUV pods will be described with reference to Figure 6, showing exemplary flows for a software function 130 for surface inspection.

[0097] Again, one or more surface images 600 of the pod surface under inspection are obtained, for example using the image obtaining methods and systems detailed above. Image registration is of particular importance if imaging is performed repeatedly on the same surface, for example for regular inspection over the lifetime of a pod.

[0098] Preprocessing of the obtained images for inspection may also be performed in the same way as described for the reference images and training data in the training stages. Preferably, steps and parameters 610 of image preprocessing in the surface inspection phase correspond completely to those of the model building phases in order to ensure validity of the classification results. Preprocessing parameters 610 may for example be stored together with the classification models 330, 520 or integrated in the final inspection module 130 as predefined parameters.

[0099] After at least one image of the surface under inspection has been captured and preprocessed, it is used as input data for the trained models. That is, the defined set of optimized convolution filters may be applied to the image for obtaining the feature maps which are then input into the trained pixel classification model 330. The defect regions obtained from this first step as an output of pixel classifier 330 may then be classified into defect type classes using the trained defect type classifier 520.

[0100] All of these processing and classification steps for live inspection may be integrated into one function 130, such that the images are used as input for the function and indications of defect regions together with an associated defect type are received as an output 630 of the function 130. The intermediary steps, such as preprocessing of images, feature maps, or indication of defect pixels from the first classification stage may or may not be given as separate outputs, which may in turn be stored, displayed or communicated to another unit. It will be understood that “live inspection” does not necessarily imply that the capturing of images has to be performed at a certain time or in any temporal relation to the classification steps; it is also possible to capture and store surface images and to perform the further steps of preprocessing and classification independently at a later time.

[0101] In some cases, changes of the surface features over time are of interest. For example, EUV pods are usually checked in regular intervals of time (e.g. each day or each week) to ensure integrity of the EUV pod. The pixel classification and defect type classification as described above may be performed, and results of the inspection and classification may be stored. In a next surface inspection at a later time, the same surfaces may be imaged using the same setup and parameters for both capturing and processing the image, and the new results of the pixel classification and / or the defect type classification may be compared to one or more previous classification result, such that any changes in defects may be identified.

[0102] The data 630 that is obtained as a result of the inspection 130 may include parameters of the defect regions, such as coordinates of the defect outlines for a defined reference frame, and information on the defect type gained from the second classification stage. Stored may further include one or more of raw or partially processed image data; defect parameters such as size, contrast level, anisometry, information on neighbouring defects (clusters, distance to other defects), image data of only the defect regions cut out from a background, parameters of the filters and classification models, and parameters of the transformation performed for image registration. The comparison of data for determining defect changes may include all or only part of the data stored. For example, if a defect at a certain location on the pod is known and has increased considerably in size at the next surface inspection scan, the change may be marked, stored and / or indicated to a user as a critical deterioration of the surface.

[0103] If a defect has been found in a previous surface inspection run, but not at a later time, a notification may be stored or indicated to a user. At least part of the inspection process from imaging to classification may be repeated, optionally with different parameters (e.g. without down-sampling of the image data), to determine whether the finding is a mistake or the defect is not present any more. In other cases, surface inspection may be performed for example before and after a pod cleaning process; in this case, it is expected that at least some defects such as objects on the surface are not present in later scans. The defect recognition may then be used to check the success of a cleaning process.

[0104] Any findings of a surface inspection run may also be analysed further, e.g. by determining the total number of critical defects or of a given defect type.

[0105] Since the pixel classification model is mainly trained based on the background regions in an image, these will be similar in pods of a different type, but having the same materials and characteristics. If small changes are made to a reticle pod or different pods are used in one setup for different types of reticles, the pixel classifier may in some cases be used without retraining the model if the relevant features are the same. Test data as described before, e.g. simulated defect pixels on a tiled background image, and a confusion matrix may be used for verifying the quality of an existing trained classification model on another surface or object. The separate defect type classification model may at least in part be applicable to other surfaces and surface regions as well, as it is trained based on parameters of the defect regions without the background.

[0106] A surface inspection system or a program module for surface inspection may include only machine learning models for both stages of classification, or may comprise additional classification methods at any stage. If more than one classification method is available at a given classification stage, a selection of classification method may be performed automatically, e.g. based on characteristics of the surface under inspection or of the imaging data, or may at least in part be based on a user selection. As an example, a system may comprise a trained machine learning model and a hysteresis threshold method as described above for pixel classification. For the defect type classification stage, another machine learning model with one or more classifiers may be present in the system as well as a threshold classification method based on characteristics of the defects. A user may then be provided with a query to select one of the available methods for each stage, or to select one of a number of predefined modes which correspond to specific combinations of classification methods for each stage.

[0107] If no fully trained model is available yet, for example due to a new pod design or material, parameter-based methods such as hysteresis threshold may be selected for all stages. Optionally, classification data obtained in this way may then also be used as labelled training data or test data for training the machine learning models on this pod design. Optionally, a user may be informed of the missing trained model and may then initialize a training stage.

[0108] User input 620 may also be used during live classification for relabelling defect data and therefore improving the model. For example, if a user disagrees with the classification output, a user input 620 indicating the correct defect type or defect image unit marking may be obtained and used for relabelling the respective defect image. Relabelled defects may then be used to retrain one or more of the machine learning classifiers.

[0109] For each region of a surface under inspection having different properties, separate libraries, separate background images, separately trained models, separate filter sets or different thresholds and parameters (e.g. for hysteresis threshold) may be determined and used. For example, for each of the different regions described in connection with Figure 1b, e.g. flat surface, edges, gas filter areas, the steps described above for obtaining the final filters and models may be run. It is also possible that some or all of the parameters and models are determined to be similar or identical for certain regions. Optionally, certain steps of the described methods may be restricted to certain regions of the surface under inspection, or may be replaced by others; for example, different preprocessing steps may be performed based on the surface region selected.

[0110] It will be understood that the methods described above may be implemented in software, i.e. program code which may be executed on a suitable processing unit. Any processing unit having sufficient processing power to perform the desired steps may be employed in the process, such as a microcontroller, microprocessor, single-core or multi-core processor, embedded systems or any other computing device capable of executing the software commands. It is not required that all steps and substeps of the methods are performed by the same unit or implemented in the same software program module; for example, the building and training of classifier models and optimizing filter sets may be separated from the program modules used for actual defect recognition, and / or the preprocessing of image data may at least in part be separated from any further image processing. Data sets, parameters, image data, models and / or executable program code may be stored in volatile and / or non-volatile storage elements such as hard disks, flash memory, or others. Also, any data, parameters, models, image data and / or program code may be transmitted using wireless or wired communication channels between two or more local or remote processing units, for example from a camera unit to a central control / processing unit of the inspection system, or from / to a remote server. Supporting software tools such as machine vision software (e.g. HALCON) or various integrated development environments (IDE) may be used to implement some parts of the described methods.

Claims

Claims1. A method for building a model (130) for defect recognition on a surface under inspection, wherein the model comprises a first supervised machine learning classifier configured for classifying image units of an image into at least two classes, said classes comprising at least a class for defect image units and a class for defect-free image units; the method comprising: defining (420) multiple sets of convolution filters (425a, 425b, 425c, 425d) for feature extraction from an image, wherein each set of the multiple sets (425a, 425b, 425c, 425d) of convolution filters comprises one or more convolution filters defined by a set of filter parameters, each set of convolution filters differing from another set of convolution filters in at least one filter parameter; applying (430) said convolution filters of each one of said multiple sets (425a, 425b, 425c, 425d) of convolution filters to one or more labelled reference images of a surface to obtain multiple groups of feature maps, each group of feature maps corresponding to one out of said multiple sets of convolution filters; obtaining (440) multiple trained first machine learning classifiers (445a, 445b, 445c, 445d) by training multiple untrained copies of said first machine learning classifier using said multiple groups of feature maps as training set, wherein each copy of said first machine learning classifiers is provided with one group of feature maps out of said multiple groups as input, such that each of said trained first machine learning classifier is associated with one out of said multiple sets (425a, 425b, 425c, 425d) of convolution filters; for each one of said trained first machine learning classifiers (445a, 445b, 445c, 445d), providing said trained classifier with an input test set, wherein said input test set comprises a group of test feature maps obtained by applying said set of convolution filters associated with said trained classifier to at least one predefined test image, and obtaining (450) a quality parameter indicating a quality of the classification result for each trained machine learning classifier; determining (460) an optimized set (465) of convolution filters out of said multiple sets (425a, 425b, 425c, 425d) of convolution filters based on said obtained quality parameters associated with each of said sets of convolution filters; applying (470) said optimized set of convolution filters to one or more labelled training images of a surface to obtain a group of optimized training feature maps; and training (380) a copy of said first machine learning classifier using said group of optimized training feature maps as training data for obtaining a first trained machine learning classifier.

2. The method of claim 1 , wherein said filter parameters defining said convolution filters comprise at least one of: a filter type, a kernel size, a stride length, a padding size, an activation flag for a convolution filter, a number of convolution filters within one set.

3. The method of claim 1 or 2, wherein said multiple sets (425a, 425b, 425c, 425d) of convolution filters are determined by randomly assigning values from a given range for one or more of said filter parameters for each set of convolution filters.

4. The method of any one of the preceding claims, wherein said test image is generated by determining at least one region of a reference image which is labelled as defect-free, building a background image by tiling multiple copies of said defect-free region of said reference image; and adding one or more simulated defect image regions (310) onto said background image, wherein said defect regions are defined by predetermined and / or random image parameters for each image unit of a simulated defect region.

5. The method of any one of the preceding claims, wherein the model (130) comprises at least a second machine learning classifier configured for classifying defect image regions in an image into one of a plurality of defect type classes; the method further comprising: displaying a plurality of defect image regions to a user, each defect image region comprising one or more defect image units; assigning at least one label indicating a defect type class to each of said defect image regions based on a user input (530); and training said at least one second machine learning classifier using said labelled defect image regions as training data (510) to obtain a trained second machine learning classifier (520).

6. A method for inspecting a surface for defects, the method comprising: obtaining (600) at least one inspection image of a surface under inspection; applying a predefined optimized set of convolution filters (465) to said at least one inspection image to obtain a group of inspection image feature maps; providing said group of inspection image feature maps as input to a model comprising at least a first trained machine learning classifier (330); obtaining, as an output (630) of said model, a classification result for image units of said at least one image, said classification result indicating at least image units of said image which are classified as defect image units.

7. The method of claim 6, further comprising: providing parameters of said defect image units as an input to at least one second trained machine learning classifier (520) of said model, wherein said at least one second trained machine learning classifier is configured for classifying defect image regions including one or more defect image unit in an image into one of a plurality of defect type classes; obtaining a classification result (630) for said defect image regions, said classification result indicating, for each defect image unit, an associated defect type class as an output of said at least one second trained machine learning classifier.

8. The method of claim 6 or 7, further comprising: providing a visual display of defect image units in said at least one inspection image to a user.

9. The method of any one of claims 6 to 8, further comprising: retrieving stored parameters of defect image units and / or defect type classes of defect image regions obtained from at least one earlier inspection image that has been captured at an earlier point in time; comparing said retrieved parameters of defect regions and / or defect image units of said earlier inspection image to said obtained classification results; and if a difference in defect regions and / or defect image units in said comparison exceeds a predefined threshold, defining said defect image region as a changed defect.

10. The method of any one of claims 6 to 9, the method further comprising: providing at least a first parameter-based classifier configured for classifying image units of an image into defect image units and defect-free image units based on predefined parameter thresholds for said image units; providing at least a second parameter-based classifier configured for classifying defect image regions including one or more defect image unit in an image into one of a plurality of defect type classes, said classifying being based on predefined parameter thresholds for said defect image regions; selecting a first classifier out of said first trained machine learning classifier and said first parameter-based classifier for classifying defect image units, and determining defect image units using the selected classifier; and selecting at least a second classifier out of said at least one second trained machine learning classifier and said second parameter-based classifier, and classifying defect regions into defect type classes using said selected second classifier; wherein said selecting of classifiers is based on one of: a user input, an image characteristic, a surface characteristic of the imaged surface.

11. The method of claim 5 or any one of claims 7 to 10, wherein said defect type class is defined based on one or more parameters of said defect image region, said parameters comprising at least one of: an indication of whether the defect is acceptable; a size of the defect image region; a contrast level of the defect image region; a parameter indicating an anisometry of the defect image region; a relative dimension of the defect image region; a defect type.

12. The method of claim 5 or any one of claims 7 to 11 , wherein said second machine learning classifier (520) comprises at least a first sub-classifier and one or more second subclassifiers in succession, wherein said first sub classifier is configured for classifying defect image regions as either an acceptable defect or a non-acceptable defect, and wherein said one or more second sub-classifiers are configured for classifying said acceptable defects or said non-acceptable defect into defect type classes.

13. The method of any one of the preceding claims, further comprising preprocessing an image prior to applying said set of convolution filters, wherein the image is at least one of a training image, a reference image, a test image, or an inspection image, wherein said preprocessing comprises at least one of: enhancing a contrast value in said image; applying a coordinate transformation for image registration based on reference coordinates; applying an image filter to said image.

14. The method of claim 13, wherein said preprocessing comprises: obtaining a darkfield and a brightfield image captured with a same field of view, obtaining contrast enhanced images by enhancing local contrast in each of the darkfield and brightfield images; determining an image background free of defects in each of the darkfield and brightfield images by calculating a mean image value; obtaining difference images for each brightfield and darkfield by calculating a difference of said image background and each of said contrast enhanced images; and combining said difference images by selecting a maximum for each image unit from said brightfield and darkfield images.

15. The method of any one of the preceding claims, wherein for each image region corresponding to a surface region having specified different surface characteristics, a separate model is trained.

16. The method of any one of the preceding claims, further comprising:defining contiguous image regions which include multiple defect image units located adjacent to each other or at least at a predefined maximum distance to each other as defect image regions.

17. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method of any previous claim.

Images