Defect inspection of semiconductor samples

Abstract

A system and method for defect inspection of a semiconductor sample is provided. The method includes obtaining a runtime image of the semiconductor sample, generating a reference image based on the runtime image using a machine learning (ML) model, and performing defect inspection on the runtime image using the generated reference image. The ML model is previously trained during setup using a training set including one or more pairs of training images, each pair of training images including a defect image and a corresponding defect-free image. The training includes, for each pair of training images, processing the defect image by the ML model to obtain a predicted image, and optimizing the ML model to minimize a difference between the predicted image and the defect-free image.

Description

TECHNICAL FIELD

[0001] The presently disclosed subject matter relates generally to the field of inspecting semiconductor samples, and more particularly to defect detection of semiconductor samples. BACKGROUND

[0002] Current demands for high density and performance associated with very large scale integration of fabricated devices require sub-micron features, increased transistor and circuit speed, and increased reliability. As semiconductor processes advance, pattern dimensions such as line width, and other types of critical dimensions, continue to shrink. These demands require that device features be formed with high precision and uniformity, which in turn necessitates careful monitoring of the fabrication process, including automated inspection of the devices while they are still in the form of semiconductor wafers.

[0003] Runtime inspection can generally employ a two-stage process, e.g., inspecting a sample, and then reviewing sampled locations of potential defects. Inspection typically involves generating certain outputs (e.g., images, signals, etc.) of the sample by directing light or electrons to the wafer and detecting light or electrons from the wafer. During the first stage, the surface of the sample is inspected at high speed and relatively low resolution. Defect detection is typically performed by applying defect detection algorithms to the inspection outputs. A defect map is produced to show suspicious locations on the sample that have a high probability of being defects. During the second stage, at least some of the suspicious locations are analyzed more thoroughly at relatively high resolution to determine different parameters of the defects, such as category, thickness, roughness, size, etc.

[0004] Inspection can be provided using non-destructive inspection tools during or after fabrication of the sample to be inspected. Inspection typically involves generating certain outputs (e.g., images, signals, etc.) of the sample by directing light or electrons to the wafer and detecting light or electrons from the wafer. As non-limiting examples, various non-destructive inspection tools include scanning electron microscopes, atomic force microscopes, optical inspection tools, etc.

[0005] An inspection process can include multiple inspection steps. A fabrication process of a semiconductor device can include various processes, such as etching, deposition, planarization, growth (such as epitaxial growth), implantation, etc. An inspection step can be performed multiple times, e.g., after certain processing programs and / or after fabrication of certain layers, etc. Additionally or alternatively, each inspection step can be repeated multiple times, e.g., for different wafer locations or for the same wafer location with different inspection settings.

[0006] Inspection processes are used at various steps during semiconductor manufacturing to detect and classify defects on samples, as well as to perform metrology-related operations. The effectiveness of the inspection can be improved through automation of the process(es), such as defect detection, automatic defect classification (ADC), automatic defect review (ADR), image segmentation, automated metrology-related operations, and the like. Automated inspection systems ensure that manufactured parts meet expected quality standards and provide useful information about possible adjustments needed for manufacturing tools, equipment, and / or ingredients, depending on the type of defects identified.

[0007] In some cases, machine learning techniques can be used to assist inspection processes in order to provide accurate and efficient solutions for automation of specific inspection applications, and to facilitate higher yields. SUMMARY

[0008] According to certain aspects of the presently disclosed subject matter, there is provided a computerized system for defect inspection of a semiconductor sample, the system comprising a processing and memory circuitry (PMC) configured to: acquire a runtime image of a semiconductor sample; generate a reference image based on the runtime image using a machine learning (ML) model; and perform defect inspection of the runtime image using the generated reference image; wherein the ML model is previously trained during a setup using a training set comprising one or more pairs of training images, each pair of training images comprising a defective image and a corresponding defect-free image, the training comprising, for each pair of training images: processing the defective image by the ML model to obtain a predicted image; and optimizing the ML model to minimize a difference between the predicted image and the defect-free image.

[0009] In addition to the features described above, the system according to this aspect of the presently disclosed subject matter can include one or more of the following listed features (i) to (xi) in any desired combination or arrangement technically possible:

[0010] (i). The ML model is a deep neural network (DNN) configured to learn a non-linear mapping between two populations of defective images and defect-free images.

[0011] (ii). The defective image in each pair of training images is one of: an actual image of a semiconductor sample acquired by an inspection tool and verified to contain a defect feature, or a simulated image obtained by synthetically implanting a defect feature on a defect-free image.

[0012] (iii). The corresponding defect-free image in each pair of training images is one of: an actual image of a semiconductor sample acquired by an inspection tool and verified to not contain a defect feature, or a simulated image generated based on design data of the semiconductor sample.

[0013] (iv). Pre-processing the defect image and the defect-free image in each pair of training images by registering the defect image with the defect-free image prior to training to correct misalignment between the defect image and the defect-free image.

[0014] (v). Pre-processing the defect image and the defect-free image in each pair of training images by performing noise filtering on the defect image and the defect-free image prior to training to reduce noise representing variations caused by one or more physical processes of the specimen.

[0015] (vi). Pre-processing the defect image and the defect-free image in each pair of training images by performing image enhancement on the defect image and the defect-free image using pattern completion prior to training to produce a plurality of defect sub-images corresponding to respective portions of the defect-free image.

[0016] (vii). Pre-processing the defect image and the defect-free image in each pair of training images by performing image registration, noise filtering, and image enhancement on the defect image and the defect-free image prior to training.

[0017] (viii). The ML model includes at least one learning module prior to the DNN, the at least one learning module trained to perform at least one of image registration and noise filtering on the defect image and the defect-free image prior to processing the defect image and the defect-free image by the DNN.

[0018] (ix). The ML model is optimized using a loss function selected from a mean squared error (MSE), a sum of absolute differences (SAD), a structural similarity index measure (SSIM), and an edge-preserving loss function.

[0019] (x). The defect inspection is one of defect detection, defect review, and defect classification.

[0020] (xi). The generated reference image has improved alignment with the runtime image and shares similar noise levels with the runtime image, and the PMC is configured to perform the defect inspection by directly comparing the runtime image with the generated reference image to obtain defect inspection results indicative of a distribution of defects on the semiconductor specimen.

[0021] According to other aspects of the presently disclosed subject matter, there is provided a computerized method for inspecting a semiconductor sample, the method comprising the steps of: acquiring a runtime image of the semiconductor sample; generating a reference image based on the runtime image using a machine learning (ML) model; and performing defect inspection on the runtime image using the generated reference image; wherein the ML model was previously trained during setup using a training set comprising one or more pairs of training images, each pair of training images comprising a defective image and a corresponding defect-free image, the training comprising, for each pair of training images: processing the defective image by the ML model to obtain a predicted image; and optimizing the ML model to minimize a difference between the predicted image and the defect-free image.

[0022] This aspect of the presently disclosed subject matter can be modified as necessary to include one or more of the features (i) to (xi) listed above with respect to the system, in any desired combination or arrangement technically possible.

[0023] According to other aspects of the presently disclosed subject matter, there is provided a non-transitory computer-readable storage medium comprising instructions, the non-transitory computer-readable storage medium tangibly embodying a program of instructions, the instructions, when executed by a computer, causing the computer to perform a computerized method for defect inspection of a semiconductor sample, the method comprising the steps of: acquiring a runtime image of the semiconductor sample; generating a reference image based on the runtime image using a machine learning (ML) model; and performing defect inspection on the runtime image using the generated reference image; wherein the ML model was previously trained during setup using a training set comprising one or more pairs of training images, each pair of training images comprising a defective image and a corresponding defect-free image, the training comprising, for each pair of training images: processing the defective image by the ML model to obtain a predicted image; and optimizing the ML model to minimize a difference between the predicted image and the defect-free image.

[0024] This aspect of the presently disclosed subject matter can be modified as necessary to include one or more of the features (i) to (xi) listed above with respect to the system, in any desired combination or arrangement technically possible. BRIEF DESCRIPTION OF DRAWINGS

[0025] For a better understanding of the present disclosure, and to show how it can be implemented in practice, embodiments will now be described, by way of non-limiting examples only, with reference to the accompanying drawings, in which:

[0026] Figure 1 A generalized block diagram of an inspection system in accordance with certain embodiments of the presently disclosed subject matter is shown.

[0027] Figure 2A generalized flowchart showing training of a machine learning model that can be used to generate reference images for defect images of semiconductor samples is shown in accordance with certain embodiments of the presently disclosed subject matter.

[0028] Figure 3 A generalized flowchart showing runtime defect inspection using a machine learning model trained to generate reference images is shown in accordance with certain embodiments of the presently disclosed subject matter.

[0029] Figure 4 A schematic diagram showing an exemplary training process for an ML model is shown in accordance with certain embodiments of the presently disclosed subject matter.

[0030] Figure 5 A schematic diagram showing an exemplary runtime inspection process for a trained ML model is shown in accordance with certain embodiments of the presently disclosed subject matter. DETAILED DESCRIPTION

[0031] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the presently disclosed subject matter can be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the presently disclosed subject matter.

[0032] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "obtaining", "examining", "generating", "learning", "training", "processing", "optimizing", "performing", "preprocessing", "registering", "filtering", "inpainting", "using", "comparing", or the like, refer to the action(s) and / or process(es) of a computer that manipulate and / or transform data into other data, the data represented as physical, such as electronic, quantities and / or the data representing the physical objects. The term "computer" should be expansively construed to encompass a variety of hardware-based electronic

[0033] The terms “non-transitory storage” and “non-transitory storage medium” used herein should be interpreted to cover any volatile or non-volatile computer memory suitable for the presently disclosed subject matter. These terms should be understood to encompass a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store a set of or multiple sets of instructions that are executable by a computer to cause the computer to perform any one or more of the methodologies disclosed. Thus, these terms should be interpreted to cover any medium that is deemed by a skilled artisan to be non-transitory. Such media include, but are not limited to, random access memory (“RAM”), read-only memory (“ROM”), magnetic disk storage mediums, optical storage mediums, flash memory devices, and the like.

[0034] The term “sample” used in this specification should be construed broadly to cover any kind of physical object or substrate, including wafers, reticles, masks, and other structures used in the manufacture of semiconductor integrated circuits, magnetic heads, flat panel displays, and other articles of manufacture for semiconductor manufacturing, combinations and / or portions thereof. The sample is also referred to herein as a semiconductor sample, and can be produced by manufacturing equipment performing a corresponding manufacturing process.

[0035] The term “inspection” used in this specification should be construed broadly to cover any kind of operation related to various types of defect detection, defect review, and / or defect classification, segmentation, and / or metrology operations during and / or after a sample manufacturing process. The inspection is provided using non-destructive inspection tools during or after the manufacture of the sample to be inspected. As non-limiting examples, the inspection process can include runtime scanning (in a single or multiple scans), imaging, sampling, detection, review, measurement, classification, and / or other operations provided with respect to the sample or portions thereof using the same or different inspection tools. Likewise, the inspection can be provided prior to the manufacture of the sample to be inspected, and can include, for example, generating an inspection recipe(s) and / or other setup operations. It should be noted that the term “inspection” or its derivatives used in this specification are not limited in resolution or size of the inspection area unless explicitly stated otherwise. As non-limiting examples, various non-destructive inspection tools include scanning electron microscopes (SEMs), atomic force microscopes (AFMs), optical inspection tools, and the like.

[0036] The term "defect" as used in the present specification should be construed broadly to cover any kind of anomaly or undesirable feature / functionality formed on a specimen. In some cases, a defect can be a defect of interest (DOI), which is a real defect that has some impact on the functionality of the manufactured device, and thus it is in the customer's interest to detect DOIs. For example, any "killer" defect that can result in yield loss can be represented as a DOI. In some other cases, a defect can be a nuisance (also referred to as a "false positive" defect), which can be ignored as the nuisance has no impact on the functionality of the finished device and does not affect yield.

[0037] The term "defect candidate" as used in the present specification should be construed broadly to cover a suspect defect location on a specimen that is detected to have a relatively high probability of being a defect of interest (DOI). Thus, at the time of review / testing, a defect candidate can actually be a DOI, or in some other cases, a defect candidate can be a nuisance as described above, or random noise caused by different variations during inspection (e.g., process variations, color variations, mechanical and electrical variations, etc.).

[0038] The term "design data" as used in the present specification should be construed broadly to cover any data indicative of a layered physical design (layout) of a specimen. The design data can be provided by a corresponding designer and / or can be derived from the physical design (e.g., through complex simulations, simple geometric and Boolean operations, etc.). The design data can be provided in different formats, such as GDSII format, OASIS format, etc., by way of non-limiting examples. The design data can be presented in a vector format, a grayscale intensity image format, or otherwise.

[0039] It should be appreciated that certain features of the presently disclosed subject matter, which are described in the context of separate embodiments, can also be provided in combination in a single embodiment, unless otherwise specifically noted, Conversely, various features of the presently disclosed subject matter, which are described in the context of a single embodiment, can also be provided separately or in any appropriate subcombination. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the methods and devices. For the purpose of clarity, the description will not list every replacement, modification, and / or substitution for every possible implementation.

[0040] Semiconductor manufacturing processes typically require multiple sequential processing steps and / or layers, each of which can result in defects, leading to yield loss. Defect inspection operations such as defect detection, defect review, and defect classification can be performed at various processing steps / layers during the manufacturing process to monitor and control the process.

[0041] Various detection methods can be used to detect defects on a specimen. As an example, a classical die-to-reference detection algorithm is commonly used, such as, for example, die-to-die (D2D). In D2D, an inspection image of a target die is captured. To detect defects in the inspection image, one or more reference images are captured from one or more reference dies (e.g., one or more neighboring dies) of the target die. The inspection image and the reference images are aligned and compared to each other. One or more difference images can be generated based on the differences between the pixel values of the inspection image and the pixel values derived from the one or more reference images. A detection threshold is then applied to the difference map and a defect map is created indicating defect candidates in the target die.

[0042] There are certain drawbacks associated with the above-mentioned die-to-reference detection methods. For example, the D2D method requires the acquisition of at least two images (i.e., in the case of one inspection image and one reference image), which doubles the image acquisition time of the inspection tool. In the case of using multiple references, the image acquisition time increases significantly depending on the number of reference images. Additionally, the inspection image and the reference images need to be registered before comparison. In some cases, they require additional pre-processing in order to compensate for noise representing variations between the two images, such as, for example, process variations and color variations. These inevitably increase the processing time of the detection method, thereby affecting the detection throughput (TpT). Additionally, the detection sensitivity can also be affected due to residual variations and noise that are not eliminated by the pre-processing.

[0043] Accordingly, certain embodiments of the presently disclosed subject matter propose to use a machine learning model to generate a synthetic reference image instead of acquiring an actual reference image on-the-fly, and use the generated reference image to perform a defect inspection operation. The proposed on-the-fly inspection system significantly reduces the image acquisition time of the tool and eliminates image pre-processing work such as image registration and noise filtering, thereby improving the detection throughput and defect detection sensitivity, as will be described in detail below.

[0044] With this in mind, attention is directed to Figure 1 , Figure 1 A functional block diagram of an inspection system in accordance with certain embodiments of the presently disclosed subject matter is shown.

[0045] Figure 1The illustrated inspection system 100 can be used for inspection of semiconductor samples (e.g., wafers, dies, or portions thereof) as part of a sample manufacturing process. As noted above, inspection as referred to herein can be interpreted to encompass any kind of operation related to defect review / detection, defect review, defect classification, segmentation, and / or metrology operations (such as, for example, critical dimension (CD) measurements, etc.) with respect to a sample. The system 100 includes one or more inspection tools 120 configured to scan a sample and capture images thereof for further processing for various inspection applications.

[0046] The term “inspection tool(s)” as used herein should be broadly interpreted to encompass any tool that can be used for inspection-related processes, including, by way of non-limiting examples, scanning (in single or multiple scans), imaging, sampling, review, measurement, classification, and / or other processes provided with respect to a sample or portions thereof. Without limiting the scope of the present disclosure in any way, it is also noted that the inspection tools 120 can be implemented as various types of machines, such as optical machines, electron beam machines (e.g., scanning electron microscopes (SEMs), atomic force microscopes (AFMs), or transmission electron microscopes (TEMs), etc.), and the like.

[0047] The one or more inspection tools 120 can include one or more review tools and / or one or more inspection tools. In some cases, at least one of the inspection tools 120 can be an inspection tool configured to scan a sample (e.g., an entire wafer, an entire die, or portions thereof) to capture inspection images (typically at relatively high speed and / or low resolution) for detecting potential defects (i.e., defect candidates). During inspection, the wafer can be moved relative to a detector of the inspection tool by a certain step size during exposure (or the wafer and tool can be moved relative to each other in opposite directions), and the wafer can be scanned by the inspection tool stepwise along a strip of the wafer, with the inspection tool imaging a component / part of the sample (within a strip) at a time. As an example, the inspection tool can be an optical inspection tool. In each step, light can be detected from a rectangular portion of the wafer, and such detected light is converted into a plurality of intensity values at a plurality of points in that portion, thereby forming an image corresponding to that component / part of the wafer. For example, in optical inspection, an array of parallel laser beams can scan the wafer surface along a strip. The strips are placed in parallel rows / columns next to each other to build up an image of the wafer surface one strip at a time. For example, the tool can scan the wafer from top to bottom along a strip, then switch to the next strip and scan the wafer from bottom to top, and so on until the entire wafer is scanned and an inspection image of the wafer is collected.

[0048] In some cases, at least one of the inspection tools 120 can be an review tool configured to capture review images of at least some of the defect candidates detected by the inspection tool for use in determining whether the defect candidates are indeed defects of interest (DOIs). Such review tools are typically configured to inspect a segment of the sample at a time (often at a relatively low speed and / or high resolution). As an example, the review tool can be an electron beam tool such as, for example, a scanning electron microscope (SEM) or the like. A SEM is an electron microscope that produces an image of a sample by scanning the sample with a focused beam of electrons. The electrons interact with atoms in the sample, producing various signals containing information about the sample's surface topography and / or composition. SEMs are capable of precisely inspecting and measuring features during semiconductor wafer fabrication.

[0049] The inspection tools and review tools can be different tools located at the same or different locations, or a single tool operating in two different modes. In some cases, the same inspection tool can provide both low resolution image data and high resolution image data. The resulting image data (low resolution image data and / or high resolution image data) can be transmitted to the system 101 directly or via one or more intermediate systems. The present disclosure is not limited to any particular type of inspection tools and / or resolution of the image data produced by the inspection tools. In some cases, at least one of the inspection tools 120 has metrology capability and can be configured to capture images and perform metrology operations on the captured images. Such inspection tools are also referred to as metrology tools.

[0050] It should be noted that the term image in the context of (multiple) runtime images, (multiple) reference images, or (multiple) training images used herein can refer to raw images of the sample captured by the inspection tools during the manufacturing process, derivatives of the captured images obtained through various pre-processing stages, and / or computer generated synthetic images (in some cases based on design data). It should be noted that in some cases, the images referred to herein can include image data (e.g., captured images, processed images, etc.) and associated digital data (e.g., metadata, handcrafted attributes, etc.). It should also be noted that the image data can include data related to one or more of: a processing step of interest, a layer of interest, multiple processing steps, and / or multiple layers of the sample.

[0051] According to certain embodiments of the presently disclosed subject matter, the inspection system 100 includes a computer-based system 101 operably connected to the inspection tools 120 and capable of performing automatic defect inspection of semiconductor samples at runtime based on runtime images obtained during the manufacturing of the samples. The system 101 is also referred to as a defect inspection system.

[0052] In particular, the system 101 includes a processor and memory circuitry (PMC) 102 operably connected to a hardware-based I / O interface 126. The PMC 102 is configured to provide processing required by an operating system, as referenced Figures 2-3 Further in detail, and including a processor (not shown separately) and memory (not shown separately). The processor of the PMC 102 can be configured to execute several functional modules in accordance with computer-readable instructions implemented on a non-transitory computer-readable memory included in the PMC. Such functional modules are referred to hereinafter as being included in the PMC.

[0053] The processor referred to herein can represent one or more general- purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processor can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processor can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processor is configured to execute instructions for performing the operations and steps discussed herein.

[0054] The memory referred to herein can include a main memory (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or a RAMbus DRAM (RDRAM), etc.) and a static memory (e.g., flash memory, static random access memory (SRAM), etc.).

[0055] According to certain embodiments of the presently disclosed subject matter, the system 101 can be a runtime defect inspection system configured to perform defect inspection operations using a trained machine learning (ML) model based on runtime images obtained during sample fabrication. In such cases, one or more functional modules included in the PMC 102 of the system 101 can include a machine learning (ML) model 108 previously trained for generating reference images and a defect inspection module 110.

[0056] In particular, the PMC 102 can be configured to obtain a runtime image of a semiconductor sample via the I / O interface 126, and provide the runtime image as input to a machine learning model (e.g., the ML model 108) for processing. The ML model 108 can generate a reference image based on the runtime image. The ML model 108 is previously trained during a setup using a training set including one or more pairs of training images, each pair of training images including a defective image and a corresponding non-defective image. The defect inspection module 110 can be configured to perform a defect inspection on the runtime image using the generated reference image.

[0057] In such a case, the ML model 108 and the defect inspection module 110 can be considered as part of a defect inspection recipe that can be used to perform a runtime defect inspection operation on an obtained runtime image. The system 101 can be considered as a runtime defect inspection system that is capable of performing runtime defect related operations using the defect inspection recipe. Details of performing the runtime inspection process are described below with reference to Figure 3 and Figure 5

[0058] In some embodiments, the system 101 can be configured as a training system that is capable of training the ML model using a particular training set during a training / setup phase. In such a case, the one or more functional modules included in the PMC 102 of the system 101 can include the training set generator 104, the training module 106, and the ML model 108. In particular, the training set generator 104 can be configured to generate a training set including one or more pairs of training images, each pair of training images including a defective image and a corresponding non-defective image. The training module 106 can be configured to train the ML model 108 using the training set. In particular, the training can include processing the defective image by the ML model to obtain a predicted image, and optimizing the ML model to minimize a difference between the predicted image and the non-defective image.

[0059] As mentioned above, the ML model, after being trained, can be used to generate a reference image for a runtime image. Details of the training process are described below with reference to Figure 2 and Figure 4

[0060] The operation of the system 100 and the system 101, the PMC 102, and the functional modules therein will be described in further detail with reference to Figure 2 and Figure 3

[0061] ​​​According to certain embodiments, the ML models 108 referred to herein can be implemented as various types of machine learning models such as, for example, decision trees, support vector machines (SVMs), artificial neural networks (ANN), regression models, Bayesian networks, or a collection / combination thereof, etc. The learning algorithm used by the ML models can be any of the following: supervised learning, unsupervised learning, or semi-supervised learning, etc. The presently disclosed subject matter is not limited to a particular type of ML model or a particular type of learning algorithm used by the ML model.

[0062] In some embodiments, the ML models can be implemented as deep neural networks (DNNs). The DNNs can include supervised or unsupervised DNN models that include layers organized according to a respective DNN architecture. As non-limiting examples, the layers of the DNNs can be organized according to a convolutional neural network (CNN) architecture, a recurrent neural network architecture, a recursive neural network architecture, a generative adversarial network (GAN) architecture, or other architectures. Optionally, at least some of the layers can be organized into multiple DNN subnetworks. Each layer of the DNNs can include a plurality of basic computational elements (CEs), commonly referred to in the art as dimensions, neurons, or nodes.

[0063] Generally, the computational elements of a given layer can be connected with the CEs of a preceding layer and / or a succeeding layer. Each connection between a CE of the preceding layer and a CE of the succeeding layer is associated with a weight value. A given CE can receive inputs from the CEs of the preceding layer via respective connections, each given connection being associated with a weight value that can be applied to the inputs of the given connection. The weight values can determine the relative strength of the connection, and thus the relative influence of the respective inputs on the output of the given CE. The given CE can be configured to compute an activation value (e.g., a weighted sum of the inputs), and further derive an output by applying an activation function to the computed activation. The activation function can be, for example, an identity function, a deterministic function (e.g., a linear function, a sigmoid function, a threshold function, etc.), a stochastic function, or other suitable function. The output from the given CE can be transmitted to the CEs of the succeeding layer via respective connections. Likewise, as noted above, each connection at the output of a CE can be associated with a weighting value that can be applied to the output before it is received as an input to a CE of the succeeding layer. In addition to the weight values, there can also be threshold values (including limiting functions) associated with the connections and the CEs.

[0064] The weights and / or threshold values of the ML models can be initially selected prior to training, and can be further iteratively adjusted or modified during training to achieve an optimal set of weights and / or threshold values in the trained model. The set of input data used to adjust the weights / threshold values of the ML model is referred to as a training set. Details of the training process are described below with reference to Figure 2 and Figure 4 The details of the training process are described below with reference to

[0065] It is noted that the teachings of the presently disclosed subject matter are not constrained by the specific architecture of the ML model or DNN as described above.

[0066] In some cases, in addition to system 101, inspection system 100 can also include one or more inspection modules, such as, for example, an additional defect detection module, an automatic defect review module (ADR), an automatic defect classification module (ADC), metrology related modules, and / or other inspection modules that can be used to inspect a semiconductor sample. The one or more inspection modules can be implemented as standalone computers, or their functionalities (or at least part thereof) can be integrated with inspection tool 120. In some cases, the output of system 101 can be provided to the one or more inspection modules (such as ADR, ADC, etc.) for further processing.

[0067] According to certain embodiments, system 100 can include a storage unit 122. Storage unit 122 can be configured to store any data required by system 101, for example, data related to the input and output of system 101, as well as intermediate processing results generated by system 101. As an example, storage unit 122 can be configured to store sample images produced by inspection tool 120 and / or derivatives thereof, such as, for example, runtime images, training sets as described above. These input data can thus be retrieved from storage unit 122 and provided to PMC 102 for further processing. The output of system 101, such as generated reference images and / or defect inspection data, can be sent to storage unit 122 for storage.

[0068] In some embodiments, system 100 can optionally include a computer-based graphical user interface (GUI) 124 configured to implement user-specified inputs related to system 101. For example, a user can be presented with a visual representation of a sample, including an image of the sample, etc. (e.g., through a display forming part of GUI 124). The user can be provided with options through the GUI to define certain operational parameters. The user can also view operational results or intermediate processing results on the GUI, such as, for example, reference images, defect inspection data, etc. In some cases, system 101 can be further configured to send certain outputs to storage unit 122 and / or to an external system (e.g., a yield management system (YMS) of a fabrication plant (FAB)).

[0069] Those skilled in the art will readily understand that the teachings of the presently disclosed subject matter are not constrained by the architecture of the system shown; equivalent and / or modified functionality can be incorporated or divided in another manner, and can be implemented in any suitable combination of software and firmware and / or hardware. Figure 1 The system shown is not limiting; equivalent and / or modified functionality can be incorporated or divided in another manner, and can be implemented in any suitable combination of software and firmware and / or hardware.

[0070] It is noted that Figure 1The illustrated system can be implemented in a distributed computing environment, where Figure 1 The foregoing components and functional modules illustrated can be distributed across several local and / or remote devices and linked through a communication network. As an example, the inspection tool 120 and the system 101 can be located at the same entity (hosted in some cases by the same device) or distributed across different entities. As another example, as noted above, in some cases the system 101 can be configured as a training system for training ML models, while in some other cases the system 101 can be configured as a runtime defect inspection system using trained ML models. Depending on the particular system configuration and implementation needs, the training system and the runtime inspection system can be located at the same entity (hosted in some cases by the same device) or distributed across different entities.

[0071] It is also noted that in some embodiments at least some of the inspection tools 120, storage units 122, and / or GUIs 124 can be external to the inspection system 100 and in data communication with the system 100 and 101 via the I / O interface 126. The system 101 can be implemented as a standalone computer(s) used in conjunction with the inspection tools and / or additional inspection modules as described above. Alternatively, various functionalities of the system 101 can be at least partially integrated with one or more of the inspection tools 120, thereby facilitating and enhancing the functionality of the inspection tools 120 in the inspection related processes.

[0072] While not necessarily so, the operational processes of the system 101 and the system 100 can correspond to some or all of the stages of the method described with respect to Figures 2-3 The method described can be implemented by the system 101 and the system 100. Thus, it is noted that embodiments discussed in relation to the method described with respect to Figures 2-3 The method described and its possible implementations can be implemented by the system 101 and the system 100. Thus, it is noted that embodiments discussed in relation to the method described with respect to Figures 2-3 The method described and its possible implementations can be implemented by the system 101 and the system 100. Thus, it is noted that embodiments discussed in relation to the method described with respect to

[0073] Referring to Figure 2 a generalized flowchart of training a machine learning model that can be used to generate reference images for defect images of semiconductor samples is shown, in accordance with certain embodiments of the presently disclosed subject matter.

[0074] As noted above, the ML models referred to herein can be implemented as various types of machine learning models. In some embodiments, the ML models can be trained based on supervised learning. Supervised learning generally refers to training a learning model based on input data that has been labeled for a particular output. The model is trained until the model is able to detect potential patterns and relationships between the input data and the output labels, thereby enabling the model to produce accurate labeled results when provided with runtime data that the model has never seen before. In such cases, the ML model can be implemented as various models with a supervised learning pattern, such as, for example, a convolutional neural network (CNN), a support vector machine (SVM), a decision tree, and the like.

[0075] To train the ML model in supervised learning, a labeled training set is needed. According to certain embodiments of the disclosure, a training set (202) can be generated (e.g., by a training set generator 104 in the PMC 102), which includes one or more pairs of training images. Each pair of training images includes a defect image and a corresponding defect-free image. As used herein, a defect image refers to an image that includes or is highly likely to include a defect feature representing an actual defect on a specimen. A defect-free image refers to a clean image that has no defect feature, or is highly likely to not contain any defect feature. The defect-free image corresponds to the defect image in the sense that it captures a similar region containing a similar pattern as the defect image. The defect-free image is used as ground truth data associated with the defect image in the same pair of training images. The ML model is trained to learn a non-linear mapping relationship between the two populations of defect images and defect-free images.

[0076] In some cases, the training images (such as the defect image and the defect-free image in each pair of training images) can be “real-world” images (i.e., actual images) of a specimen acquired by an inspection tool during a manufacturing process of the semiconductor specimen. As an example, the defect image can be an inspection image captured by the tool from an inspection region of the specimen, which is verified to contain a defect feature. For example, the inspection image can be verified as a defect image containing a defect feature in a defect detection process (in which case the inspection image is considered to have a high probability of containing a defect feature), or in a defect review process, e.g., manually reviewed by a user, or automatically reviewed by an ADR process. The defect-free image can be captured by the tool from one or more reference regions (such as, for example, one or more neighboring dies of the inspected die in a D2D inspection) of the inspection region, which are known to be defect-free, or have a high probability of being defect-free. The defect-free image can also be any inspection image that is verified to not contain any defect feature.

[0077] To provide a trained, accurate model that is robust to different variations in actual production, the training images must be sufficient in terms of quantity, quality, and variance, among other things. However, in some cases, such training data can be difficult to collect. For example, defect training images of the samples to be inspected can not be available during a setup phase, or can be very rare and do not include enough defect features of the samples. In particular, it can also be challenging to obtain corresponding defect-free images as ground truth data, which often takes time and effort.

[0078] In some embodiments, in addition to “real world” training images, the training set used to train the ML model can be enriched by using one or more synthetic images simulated for the semiconductor sample.

[0079] In some cases, at least one of the defect image and / or the defect-free image in the pair of training images can be a simulated image. As an example, the defect-free image can be simulated based on design data of the semiconductor sample. The design data can refer to original design layout data, such as CAD data, of the sample obtained from a customer. Alternatively, the design data can refer to simulated design data generated based on image data of the sample. As an example, a SEM image of the sample can be analyzed and feature extraction can be performed on the SEM image. For example, the extracted features can include features representing structures and / or patterns of structural elements, such as, for example, edges, corners, pixel intensities, etc., and optionally one or more measurements about certain structural elements. The extracted features can be used to simulate a design image (e.g., a simulated CAD image) corresponding to the SEM image.

[0080] After obtaining design data (such as original design layout or simulated design data, such as CAD) of the sample, image simulation can be performed based on the design data to simulate one or more effects caused by one or more physical processes of the semiconductor sample, thereby producing a defect-free image. According to certain embodiments, the effects can refer to changes caused by one or more of the following physical processes: fabrication / manufacturing processes of the sample (e.g., printing the design pattern of the sample on a wafer by a lithography tool), inspection processes (e.g., scanning processes and signal processing processes of an inspection tool), etc. As an example, the effects caused by the manufacturing processes can be simulated to represent how the design pattern in the design image actually appears on the wafer. In other words, the simulation transfers the design intent layout to the intended processed pattern on the wafer. In some cases, process variations (PVs) can be considered during such simulation. Optionally, in addition to the effects caused by the manufacturing processes as described above, grayscale (GL) variations caused by the inspection processes of the tool can also be simulated. Different variants are described in more details below.

[0081] In some embodiments, the defect image can be simulated by synthetically implanting the defect features on a defect-free image (either an actual defect-free image or a simulated image). Various image enhancement ways for implanting synthetic defects on an image based on certain features of the defect (such as, for example, type and expected location, etc.) can be used, and the present disclosure is not limited to a particular implementation.

[0082] Continuing Figure 2 As described in the Background, the training of the ML model includes, for each pair of training images of one or more pairs of training images, processing (204) the defect image by the ML model (e.g., the ML model 108) to obtain a predicted image, and optimizing (206) the ML model (e.g., by the training model 106 in the PMC 102) to minimize the difference between the predicted image and the defect-free image.

[0083] Figure 4 A schematic diagram illustrating an exemplary training process of a ML model in accordance with certain embodiments of the presently disclosed subject matter is shown.

[0084] An example pair of training images including a defect image 402 and a defect-free image 404 is illustrated. As shown, the defect image 402 includes a defect feature 403 (such as, for example, a bridge formed between two line structures). The defect-free image 404 corresponds to the defect image 402 (e.g., it captures a region having a similar pattern to the defect image), and does not include any defect feature. In some embodiments, the defect image and the defect-free image in each pair of training images can be pre-processed (406) between the operations described in blocks 202 and 204, and before being fed to the ML model for training the model. The pre-processing can include one or more of the following operations: image registration, noise filtering, and image enhancement. Figure 2

[0085] In some embodiments, the pre-processing can include image registration between the pair of defect image and defect-free image. As an example, the image registration can be performed in order to correct any misalignment between the two images. In some cases, the image registration can include global registration and / or local registration. The global registration aims to address misalignment caused by various factors such as, for example, navigation errors of the tool, drift caused by certain physical effects (such as charging effects and / or shrinkage of the sample, etc.), and the like. The global registration can be implemented in accordance with any suitable registration algorithm known in the art. As an example, the global registration can be performed by determining a geometric transformation between the two images, such as, for example, by determining transformation parameters representing an affine transformation using an affine registration algorithm, including translation, rotation, scaling, shear mapping, and the like, and any combination thereof in any order. For example, as one affine registration, a phase correlation registration (PCR) can be performed using a frequency domain analysis (where a phase difference in the Fourier domain is converted to a registration in the image domain).​

[0086] Local registration aims to address misalignment caused by various changes between images, such as, for example, process variations. As an example, in some cases, registration can be performed using one or more of the following registration algorithms: region-based algorithms, feature-based registration, or phase correlation registration. One example of a region-based approach is registration using optical flow, such as the Lucas-Kanade (LK) algorithm or the Horn and Schunk algorithm. Feature-based approaches are based on finding different points of information (“features”) in two images and computing the required transformation between each pair of images from the correspondence of the features. This allows for elastic registration (i.e., non-rigid registration), where different regions move independently.

[0087] In some embodiments, pre-processing can include performing noise filtering on pairs of defect images and defect-free images to reduce noise representing various changes caused by one or more physical processes of the specimen, such as, for example, gray level (GL) variations, process variations (PV), and the like.

[0088] GL variations can be caused by physical effects of the specimen inspection process. As an example, the gray level of images taken from different locations of the same wafer can vary. Additionally, there can be some difference in the gray level of images taken by different inspection tools, even when the images are captured at the same location of the same wafer at the same resolution. In some cases, GL variations can include color variations (CV). Color variations can occur within a single image (e.g., due to layer thickness variations), or between two images, such as between an inspection image and a reference image (or a defect image and a corresponding defect-free image). GL variations and CV can be caused by various factors, such as, for example, different physical configurations and calibrations of the inspection tools / detectors, and the like, and can be characterized by local variations in the gray level intensity in the images.

[0089] In some cases, pattern variations can occur in two images, which can be caused by effects of process variations (PV). Process variations can refer to variations caused by changes in the manufacturing process of the specimen. As an example, the manufacturing process can cause slight shifts / scales / warps of certain structures / patterns between different inspection images, which results in pattern variations in the images. As another example, the manufacturing process can cause thickness variations of the specimen, which affect reflectivity, which in turn affects the gray level of the resulting inspection images. For example, inter-die material thickness variations can cause the reflectivity between two of the dies in the die to be different, which results in the images of the two dies having different background gray level values.

[0090] In some cases, additionally or alternatively, other possible effects of the physical process can include, but are not limited to: random noise, focus errors (e.g., some patterns are more sensitive to focus errors than others), charging effects (which can create artificial differences between the inspection image and the reference image and can significantly affect the inspection process), calibration of the four detectors (which can be suboptimal, leading to images overlapping at different viewing angles), etc.

[0091] Training the ML model based on images with such variations can degrade the generalization capability of the ML model, as some differences learned by the ML model between the defect images and the defect-free images can be due to these variations rather than actual defect features. As a result, the trained ML model can be affected by these variations and provide unstable prediction outputs. The reference images generated by the ML model when used for defect detection can provide detection results with less than ideal detection sensitivity.

[0092] To reduce the impact of these variations, in some embodiments, the defect images and the defect-free images can be pre-processed to filter out noise representing one or more of the above variations. As an example, one type of such noise representing a background grayscale variation between two images can be estimated, e.g., by modeling a systematic transformation of pixel values from one image to the other. For example, the transformation can be represented by a polynomial relationship between the pixel values of the two images. As another example, another type of noise representing a pattern variation can be estimated, e.g., by modeling a spatial shift / scaling between pixel values constituting the patterns in the two images. In addition to or instead of the above methods, other noise filtering methods can also be performed on the two images. Such image pre-processing enables increasing the robustness of the trained ML model to different variations of the runtime images.

[0093] In some further embodiments, the pre-processing can include performing image enhancement on pairs of defect images and defect-free images. As an example, the image enhancement can be performed using an image inpainting technique. Image inpainting refers to the reconstruction of missing parts of an image. For example, the defect image can be segmented into a set of sub-regions of n x n pixel size using a grid. The set of sub-regions is then randomly divided into m disjoint subsets. For each subset, the sub-regions belonging to that subset are removed from the defect image, thereby resulting in m defect sub-images, each having a subset of removed sub-regions.

[0094] Accordingly, each defect image is converted into m defect sub-images corresponding to the various portions of the defect-free image, thereby enriching the training set in the sense of the number of input images processed by the ML model. For each defect sub-image, the ML model reconstructs the removed sub-region in the defect sub-image as an image inpainting network, thereby producing an output sub-image containing the reconstructed sub-region missing in the corresponding input defect sub-image. By processing the m defect sub-images, the ML model will generate m reconstructed sub-images. The m reconstructed sub-images can be combined into a single reconstructed image. The reconstructed image will be evaluated with respect to the defect-free image.

[0095] It is noted that in some cases, the image pre-processing can include at least one of image registration, noise filtering, or image enhancement as described above. For example, the image pre-processing can include both image registration and noise filtering. In some other cases, the image pre-processing can include all three types of pre-processing operations that can be performed in any suitable order, such as, for example, image registration, followed by noise filtering, and then image enhancement.

[0096] After the image pre-processing 406, the processed defect image is fed into the ML model 408 for processing. The output of the ML model 408 is a predicted image 410. The predicted image 410 is evaluated with respect to the processed defect image (which serves as ground truth data for the predicted image) using a loss function 412 (also referred to as a cost function). The loss function 412 can be a difference metric configured to represent the difference between the predicted image and the defect-free image. The ML model 408 can be optimized by minimizing the value of the loss function 412. As an example, the ML model 408 can be optimized using a loss function such as, for example, mean squared error (MSE), sum of absolute differences (SAD), structural similarity index measure (SSIM), or an edge-preserving loss function. It is noted that the term “minimize” or “minimizing” used herein refers to attempting to reduce the difference value represented by the loss function to a certain level / degree (which can be pre-defined), but not necessarily to reach an actual minimum value.

[0097] As noted above, the ML model can be implemented as various supervised learning models. In one embodiment, the ML model can be implemented as a convolutional neural network (CNN). A CNN generally has a structure that includes an input layer and an output layer, as well as multiple hidden layers. The hidden layers of a CNN typically contain a series of convolutional layers that convolve with a multiplication or other dot product, followed by additional layers such as pooling layers, fully connected layers, and normalization layers. In some cases, a CNN can be viewed as consisting of two main functions: feature extraction and classification. As an example, the feature extraction portion can include several convolutional layers, followed by max pooling and activation functions. The classification portion generally includes fully connected layers. As an example, in some cases, certain intermediate layers in the feature extraction portion of a supervised model (e.g., one or more of the convolutional layers) can provide a layer output in the form of a feature map (also referred to as a feature vector). For example, an output feature map can be generated, e.g., by convolving each filter of a particular layer over the width and height of an input feature map, computing a dot product between the entries of the filter and the input, and generating a two-dimensional activation map that gives the response of that filter at each spatial location. Stacking the activation maps of all filters along the depth dimension forms the complete output feature map of the convolutional layer. As a result, the CNN learns a filter that activates when it detects some particular type of feature in the input at a certain spatial location.

[0098] Training of a CNN can be performed by iteratively adjusting the weights and / or thresholds of the CNN during each epoch using backpropagation such that the predictions match the ground truth (i.e., the loss function is minimized to some level / threshold). With each epoch of weight adjustment, the CNN improves in prediction accuracy and eventually realizes an optimal set of weights that meet the performance requirements in the trained model.

[0099] In cases where the defective image is pre-processed by image enhancement, such as by using an image inpainting technique, the CNN functions as an image inpainting network as described above. The m reconstructed sub-images output by the CNN can be combined into a single reconstructed image. The reconstructed image will be evaluated relative to the defect-free image. Use of an image inpainting network in such cases can benefit from the ability of image inpainting networks to predict normal patterns that has been demonstrated, improve the prediction performance of the trained ML model in predicting the defect-free reference image, and enhance the robustness of the trained ML model relative to various defect features.

[0100] In some embodiments, the functions of the pre-processing operations, such as image registration, noise filtering, etc., as described above, can be implemented at least partially as part of the ML model. As an example, the ML model can include one or more learning modules (e.g., as a pre-CNN module or a pre-DNN module) configured to perform image registration and / or noise filtering on pairs of defect images and defect-free images. In such cases, the ML model can be viewed as a network ensemble that includes a registration learning module and / or a noise filtering learning module positioned before the CNN module.

[0101] For example, the registration learning module can be trained with or separate from the CNN, using pairs of original training images of defect images and defect-free images as input, and using their corresponding registered pairs of images as ground truth. Similarly, the noise filtering learning module can also be trained with or separate from the CNN, using at least one of the defect images and defect-free images as input, and using at least one corresponding image with reduced noise as ground truth. The present disclosure is not limited to the specific implementation of such functions, whether as pre-processing modules or as part of the ML model.

[0102] The pre-processing of training image pairs, whether implemented as a separate pre-processing module before the ML model or as an inherent part of the ML model itself, enables image registration and / or noise filtering between the image pairs, thereby eliminating the effects of these variations on the trained ML model. The reference images predicted by the ML model should have improved alignment with the runtime images, and share similar noise levels with the runtime images. Using such generated reference images for runtime defect inspection can reduce image pre-processing efforts, such as image registration and noise filtering, as would be required if the runtime images and separately acquired reference images with inherent misalignment and variations were processed as previously. For example, in many cases, the generated reference images can be used for direct comparison with the runtime images without the need for pre-processing for image registration and / or noise filtering with respect to the runtime images. The use of such generated reference images in defect inspection not only improves defect detection throughput, but also increases detection sensitivity (in terms of higher capture rate and lower false positive rate).

[0103] Once the ML model is trained, the ML model can be tested and validated using an additional data set. If the prediction performance during testing and validation meets the requirements, the trained ML model can be deployed at runtime for defect inspection.

[0104] Turning now to Figure 3 shows a generalized flowchart of runtime defect inspection using a machine learning model trained to generate reference images, in accordance with certain embodiments of the presently disclosed subject matter.

[0105] A runtime image of the semiconductor sample can be obtained (302) during runtime inspection of the sample (e.g., by inspection tool 120).

[0106] The runtime image can be provided as input to a machine learning model (e.g., ML model 108 in PMC 102) for processing. A reference image can be generated (304) based on the runtime image as output of the ML model. The ML model was previously trained during setup using a training set including one or more pairs of training images, each pair including a defective image and a corresponding non-defective image, as described above with reference to Figure 2 and Figure 4 As described above, the ML model can be implemented using various learning models.

[0107] Defect inspection can be performed (306) on the runtime image using the generated reference image (e.g., by defect inspection module 110 in PMC 102). As described above, the generated reference image is believed to be aligned with the runtime image and share a similar noise level with the runtime image. Defect inspection can be performed by directly comparing the runtime image with the generated reference image (without pre-processing such as registration and noise filtering) to obtain defect inspection results indicative of a distribution of defects on the semiconductor sample. Defect inspection can refer to one or more of defect detection, defect review, and defect classification.

[0108] As described above, defect detection refers to capturing inspection images of a sample (such as, for example, in D2D inspection) and detecting potential defects based on the images according to a defect detection algorithm. For each inspection image, the trained ML model can process the inspection image and generate a corresponding reference image. The reference image can be used for comparison with the inspection image, resulting in a defect map indicative of a distribution of defect candidates on the semiconductor sample. In some cases, a list of defect candidates can be further selected from the defect map as candidates that have a higher probability of being defects of interest (DOIs).

[0109] After defect detection, defect candidates can be provided to a defect review tool (such as, for example, ADR). The review tool is configured to capture review images (typically with higher resolution) at respective defect candidate locations, and review the review images to determine whether the defect candidates are indeed DOIs. Similarly, in such cases, the trained ML model can be used to process the review images and generate corresponding reference images. The reference images can be used for comparison with the review images. The output of the review tool can include label data respectively associated with the defect candidates, which provides information of whether each defect candidate is a DOI.

[0110] In some cases, a defect classification tool (such as, for example, an ADC) is used in addition to or instead of a defect review (DR) tool. As an example, the classification tool can provide label data that provides information of whether each defect candidate is a DOI, and for those defect candidates that are labeled as a DOI, also provides information of the class or type of DOI. A trained ML model can be used in a similar manner here.

[0111] Figure 5 A schematic diagram of an exemplary runtime inspection process of a trained ML model according to certain embodiments of the presently disclosed subject matter is shown. A runtime image 502 of a specimen is acquired by an inspection tool. The runtime image 502 is fed into a trained ML model 504 for processing. The ML model 504 has been previously trained according to the descriptions of Figure 2 and Figure 4 . After processing the image 502, the ML model 504 provides a reference image 506 as output. The runtime image 502 is provided together with the reference image 506 to a defect inspection module, such as an exemplary ADR / ADC module 508, for defect inspection of the specimen.

[0112] According to certain embodiments, alternatively, the ML model can be trained based on unsupervised learning. Unsupervised learning generally refers to learning patterns from unlabelled data. In unsupervised learning, the ML model learns to recognize patterns from unsorted information according to similarity and difference, even without providing classes or labels. The ML model can be implemented as various models with unsupervised learning patterns, such as, for example, an autoencoder (AE), a variational autoencoder (VAE), a generative adversarial network (GAN), and a visual transformer (ViT).

[0113] To train the ML model in unsupervised learning, a training set without labels can be provided. According to certain embodiments of the present disclosure, the training set can be generated (e.g., by the training set generator 104 in the PMC 102) that includes a plurality of no-image. As described above, a no-defect image refers to a clean image that has no defect features, or is likely to not contain any defect features. The no-defect images can be collected from “real-world” images of the specimen, or alternatively, can be simulated based on design data of the specimen for at least a portion of the images, as described above.

[0114] As an example, in unsupervised learning, the ML model can be implemented as an autoencoder (AE) or a variant thereof (e.g., VAE). An autoencoder is a type of neural network that is typically used for data reproduction by learning an efficient data encoding and reconstructing its input (e.g., minimizing the difference between the input and the output). An autoencoder has an input layer, an output layer, and one or more hidden layers connecting them. Generally, an autoencoder can be viewed as comprising two parts: an encoder and a decoder. The autoencoder learns to compress data from the input layer into a short code (i.e., the encoder part), and then decompresses the code into an output that closely matches the original data (i.e., the decoder part). The output of the encoder is referred to as a code, a latent variable, or a latent representation that represents the input image. The code can pass through the hidden layers in the decoder and can be reconstructed in the output layer as an output image that corresponds to the input image.

[0115] For each input defect-free image in the training set, the autoencoder can extract features representing the input image and use the representative features to reconstruct a corresponding output image, which can be evaluated by comparison with the input image. The autoencoder is trained and optimized so as to learn representative features in the input training images (e.g., the features can represent structural elements, patterns, pixel distribution, etc. in the training images). Since the training images are defect-free images, the autoencoder is trained to learn the distribution of normal patterns and features of the defect-free images.

[0116] Once the autoencoder is trained based on the training set, the trained autoencoder is able to generate a reconstructed output image that closely matches the input based on its latent representation for each runtime image. Since the autoencoder is trained using only defect-free images, it will not be able to reconstruct abnormal patterns (defect patterns) that were not observed during training. In the case where the runtime image is a defect image, the autoencoder will reconstruct a corresponding defect-free image of the defect image. Thus, the trained autoencoder can be used at runtime to generate a reference image for the runtime image.

[0117] However, in some cases, certain autoencoders can have high generalization capabilities and thus can reconstruct some of the defect features in the output image regardless. This violates the assumption described above and makes it impossible to distinguish between defect features and defect-free features when using the generated reference image for defect inspection.

[0118] In such cases, the image inpainting technique as described above can be used for image reconstruction. Image inpainting refers to the reconstruction of missing parts of an image. For example, a grid can be used to segment the defect-free image into a set of sub-regions of n x n pixel size. This set of sub-regions is then randomly divided into m disjoint subsets. For each subset, the sub-regions belonging to that subset are removed from the defect-free image, thereby resulting in m defect-free sub-images, each having the subset of sub-regions removed.

[0119] For each defect-free sub-image, the ML model reconstructs the removed sub-regions as an image inpainting network, thereby resulting in an output sub-image containing the reconstructed sub-regions missing in the corresponding input sub-image. The missing sub-regions are reconstructed based only on their immediate neighborhood pixels, excluding the original pixels in these regions. Thus, there is a very low likelihood of reconstructing the defect features by generalizing their neighborhood appearance (containing normal features). Additionally, since the ML model is trained using only defect-free images, the reconstruction of defect-free regions can be well-modeled. Thus, using the image inpainting network can improve the reconstruction performance of normal patterns, thereby improving the ability to detect defect features using the reconstructed images.

[0120] It is well-known that defects on a semiconductor sample have different characteristics from typical anomalies on an object. For example, defects are typically small in size and can often present a similar appearance to normal patterns. In some cases, even with the image inpainting network as described above, some of the defects can still be reconstructed by the network (e.g., when the image is randomly segmented into sub-images with removed sub-regions, the defect features can not be completely removed), thereby affecting the stability and accuracy of defect detection.

[0121] In contrast, as described above with respect to Figure 2 and Figure 4 training the ML model using training image pairs of defect images and defect-free images in supervised learning enables the ML model to learn and rank based on the ground truth data provided, thereby allowing the accuracy of the supervised learning output to be verified in a way that is not possible with unsupervised learning. The proposed supervised learning using a specially configured training set can ensure the accuracy and stability of the prediction performance, thereby generating satisfactory reference images that can be used to enhance downstream defect inspection.

[0122] On the other hand, unsupervised learning can identify previously unknown patterns in the data. It is easier, faster, and less costly than supervised learning because unsupervised learning does not require associating the training inputs with the label data required for supervised learning, which can be difficult to achieve in some cases. For example, defect images are relatively less compared to defect-free images. Using only defect-free images as training data can make data preparation much easier compared to preparing image pairs including corresponding defect images and defect-free images.

[0123] It should be noted that the above examples of CNNs, autoencoders, or image inpainting networks are described for exemplary and illustrative purposes only, and should not be considered limiting the present disclosure in any way. Other types of supervised and unsupervised models can also be used in addition to or instead of the above models.

[0124] It should be noted that the examples shown in the present disclosure, such as, for example, pre-processing techniques, exemplary algorithms for image registration, noise filtering, and / or image enhancement, exemplary image simulation techniques, etc., are shown for exemplary purposes and should not be considered limiting the present disclosure in any way. Other appropriate examples / implementations can also be used in addition to or instead of the above examples.

[0125] One of the advantages of certain embodiments of the presently disclosed subject matter as described herein is that instead of additional image acquisition of actual reference images, a machine learning model is provided that is capable of generating synthetic reference images of runtime images, and performing defect inspection operations using the generated reference images. The proposed system significantly reduces the image acquisition time of the inspection tool.

[0126] Additionally, the pre-processing of the training image pairs, whether implemented as a separate pre-processing module prior to the ML model or as an inherent part of the ML model itself, enables image registration and / or noise filtering between the image pairs. The reference images predicted by the ML model should have improved alignment with the runtime images and share similar noise levels with the runtime images. Using such generated reference images can reduce the image pre-processing effort required to previously process the runtime images, as well as the separately acquired reference images with inherent misalignment and variations. Using such generated reference images in defect inspection not only improves defect detection throughput, but also increases detection sensitivity.

[0127] One of the further advantages of certain embodiments of the presently disclosed subject matter as described herein is that the use of training image pairs of defect images and defect-free images in specific configurations in supervised learning to train the ML model ensures the accuracy and stability of the predictive performance of the model, thereby providing satisfactory reference images that can be used to enhance downstream defect inspection.

[0128] One of the further advantages of certain embodiments of the presently disclosed subject matter as described herein is that sufficient training data is generated for training the ML model, including simulated defect images and / or defect-free images. The image simulation can take into account one or more effects / changes caused by one or more physical processes, such as manufacturing processes and / or inspection processes of the sample, thereby increasing the diversity of the training data with controlled amounts of variance.

[0129] It should be understood that the disclosure is not limited in its application to the details set forth in the description contained herein or in the examples shown in the drawings.

[0130] It should also be understood that the system according to the disclosure can be implemented at least partially in a suitably programmed computer. Likewise, the disclosure contemplates a computer program being readable by a computer for executing the methods disclosed by the disclosure. The disclosure further contemplates a non-transitory computer- readable storage medium having stored thereon a program of instructions executable by a computer to perform the method disclosed by the disclosure.

[0131] The disclosure is capable of other embodiments and of being practiced or being carried out in various ways. Accordingly, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, can readily be utilized as a basis for the designing of other structures, methods and systems for carrying out the several purposes of the present disclosed subject matter.

[0132] It will be readily understood to those skilled in the art that various modifications and changes can be applied to the embodiments of the present disclosure described above without departing from the scope of the present disclosure defined by the appended claims.

Claims

1. A computerized system for defect inspection of a semiconductor specimen, the system comprising processing and memory circuitry (PMC) configured to: acquire a runtime image of the semiconductor specimen; generate a reference image based on the runtime image using a machine learning (ML) model, wherein the ML model is previously trained during setup using a training set comprising one or more pairs of training images, each pair of training images comprising a defect image and a corresponding defect-free image, the training comprising, for each pair of training images: processing the defect image by the ML model to obtain a predicted image; and optimizing the ML model to minimize a difference between the predicted image and the defect-free image, wherein the defect image and the defect-free image of each pair of training images are pre-processed prior to the training by performing image enhancement on the defect image and the defect-free image using inpainting, resulting in a plurality of defect sub-images corresponding to respective portions of the defect-free image, and wherein each defect sub-image has a subset of removed sub-regions, and the ML model reconstructs the subset of removed sub-regions in the defect sub-images as an inpainting network; and performing defect inspection on the runtime image using the generated reference image.

2. The computerized system of claim 1, wherein the ML model is a deep neural network (DNN) configured to learn a non-linear mapping between two populations of defect images and defect-free images.

3. The computerized system of claim 1, wherein the defect image in each pair of training images is one of: an actual image of the semiconductor specimen acquired by an inspection tool and verified to contain defect features, or a simulated image resulting from synthetically implanting defect features on a defect-free image, and wherein the corresponding defect-free image in each pair of training images is one of: an actual image of the semiconductor specimen acquired by an inspection tool and verified to not contain defect features, or a simulated image generated based on design data of the semiconductor specimen.

4. The computerized system of claim 1, wherein the defect image and the defect-free image of each pair of training images are pre-processed prior to the training by registering the defect image with the defect-free image to correct for misalignment between the defect image and the defect-free image.

5. The computerized system of claim 1, wherein the defect image and the defect-free image of each pair of training images are pre-processed prior to the training by performing noise filtering on the defect image and the defect-free image to reduce noise representing variations caused by one or more physical processes of the specimen.

6. The computerized system of claim 1, wherein the defect image and the defect-free image of each pair of training images are pre-processed prior to the training by performing image registration, noise filtering, and image enhancement on the defect image and the defect-free image.

7. The computerized system of claim 1, wherein the ML model comprises at least one learning module preceding a DNN, the at least one learning module trained to perform at least one of image registration and noise filtering on the defect image and the defect-free image prior to processing each pair of training images by the DNN.

8. The computerized system of claim 1, wherein the ML model is optimized using a loss function selected from the group consisting of: mean squared error (MSE), sum of absolute differences (SAD), structural similarity index measure (SSIM), and an edge-preserving loss function.

9. The computerized system of claim 1, wherein the defect inspection is one of: defect detection, defect review, and defect classification.

10. The computerized system of claim 1, wherein the generated reference image has improved alignment with the runtime image and shares similar noise levels with the runtime image, and the PMC is configured to perform defect inspection by directly comparing the runtime image with the generated reference image to obtain defect inspection results indicative of defect distribution on the semiconductor sample.

11. A computerized method for defect inspection of a semiconductor sample, the method comprising the steps of: acquiring a runtime image of the semiconductor sample; generating a reference image based on the runtime image using a machine learning (ML) model, wherein the ML model was previously trained during setup using a training set comprising one or more pairs of training images, each pair of training images comprising a defect image and a corresponding defect-free image, the training comprising, for each pair of training images: processing the defect image by the ML model to obtain a predicted image; and optimizing the ML model to minimize a difference between the predicted image and the defect-free image, wherein the defect image and the defect-free image are pre-processed prior to the training by performing image enhancement on the defect image and the defect-free image of each pair of training images using image inpainting, thereby producing a plurality of defect sub-images corresponding to respective portions of the defect-free image, and wherein each defect sub-image has a subset of removed sub- regions, and the ML model reconstructs the subset of removed sub-regions in the defect sub- images as an image inpainting network; and performing defect inspection on the runtime image using the generated reference image.

12. The computerized method of claim 11, wherein the defect image and the defect- free image of each pair of training images are pre-processed by performing image enhancement on the defect image and the defect-free image using image inpainting, thereby producing a plurality of defect sub-images corresponding to respective portions of the defect-free image, and wherein each defect sub-image has a subset of removed sub-regions, and the ML model reconstructs the subset of removed sub-regions in the defect sub-images as an image inpainting network.

13. The computerized method of claim 11, wherein the ML model is optimized using a loss function selected from the group consisting of: mean squared error (MSE), sum of absolute differences (SAD), structural similarity index measure (SSIM), and an edge-preserving loss function.

14. The computerized method of claim 11, wherein the defect inspection is one of: defect detection, defect review, and defect classification.

15. The computerized method of claim 11, wherein the generated reference image has improved alignment with the runtime image and shares similar noise levels with the runtime image, and the PMC is configured to perform defect inspection by directly comparing the runtime image with the generated reference image to obtain defect inspection results indicative of defect distribution on the semiconductor sample.

12. The computerized method of claim 11, wherein the defect image in each pair of training images is one of: an actual image of the semiconductor sample acquired by an inspection tool and verified to contain defect features, or a simulated image obtained by synthetically implanting defect features on a defect-free image, and wherein the corresponding defect-free image in each pair of training images is one of: an actual image of the semiconductor sample acquired by an inspection tool and verified to not contain defect features, or a simulated image generated based on design data of the semiconductor sample.

13. The computerized method of claim 11, wherein the defect image and the defect-free image in each pair of training images are pre-processed prior to the training by registering the defect image with the defect-free image to correct for misalignment between the defect image and the defect-free image.

14. The computerized method of claim 11, wherein the defect image and the defect-free image in each pair of training images are pre-processed prior to the training by performing noise filtering on the defect image and the defect-free image to reduce noise representing variations caused by one or more physical processes of the sample.

15. The computerized method of claim 11, wherein the ML model includes at least one learning module prior to a DNN, the at least one learning module trained to perform at least one of image registration and noise filtering on the defect image and the defect-free image in each pair of training images prior to processing the defect image and the defect-free image by the DNN.

16. The computerized method of claim 11, wherein the defect inspection is one of: defect detection, defect review, and defect classification.

17. The computerized method of claim 11, wherein the generated reference image has improved alignment with the runtime image and shares similar noise level with the runtime image, and the step of performing defect inspection includes directly comparing the runtime image with the generated reference image to obtain defect inspection results indicative of defect distribution on the semiconductor sample.

18. A non-transitory computer-readable storage medium tangibly embodying a program of instructions, the program of instructions, when executed by a computer, causing the computer to perform a method for defect inspection of a semiconductor sample, the method comprising the steps of: acquiring a runtime image of the semiconductor sample; generating a reference image based on the runtime image using a machine learning (ML) model, wherein the ML model was previously trained during setup using a training set comprising one or more pairs of training images, each pair of training images comprising a defect image and a corresponding defect-free image, the training comprising, for each pair of training images: processing the defect image by the ML model to obtain a predicted image; and optimizing the ML model to minimize a difference between the predicted image and the defect-free image.

19. The method of claim 18, wherein the defect image in each pair of training images is one of: an actual image of the semiconductor sample acquired by an inspection tool and verified to contain defect features, or a simulated image obtained by synthetically implanting defect features on a defect-free image, and wherein the corresponding defect-free image in each pair of training images is one of: an actual image of the semiconductor sample acquired by an inspection tool and verified to not contain defect features, or a simulated image generated based on design data of the semiconductor sample.

20. The method of claim 18, wherein the defect image and the defect-free image in each pair of training images are pre-processed prior to the training by registering the defect image with the defect-free image to correct for misalignment between the defect image and the defect-free image.

21. The method of claim 18, wherein the defect image and the defect-free image in each pair of training images are pre-processed prior to the training by performing noise filtering on the defect image and the defect-free image to reduce noise representing variations caused by one or more physical processes of the sample.

22. The method of claim 18, wherein the ML model includes at least one learning module prior to a DNN, the at least one learning module trained to perform at least one of image registration and noise filtering on the defect image and the defect-free image in each pair of training images prior to processing the defect image and the defect-free image by the DNN.

23. The method of claim 18, wherein the defect inspection is one of: defect detection, defect review, and defect classification.

24. The method of claim 18, wherein the generated reference image has improved alignment with the runtime image and shares similar noise level with the runtime image, and the step of performing defect inspection includes directly comparing the runtime image with the generated reference image to obtain defect inspection results indicative of defect distribution on the semiconductor sample. wherein the defect images and the defect-free images are pre-processed prior to the training by performing image enhancement on the defect images and the defect-free images in each pair of training images using image inpainting, resulting in a plurality of defect sub-images corresponding to respective portions of the defect-free images, and wherein each defect sub-image has a subset of removed sub-regions, and the ML model reconstructs the subset of removed sub-regions in the defect sub-images as an image inpainting network; and performing defect inspection on the runtime images using the generated reference images.

Images