Ensemble of deep learning models for defect review in mass production

Abstract

A method and system for detecting defects in a sample image are provided. One system includes a computer subsystem configured to train an ensemble of deep learning models by modifying one or more parameters of the ensemble until a pseudo-loss function determined based on the output of the ensemble is approximately equal to but less than or equal to 0.5. The computer subsystem is also configured to detect defects in a runtime sample image by inputting a runtime sample image into the trained ensemble and generating a runtime label for the runtime sample image indicating whether defects were detected in the runtime sample image, based on the output of the deep learning models in the trained ensemble.

Description

[Technical Field]

[0001] The present invention generally relates to methods and systems for detecting defects in images of samples. Specific embodiments relate to an ensemble of deep learning models for high-precision, high-throughput scanning electron microscope (SEM) defect review in mass production. [Background technology]

[0002] The following descriptions and examples are not considered prior art simply because they are included in this section.

[0003] Manufacturing semiconductor devices, such as logic devices and memory devices, typically involves processing a substrate, such as a semiconductor wafer, using a number of semiconductor manufacturing processes to form various features and multiple levels of the semiconductor device. For example, lithography is a semiconductor manufacturing process that involves transferring a pattern from a reticle to a resist that is placed on a semiconductor wafer. Additional examples of semiconductor manufacturing processes, but not limited to, include chemical mechanical polishing (CMP), etching, deposition, and ion implantation. Multiple semiconductor devices may be manufactured in an array on a single semiconductor wafer and then divided into individual semiconductor devices.

[0004] Inspection processes are used at various steps during the semiconductor manufacturing process to detect defects on samples, promoting higher yields and, consequently, higher profits in the manufacturing process. Inspection has always been a crucial part of semiconductor device manufacturing. However, as the dimensions of semiconductor devices decrease, inspection becomes even more critical for the successful manufacture of acceptable semiconductor devices, because even smaller defects can cause device failure.

[0005] Defect review typically involves re-detecting defects identified during the inspection process and generating additional information about the defects at higher resolution using either a high-magnification optical system or a scanning electron microscope (SEM). Therefore, defect review is performed at the individual locations on the sample where defects were detected during inspection. The high-resolution data on defects generated by defect review is better suited for determining defect attributes, such as shape, roughness, and more precise size information. Generally, defects can be classified into types more accurately based on the information determined by defect review than by inspection alone.

[0006] Measurement processes are also used at various steps during semiconductor manufacturing processes to monitor and control the process. Unlike inspection processes, which detect defects on a sample, measurement processes measure one or more characteristics of a sample that cannot be determined by currently used inspection tools. For example, a measurement process can be used to measure one or more characteristics of a sample, such as the dimensions of features formed on the sample during the process (e.g., line width, thickness), and as a result, the performance of the process can be determined from one or more characteristics. In addition, if one or more characteristics of a sample are unacceptable (e.g., the characteristics are outside a given range), the measured values ​​of one or more characteristics of the sample can be used to modify one or more parameters of the process so that additional samples produced by that process have acceptable characteristics.

[0007] The measurement process also differs from the defect review process in that defects detected during inspection are re-examined in the defect review; the measurement process can be performed at locations where no defects have been detected. In other words, unlike the defect review, the location on the sample where the measurement process is performed may be independent of the results of the inspection process performed on the sample. In particular, the location where the measurement process is performed may be selected independently of the inspection results. In addition, because the location on the sample where the measurement is performed may be selected independently of the inspection results, unlike the defect review where the location on the sample where the defect review is performed cannot be determined until the inspection results for the sample are generated and available, the location where the measurement process is performed may be determined before the inspection process is performed on the sample.

[0008] Advances in deep learning have made it an attractive framework for use in processes like those described above. For example, some inspection processes use a single convolutional neural network (CNN) for defect detection. In some of these methods, the network is trained using hundreds of examples, and approximately 10 -3 This achieves capturing at least 80% of defects with a false positive rate. Figure 1 shows the receiver operational characteristic (ROC) curves of an optimally trained neural network configured to detect defects from SEM images in the backend layer. Each curve represents a different imaging condition (EPP = electrons per pixel, XF = extraction field). To capture more than 80% of defects, this network needs to operate with a false positive rate of at least 0.3%.

[0009] In this method, a single neural network can be trained using positive samples, negative samples, and corresponding labels. Labels are images where the value is 0 in all regions except for a relatively small area surrounding a defect. Therefore, all negative samples are perfectly 0. The network may contain six or more convolutional layers and output a real-valued array the same size as the input image. The network is trained by minimizing the cross-entropy between the network output and the labels. Thus, the network is trained to produce images that exactly match the labels.

[0010] Inference is performed using SEM images from multiple detectors of the SEM and binary images of design patterns corresponding to the acquisition locations. This data is input to a neural network as a 3D array, with the third dimension containing multiple detector views and design clips of the image. The network outputs a 2D array of the same size as the input SEM image. This output assigns a numerical value between 0 and 1 to each pixel, with pixels close to 0 being less likely to be defects and pixels close to 1 being more likely to be defects. A single threshold T is selected, and the image is binarized as follows: for each pixel, if its value is below T, it is reassigned a value of 0; otherwise, it is assigned a value of 1.

[0011] Connected pixels with a value of 1 are grouped to reduce the effects of shot noise. If the resulting image contains any region with a value of 1, that region is classified as a defective area. Otherwise, it is ignored, and the corresponding area is classified as a non-defective area.

[0012] The currently used methods described above have several significant drawbacks. For example, they are intended for research and development in semiconductor manufacturing facilities, not for high-volume production (HVM). As a result, their performance falls far short of the performance required for defect rarity in mature HVM processes. Specifically, their false positive rate is approximately 100 times higher than the value required for HVM review. In addition, there is no theoretical basis or principle to guarantee that the method's performance will improve with increasing computational resources. Furthermore, the currently used methods are highly sensitive to the selection of optimization algorithm parameters and require costly tuning. [Prior art documents] [Patent Documents]

[0013] [Patent Document 1] U.S. Patent No. 8,664,594 [Patent Document 2] U.S. Patent No. 8,692,204 [Patent Document 3] U.S. Patent No. 8,698,093 [Patent Document 4] U.S. Patent No. 8,716,662 [Overview of the project] [Problems that the invention aims to solve]

[0014] Therefore, it is advantageous to develop systems and methods for detecting defects in images of samples that do not have one or more of the above-mentioned defects. [Means for solving the problem]

[0015] The following descriptions of various embodiments should not be construed as limiting the subject matter of the invention in the attached claims.

[0016] One embodiment relates to a system configured to detect defects in an image of a sample. The system includes a computer subsystem and one or more components executed by the computer subsystem, the components including an ensemble of deep learning (DL) models and a pseudo-loss function based on the output generated by the ensemble of DL models. The computer subsystem is configured to train the ensemble using a training data set, the training data set including training sample images and training labels indicating whether a defect was detected in the training sample images. Training includes changing one or more parameters of the ensemble until a pseudo-loss function determined based on the output of the ensemble is approximately equal to 0.5 but less than or equal to 0.5. The computer subsystem is also configured to detect defects in a runtime sample image by inputting the runtime sample image into the trained ensemble of DL models and generating a runtime label for the runtime sample image indicating whether a defect was detected in the runtime sample image based on the output of the DL models within the trained ensemble. The system may be further configured as described herein.

[0017] Another embodiment relates to a computer-implemented method for detecting defects in an image of a sample. The method includes the training and detection steps described above, which are executed by a computer subsystem. Each step of the method may be executed as further described herein. The method described above may include any other steps of any other method described herein and may be executed by any of the systems described herein.

[0018] Another embodiment relates to a non - transient computer - readable medium that stores program instructions for causing a computer - implemented method for detecting defects in an image of a sample to be executed on a computer system. The computer - implemented method includes the steps of the method described above. The computer - readable medium may be further configured as described herein. The steps of the computer - implemented method may be executed as further described herein. Additionally, a computer - implemented method executable by the program instructions may include any other steps of any other method described herein.

Brief Description of the Drawings

[0019] Further advantages of the present invention will become apparent to those skilled in the art by reference to the following detailed description of the preferred embodiments and the accompanying drawings.

[0020] [Figure 1] A plot of the receiver operating characteristic (ROC) curve of a trained neural network for detecting defects from a scanning electron microscope (SEM) image of a backend layer. [Figure 2] A schematic diagram showing a side view of an embodiment of a system configured as described herein. [Figure 2a] A schematic diagram showing a side view of an embodiment of a system configured as described herein. [Figure 3] A block diagram showing an embodiment of one or more components that can be executed by a computer subsystem of an embodiment of the system. [Figure 4] A plot of an example of a defect detection result using an SEM image from a backend layer achieved by the embodiment described herein. [Figure 5] A plot of an example of the sensitivity of performance to hyperparameter tuning for an embodiment of a single neural network and an ensemble of the DL models described herein. [Figure 6]This is a block diagram showing one embodiment of a non-temporary computer-readable medium that stores program instructions for causing a computer system to execute the computer implementation method described herein. [Modes for carrying out the invention]

[0021] While various modifications and alternative forms are possible for the present invention, specific embodiments are shown in the drawings as examples and described in detail herein. The drawings may not be to scale. However, it should be understood that the drawings and their detailed description are not intended to limit the invention to any particular form disclosed, but rather to encompass all modifications, equivalents, and alternatives that fall within the spirit and scope of the invention as defined by the appended claims.

[0022] Turning our attention to the drawings, it should be noted that they are not drawn to a consistent scale. In particular, the scale of some of the elements in the drawings is greatly exaggerated to emphasize the characteristics of those elements. Also, note that the drawings are not drawn to the same scale. Elements shown in two or more drawings that can be similarly constructed are indicated using the same reference number. Unless otherwise specified herein, any of the elements described and illustrated may include any suitable commercially available elements.

[0023] Generally, the embodiments described herein are configured to determine information about a sample for defect review applications, such as defect detection in images of the sample, and / or, optionally, other semiconductor-based applications such as measurement and inspection. For example, the embodiments described herein provide an ensemble of deep learning (DL) models, such as a deep neural network, for high-precision, high-throughput scanning electron microscopy (SEM) or other defect review in high-volume manufacturing (HVM). In one embodiment, the runtime sample image input to the ensemble of trained DL models is generated during a defect review process performed on the sample in the HVM process. In another embodiment, the trained and runtime sample images are generated by the imaging subsystem of the defect review tool. In a further embodiment, the trained and runtime sample images are generated by an electron beam-based imaging subsystem. In an additional embodiment, the runtime sample image input to the ensemble of trained DL models for any one location on the sample includes an image generated using multiple detectors of the imaging subsystem. All of these images may be generated by the imaging subsystems and tools described herein, as further described herein.

[0024] Optical inspection of patterned semiconductor wafers is performed at wavelengths far exceeding the size of the defects being detected. This is required by the fact that wafer material strongly interacts only with light of wavelengths above approximately 100 nm. Furthermore, optical inspection remains the only technique capable of scanning the entire wafer without damaging it within one hour. This throughput requirement is essential to maintain the yield necessary to keep HVM economics in check.

[0025] As a result, unseparated optical images are used as the first and essential filter in defect detection. The presence of defects in these images is typically determined by statistical anomaly detection algorithms. These algorithms must accept many false positives in order to find minute defects across the entire wafer. To distinguish between true detections and false positives, high-resolution SEMs are used to image the areas flagged by optical inspection. Due to the extremely small size of the defects, the number of flagged areas is expected to be in the millions. Therefore, automated algorithms are needed to review 1 to 5 million SEM images.

[0026] As used herein, the term “mass production process” is defined as a process that has been released for mass production of semiconductor devices due to relatively well-controlled characteristics, although it is not in the research and development stage, and these relatively well-controlled characteristics may be established through research and development or other means. In HVM, the process is sufficiently mature that notable defects (DOIs) are rare. DOIs may occur randomly on a wafer in quantities of 10 to 100. Therefore, the automated review algorithm needs to correctly identify 10 to 100 images containing defects. To facilitate human review of its output, the algorithm should not generate more than approximately 100 false positives.

[0027] In light of the above considerations, the embodiments described herein have been devised to provide detection techniques that can be used to satisfy one or more of the following requirements: One requirement is that true defects in the SEM image are captured with a probability of more than 80%. Another requirement is that false positive detection, i.e., reporting a defect when no defect exists, is not 10%. -5 The requirement is that the defect review must be completed within one hour. An additional requirement is that the defect review method must be trainable with approximately 100 positive and negative defect examples. Furthermore, the algorithm's performance should clearly improve with increasing computational resources.

[0028] In some embodiments, the sample is a wafer. The wafer may include any wafer well known in semiconductor technology. In some embodiments, this specification may describe one or more wafers, but embodiments are not limited to the sample that can be used. For example, embodiments described herein may be used for samples such as reticles, flat panels, personal computer (PC) boards, and other semiconductor samples.

[0029] Figure 2 shows one embodiment of a system configured to detect defects in an image of a sample. In some embodiments, the system 10 includes an imaging subsystem 100. The imaging subsystem includes, and / or is coupled to, computer subsystems, e.g., computer subsystem 36 and / or one or more computer systems 102. Generally, the imaging subsystem described herein includes at least an energy source, a detector, and a scanning subsystem. The energy source is configured to generate energy directed to the sample by the imaging subsystem. The detector is configured to detect energy from the sample and generate an output in response to the detected energy. The scanning subsystem is configured to change the location on the sample to which the energy is directed and the location on the sample to which the energy is detected. In one embodiment, as shown in Figure 2, the imaging subsystem is configured as an optical imaging subsystem.

[0030] In the light-based imaging subsystem described herein, the energy directed towards the sample includes light, and the energy detected from the sample includes light. For example, as shown in Figure 2, the imaging subsystem includes an illumination subsystem configured to direct light onto the sample 14. The illumination subsystem includes at least one light source, e.g., light source 16. The illumination subsystem is configured to direct light onto the sample at one or more angles of incidence, which may include one or more oblique angles and / or one or more normal angles. For example, as shown in Figure 2, light from light source 16 passes through an optical element 18 and then through a lens 20 to be directed onto the sample 14 at an oblique angle of incidence. The oblique angle of incidence may include any suitable angle of incidence, which may vary depending, for example, on the characteristics of the sample and the process being performed on the sample.

[0031] The illumination subsystem may be configured to direct light onto the sample at different angles of incidence and at different points in time. For example, the imaging subsystem may be configured to modify one or more properties of one or more elements of the illumination subsystem so that light can be directed onto the sample at angles of incidence different from those shown in Figure 2. In such an example, the imaging subsystem may be configured to move the light source 16, the optical element 18, and the lens 20 so that light can be directed onto the sample at different oblique or normal (or nearly normal) angles of incidence.

[0032] In some examples, the imaging subsystem may be configured to direct light onto the sample simultaneously at two or more angles of incidence. For example, the illumination subsystem may include two or more illumination channels, one of which may include a light source 16, an optical element 18, and a lens 20, as shown in Figure 2, and another illumination channel (not shown) may include similar elements, which may be different or identical in configuration, or may include at least one light source and, optionally, one or more other components as further described herein. When such light is directed onto the sample simultaneously with other light, one or more properties (e.g., wavelength, polarization, etc.) of the light directed onto the sample at different angles of incidence may be different, and as a result, the light produced by the illumination of the sample at different angles of incidence can be distinguished from one another by the detector.

[0033] In another example, the illumination subsystem may include only one light source (e.g., light source 16 shown in Figure 2), and the light from the light source may be split into different optical paths (e.g., based on wavelength, polarization, etc.) by one or more optical elements (not shown) of the illumination subsystem. Each of the light from the different optical paths may then be directed to the sample. Multiple illumination channels may be configured to direct light to the sample simultaneously or at different times (e.g., when sequentially illuminating the sample using different illumination channels). In another example, the same illumination channel may be configured to direct light to the sample at different times with different characteristics. For example, optical element 18 may be configured as a spectral filter, and the characteristics of the spectral filter may be changed in various different ways (e.g., by swapping one spectral filter with another), so that different wavelengths of light can be directed to the sample at different times. The illumination subsystem may have any other suitable configuration known in the art to direct light with different or the same characteristics sequentially or simultaneously to the sample at different or the same angle of incidence.

[0034] Light source 16 may include a broadband plasma (BBP) light source. Thus, the light generated by the light source and directed towards the sample may include broadband light. However, the light source may include any other suitable light source, for example, any suitable laser known in the art configured to generate light at any suitable wavelength. The laser may be configured to generate monochromatic or nearly monochromatic light. Thus, the laser may be a narrowband laser. The light source may also include a polychromatic light source that generates light at multiple discrete wavelengths or wavelength bands.

[0035] Light from the optical element 18 can be focused onto the sample 14 by the lens 20. Although the lens 20 is shown as a single refractive optical element in Figure 2, in practice the lens 20 may include multiple refractive and / or reflective optical elements, which, in combination, focus the light from the optical elements onto the sample. The illumination subsystem shown in Figure 2 and described herein may include any other suitable optical elements (not shown). Examples of such optical elements include, but are not limited to, polarization components, spectral filters, spatial filters, reflective optical elements, apodizers, beam splitters, and apertures, which may include any suitable optical elements known in the art. In addition, the system may be configured to change one or more elements of the illumination subsystem based on the type of illumination used for imaging.

[0036] The imaging subsystem may also include a scanning subsystem configured to change the position on the sample from which light is directed and detected, and possibly to scan the sample. For example, the imaging subsystem may include a stage 22 on which the sample 14 is positioned during imaging. The scanning subsystem may include any suitable mechanical and / or robotic assembly (including the stage 22) configured to move the sample, thereby directing light to different positions on the sample from which it can be detected. In addition, or alternatively, the imaging subsystem may be configured such that one or more optical elements of the imaging subsystem perform a scan of light on the sample, thereby directing light to different positions on the sample from which it can be detected. In the example where light is scanned on the sample, the light may be scanned on the sample in any suitable way, for example, a meandering path or a helical path.

[0037] The imaging subsystem further includes one or more detection channels. At least one of the detection channels includes a detector configured to detect light from a sample due to illumination of the sample by the imaging subsystem and to produce an output in response to the detected light. For example, the imaging subsystem shown in Figure 2 includes two detection channels, one formed by a condenser 24, an element 26, and a detector 28, and the other formed by a condenser 30, an element 32, and a detector 34. As shown in Figure 2, the two detection channels are configured to collect and detect light at different focusing angles. In some cases, both detection channels are configured to detect scattered light, and the detection channels are configured to detect light scattered from the sample at different angles. However, one or more of the detection channels may be configured to detect another type of light from the sample (e.g., reflected light).

[0038] As further shown in Figure 2, both detection channels are shown positioned within the plane of the paper, and the illumination subsystem is also shown positioned within the plane of the paper. Thus, in this embodiment, both detection channels are positioned within the plane of incidence (e.g., at the center). However, one or more detection channels may be positioned outside the plane of incidence. For example, a detection channel formed by the concentrator 30, element 32, and detector 34 may be configured to collect and detect light scattered outside the plane of incidence. Thus, such a detection channel may be commonly referred to as a “side” channel, and such a side channel may be positioned at the center in a plane substantially perpendicular to the plane of incidence.

[0039] Figure 2 shows one embodiment of an imaging subsystem including two detection channels, but the imaging subsystem may include a different number of detection channels (e.g., only one detection channel or two or more detection channels). In such an example, the detection channel formed by the condenser 30, element 32, and detector 34 may form one side channel as described above, and the imaging subsystem may include an additional detection channel (not shown) formed as another side channel located on the opposite side of the incident plane. Thus, the imaging subsystem may include a detection channel which includes a condenser 24, element 26, and detector 28, located in the center of the incident plane and configured to collect and detect light at a scattering angle perpendicular or nearly perpendicular to the sample plane. Thus, this detection channel may generally be called the “top” channel, and the imaging subsystem may also include two or more side channels configured as described above. In this way, the imaging subsystem may include at least three channels (i.e., one top channel and two side channels), each of the at least three channels having its own condenser, each condenser configured to collect light at a different scattering angle than each of the other condensers.

[0040] Furthermore, as described above, each detection channel included in the imaging subsystem may be configured to detect scattered light. Therefore, the imaging subsystem shown in Figure 2 may be configured for dark-field (DF) imaging of a sample. However, the imaging subsystem may further, or alternatively, include detection channels configured for bright-field (BF) imaging of a sample. In other words, the imaging subsystem may include at least one detection channel configured to detect light specularly reflected from the sample. Therefore, the imaging subsystems described herein may be configured for DF imaging only, BF imaging only, or both DF and BF imaging. Although each of the condensers is shown as a single refractive optical element in Figure 2, each of the condensers may include one or more refractive optical elements and / or one or more reflective optical elements.

[0041] One or more detection channels may include any suitable detector known in the art, such as a photomultiplier tube (PMT), a charge-coupled device (CCD), and a time-delay integral (TDI) camera. Detectors may also include non-imaging detectors or imaging detectors. If the detectors are non-imaging detectors, each detector may be configured to detect certain properties of scattered light, such as intensity, but not necessarily as a function of position in the imaging plane. Thus, the output produced by each detector included in each detection channel of the imaging subsystem may be a signal or data, but not necessarily an image signal or image data. In such cases, a computer subsystem, such as computer subsystem 36, may be configured to generate an image of the sample from the non-imaging output of the detectors. However, in other cases, the detectors may be configured as imaging detectors configured to generate imaging signals or image data. Thus, the imaging subsystem may be configured to generate images in many ways.

[0042] Figure 2 is provided herein to generally illustrate the configuration of an imaging subsystem that may be included in embodiments of the system described herein. Obviously, the configuration of the imaging subsystem described herein can be modified to optimize the performance of the imaging subsystem, as is typically done when designing commercially available imaging systems. In addition, the system described herein can be implemented using existing systems (for example, by adding the functions described herein to an existing inspection system), such as a commercially available tool from KLA Corporation in Milpitas, California. In some such systems, the method described herein may be offered as an optional feature of the system (for example, in addition to other features of the system). Alternatively, the system described herein can be designed "from scratch" to provide a completely new system.

[0043] The computer subsystem 36 may be coupled to the detector of the imaging subsystem in any suitable manner (for example, via one or more transmission media which may include wired and / or wireless transmission media) so that the computer subsystem can receive the output generated by the detector. The computer subsystem 36 may be configured to perform several functions using the output of the detector. For example, if the system is configured as a defect review system, the computer subsystem may be configured to detect defects on a sample using the output of the detector. Detection of defects on a sample may be performed as further described herein.

[0044] The computer subsystem 36 may be further configured as described herein. For example, the computer subsystem 36 may be configured to perform the steps described herein. Thus, the steps described herein may be performed “on-tool” by a computer subsystem that is coupled to or part of the imaging subsystem. In addition, or alternatively, the computer system 102 may perform one or more of the steps described herein. Thus, one or more of the steps described herein may be performed “off-tool” by a computer system that is not directly coupled to the imaging subsystem.

[0045] Computer subsystem 36 (and other computer subsystems described herein) may also be referred to herein as computer systems. Each of the computer subsystems or systems described herein can take various forms, including personal computer systems, image computers, mainframe computer systems, workstations, network appliances, internet appliances, or other devices. Generally, the term “computer system” can be broadly defined to encompass any device having one or more processors, where processors execute instructions from a memory medium. A computer subsystem or system may also include any suitable processors well known in the art, such as parallel processors. In addition, a computer subsystem or system may include a computer platform with high-speed processing and software, either as a standalone or networked tool.

[0046] If a system includes two or more computer subsystems, the different computer subsystems may be coupled to one another, and images, data, information, instructions, etc., may be transmitted between them. For example, computer subsystem 36 may be coupled to computer system 102 by any suitable transmission medium, as shown by the dashed line in Figure 2, and the transmission medium may include any suitable wired and / or wireless transmission medium known in the art. Two or more such computer subsystems may also be effectively coupled by a shared computer-readable storage medium (not shown).

[0047] Although the imaging subsystem has been described above as an optical or light-based imaging subsystem, in another embodiment, the imaging subsystem is configured as an electron beam imaging subsystem. In an electron beam imaging subsystem, the energy directed to the sample includes electrons, and the energy detected from the sample includes electrons. In one such embodiment shown in Figure 2a, the imaging subsystem includes an electron column 122, and the system includes a computer subsystem 124 coupled to the imaging subsystem. The computer subsystem 124 may be configured as described above. In addition, such an imaging subsystem may be coupled to one or more other computer systems in the same manner as described above and in Figure 2.

[0048] As also shown in Figure 2a, the electron column includes an electron beam source 126 configured to generate electrons focused on a sample 128 by one or more elements 130. The electron beam source may include, for example, a cathode source or an emitter tip, and the one or more elements 130 may include, for example, a gun lens, an anode, a beam limiting aperture, a gate valve, a beam current selecting aperture, an objective lens, and a scanning subsystem, all of which may include any suitable elements known in the art.

[0049] Electrons returning from the sample (e.g., secondary electrons) can be focused onto the detector 134 by one or more elements 132. One or more elements 132 may include, for example, a scanning subsystem, which may be the same scanning subsystem as that included in element 130.

[0050] The electronic column may include any other suitable elements well known in the art. In addition, the electronic column may be further configured as described in U.S. Patent No. 8,664,594 issued to Jiang et al. on April 4, 2014, U.S. Patent No. 8,692,204 issued to Kojima et al. on April 8, 2014, U.S. Patent No. 8,698,093 issued to Gubbens et al. on April 15, 2014, and U.S. Patent No. 8,716,662 issued to MacDonald et al. on May 6, 2014, which are incorporated by reference as if they were fully described herein.

[0051] In Figure 2a, the electron column is configured such that electrons are directed towards the sample at an oblique incidence angle and scattered from the sample at another oblique angle, although the electron beam can be directed towards the sample at any suitable angle and scattered from the sample. In addition, the electron beam imaging subsystem can be configured to generate output to the sample using multiple modes (e.g., using different illumination angles, focusing angles, etc.), as further described herein. The multiple modes of the electron beam imaging subsystem may differ in any output generation parameters of the imaging subsystem.

[0052] The computer subsystem 124 may be coupled to the detector 134 as described above. The detector may detect electrons returning from the surface of the sample, thereby forming an electron beam image (or other output to the sample) of the sample. The electron beam image may include any suitable electron beam image. The computer subsystem 124 may be configured to detect defects on the sample using the output generated by the detector 134, which may be performed as further described herein. The computer subsystem 124 may be configured to perform any additional steps as described herein. The system including the imaging subsystem shown in Figure 2a may be further configured as described herein.

[0053] Figure 2a is provided herein to generally illustrate the configuration of an electron beam imaging subsystem that may be included in the embodiments described herein. Similar to the optical imaging subsystem described above, the configuration of the electron beam imaging subsystem described herein may be modified to optimize the performance of the imaging subsystem, which is typically done when designing commercially available systems. In addition, the systems described herein may be implemented using existing systems, such as commercially available tools from KLA (for example, by adding the functionality described herein to an existing system). For some such systems, the methods described herein may be provided as optional functionality of the system (for example, in addition to other functionality of the system). Alternatively, the systems described herein may be designed "from scratch" to provide a completely new system.

[0054] Although the imaging subsystem is described above as an optical or electron beam imaging subsystem, the imaging subsystem may also be an ion beam imaging subsystem. Such an imaging subsystem may be configured as shown in Figure 2a, except that the electron beam source can be replaced with any suitable ion beam source known in the art. In addition, the imaging subsystem may include any other suitable ion beam imaging system, such as those included in commercially available focused ion beam (FIB) systems, helium ion microscope (HIM) systems, and secondary ion mass spectrometry (SIMS) systems.

[0055] Furthermore, as described above, the imaging subsystem may be configured to have multiple modes. Generally, a “mode” is defined by the value of a parameter of the imaging subsystem used to generate the output from the sample. Thus, different modes may differ in value with respect to at least one of the imaging parameters of the imaging subsystem (other than the position on the sample where the output is generated). For example, in an optically based imaging subsystem, different modes may use light of different wavelengths. Modes may differ with respect to the wavelength of light directed at the sample (e.g., by using different light sources, different spectral filters, etc., for different modes), as will be further described herein. In another embodiment, different modes may use different illumination channels. For example, as described above, the imaging subsystem may include two or more illumination channels. Thus, different illumination channels may be used in different modes.

[0056] Multiple modes may further differ with respect to illumination and / or focusing / detection. For example, as further described above, the imaging subsystem may include multiple detectors. Thus, one of the detectors may be used for one mode and another detector for another mode. Furthermore, the modes may differ from one another in two or more ways as described herein (for example, different modes may have one or more different illumination parameters and one or more different detection parameters). The imaging subsystem may be configured to scan a sample simultaneously using multiple modes, for example, to scan the sample with different modes in the same or different scans.

[0057] In some cases, the systems described herein may be configured as defect review systems. However, the systems described herein may also be configured as other types of semiconductor-related quality control systems, such as inspection systems and measurement systems. For example, embodiments of the imaging subsystems described herein and shown in Figures 2 and 2a may provide different imaging capabilities depending on the application in which they are used by changing one or more parameters. In one embodiment, the imaging subsystem is configured as an electron beam defect review subsystem. For example, the imaging subsystem shown in Figure 2a may be configured to have a higher resolution when used for defect review or measurement rather than inspection. In other words, the embodiments of the imaging subsystems shown in Figures 2 and 2a illustrate general and various configurations of imaging subsystems that can be adjusted in several ways obvious to those skilled in the art, thereby making it possible to create imaging subsystems with different imaging capabilities that are more or less suitable for various applications.

[0058] As described above, the imaging subsystem may be configured to direct and / or scan energy (light, electrons, etc.) over a physical version of the sample, thereby generating a real image of the physical version of the sample. Thus, the imaging subsystem may be configured as a "real" imaging subsystem rather than a "virtual" system. However, the storage medium (not shown) and computer subsystem 102 shown in Figure 2 may be configured as a "virtual" system. In particular, the storage medium and computer subsystem are not part of the imaging subsystem 100 and do not have any ability to handle a physical version of the sample, but they may be configured using stored detector outputs as a virtual inspection device that performs functions such as inspection, a virtual measurement system that performs functions such as measurement, a virtual defect review tool that performs functions such as defect review, and so on. Systems and methods configured as “virtual” systems are described in U.S. Patent No. 8,126,255 issued to Bhaskar et al. on 28 February 2012, U.S. Patent No. 9,222,895 issued to Duffy et al. on 29 December 2015, and U.S. Patent No. 9,816,939 issued to Duffy et al. on 14 November 2017, which are incorporated by reference as if they were fully described herein. Embodiments described herein may be further configured as described in those patents. For example, the computer subsystem described herein may be further configured as described in those patents.

[0059] The system includes a computer subsystem which may include any configuration of the computer subsystem or system described above, and one or more components which are executed by the computer subsystem. For example, as shown in Figure 2, the system may include a computer subsystem 36 and one or more components 104 which are executed by the computer subsystem.

[0060] As shown in Figure 3, one or more components include an ensemble of deep learning (DL) models 302 and a pseudo-loss function 326 based on the output generated by the ensemble of DL models. The ensemble 302 may include, for example, DL model 1 (304), DL model 2 (306), DL model 3 (308), ..., and DL model N (310). Although at least four DL models are shown in Figure 3, one or more components may include any appropriate number of DL models, e.g., two or more DL models, five or more DL models, etc. DL models do not model any physical processes. Instead, DL models are trained to determine information generatively, rather than deterministically, from input images.

[0061] Each DL model may have, or include, any suitable DL architecture, such as a convolutional neural network (CNN) architecture. The CNN may have any suitable architecture known in the art. In addition, the CNN architectures of each DL model may be the same or different. In other words, one or more DL models may have the same CNN architecture, and / or one or more DL models may have different CNN architectures. If one or more DL models are CNNs or include CNNs, each CNN may include any suitable type of layer, such as convolution, pooling, fully connected, softmax, or any other suitable configuration known in the art. In one embodiment, each DL model includes six convolutional layers and about 100,000 parameters. For example, a relatively small neural network with six convolutional layers and about 100,000 parameters per network may be used.

[0062] DL models are configured as an ensemble of models. In some embodiments, the output of each DL model in the trained ensemble is the same type of information about the sample image at runtime. “Ensemble of models” is defined herein as a group of individual DL models trained independently or jointly to solve a common task. In other words, the DL models are not configured to determine different information about the sample. In some applications where the embodiments described herein may be used, the common task is to predict the location of defects on the sample.

[0063] Embodiments described herein may have one or more of the following novel features. For example, in one embodiment, a sequence of weight matrices and bias matrices represents a layer of each DL model. In another embodiment, the training described further herein includes determining a pseudo-loss function for each DL model in the ensemble and for each sample in the training dataset used for training. Thus, the embodiment may include a function that evaluates the pseudo-loss for each training sample. In another embodiment, the computer subsystem is configured to compute a probability weight for each of a plurality of samples in the training dataset and to select the plurality of samples used for training based on the probability weights computed for each of the plurality of samples. For example, the embodiment described herein may include a function configured to compute a probability weight for each training sample and to extract the training sample using the computed probabilities. In an additional embodiment, the output of the DL models in the trained ensemble includes a two-dimensional (2D) probability map output, and runtime label generation includes generating a 2D weighted average map from the 2D probability map output and assigning runtime labels based on whether the 2D weighted average map contains one or more detections that satisfy predetermined threshold and size criteria. For example, an embodiment may include a function that constructs the 2D probability map output of each network into a 2D weighted average map, and a decision function that outputs 0 or 1 based on whether the weighted average map contains one or more detections that satisfy both a threshold and a size criterion.

[0064] The embodiment may include two phases: training and inference.

[0065] The computer subsystem is configured to train an ensemble using a training dataset that includes training sample images and training labels indicating whether defects were detected in the training sample images. In one embodiment, the training sample images include fewer than 150 positive and negative samples of defects. For example, during training, a dataset may be compiled that includes approximately 100 images of non-defective areas and 20 or more images of defective areas. During training, sample images 300 may include training sample images and be input into the ensemble 302 along with any other training data in the training dataset. The computer subsystem may input the training sample images and any other information described herein into the ensemble of DL models in any suitable method known in the art.

[0066] The computer subsystem can acquire or generate training sample images, as further described herein, which are input into the DL model by the computer subsystem. For example, training sample images may be generated by the imaging subsystem and / or the computer subsystem, as further described herein. Each image may be manually reviewed to create a label for the training sample image. The label may be a binary image of the same size as the image in the dataset. Pixels in the label take a zero value at all positions except for the rectangular region surrounding the defect when placed on the real image. Thus, all negative samples have labels that are completely zero.

[0067] The images input to each DL model may be identical. In other words, all training sample images can be input to each DL model. However, the way each DL model uses the training sample images may differ. In other words, DL models do not need to have the same architectural configuration with the same parameters (but they may have the same architectural configuration with different parameters, different architectures, etc.). Thus, DL models may not simply be multiple instances of the same DL model. In any case, the input to each DL model is the same, and the output of each DL model will be the same type of information, even if different DL models produce different results.

[0068] Training performed using the training dataset can otherwise be carried out in any suitable method known in the art. The objective of training the DL models in the ensemble is to reduce the pseudo-loss function to a value less than 0.5, but not too low, such as less than 0.4. For example, training involves modifying one or more parameters of the ensemble until the pseudo-loss function determined based on the output of the ensemble is approximately equal to but less than or equal to 0.5. One important new feature of the embodiments described herein is the pseudo-loss function, which plays a crucial role in maintaining the desired probability of true detection while suppressing the probability of false positives. Another important new feature of the embodiments described herein is the inclusion of an ensemble of DL models in which the individual DL models are highly accurate on one measure and less accurate on the pseudo-loss measure.

[0069] The performance of the embodiments described herein largely depends on a pseudo-loss function defined as a function of input x and parameter ξ, as shown in the following equation.

number

number

[0070] In the above equation, C(x) is a 2D array of the same size as x and represents the aforementioned label. P(x) is a 2D array that takes continuous values ​​from 0 to 1 and represents the output of a single neural network. For example, as shown in Figure 3, DL model 1 produces the output of DL model 1 (312), DL model 2 produces the output of DL model 2 (314), DL model 3 produces the output of DL model 3 (316), ..., and DL model N produces the output of DL model N (318). These outputs may be the outputs of individual DL models. The computer subsystem can combine the outputs of individual DL models as shown in step 320 of Figure 3, and perform defect detection based on the combined output as shown in step 322, both of which may be performed as further described herein. Next, the computer subsystem may generate an image label 324 based on the result of defect detection, which may be a training image label during the training phase and may be generated as further described herein. Next, the training image labels and the outputs of the individual DL models can be input to the pseudo-loss function 326. Then, one or more parameters of the DL model ensemble can be modified using the pseudo-loss function value determined based on the training image labels and the outputs of the individual DL models.

[0071] The form of the pseudo-loss function is crucial to the embodiments described herein. For example, in one embodiment, the pseudo-loss function is configured such that false positive defect detections continuously increase the value of the pseudo-loss function, and true defect detections decrease the value of the pseudo-loss function in proportion to the maximum size of the true defect detection. In this form of pseudo-loss, any occurrence of false positives continuously increases the value of the function. True detections ω > 0 decrease the value of ρ. This decrease is proportional to the maximum size of the detection, i.e., the size of the label box. Thus, the pseudo-loss function can provide the desired functionality of maintaining a high penalty for false positives and zero penalty for true positives. In another embodiment, the pseudo-loss function is configured such that weak true detections and false positive detections affect the pseudo-loss function on a continuous scale. Thus, weak true detections and any false detections are penalized on a continuous scale.

[0072] On the other hand, weak detection is still valid detection, and false positives in defective images do not affect the total number of parts output for human review. Therefore, the observed performance of the ensemble in output counts may be such that the false negative and false positive rates are significantly lower, while the pseudo-loss function remains high. Ensemble training works only if a relatively high pseudo-loss value is obtained for the majority of the training samples. A relatively high pseudo-loss value ensures that a sufficiently large proportion of the training data is retained as the network sequence is trained. If this loss is too low, ensemble training stops with only a small number of samples that have non-vanishing probability weights. The pseudo-loss function is calculated for each DL model in the ensemble and for each sample of the training data.

[0073] In some embodiments, training involves hypothesis boosting. For example, a relatively large network with the same number of weights as the total number of weights in an ensemble has, in principle, the same learning capacity. The distinguishing feature of this ensemble is that these weights are carefully divided into separate small networks, which are then trained using a hypothesis boosting mechanism. By rigorous proof, the probability of a classification error in the hypothesis boosting mechanism is guaranteed to be zero over the entire probability distribution of the input.

[0074] In one embodiment, at least one of the DL models in the ensemble is configured as a weak learning algorithm. In another embodiment, training includes hypothesis boosting. For example, training may implement two classes: WeakLearn and HypothesisBoosting. An instantiation of class WeakLearn instantiates a 6-layer CNN. The instantiation may include a component that includes a member function that iterates through batches of training data and computes weight updates based on gradients averaged across the batches, a member function that computes gradients averaged across the batches and computes weight updates, and a member function that computes a pseudo-loss function when training is complete. HypothesisBoosting can then be implemented in any suitable way known in the art. Each network h t and the corresponding scalar value β t The data is saved in any suitable file format, for example, the HDF5 file format which is convenient for the embodiments described herein, as it naturally stores both sequence and scalar data.

[0075] During inference, the computer subsystem is configured to detect defects in the runtime sample image by inputting the runtime sample image into an ensemble of trained DL models and generating a runtime label for the runtime sample image indicating whether defects were detected in the runtime sample image, based on the output of the DL models in the trained ensemble. For example, during inference, the sample image 300 shown in Figure 3 is the runtime sample image input to the ensemble of DL models 302 by the computer subsystem. Inference is performed by supplying input images with the same number of imaging subsystem channels (modes), image size, and with or without design clips as the training dataset. Design clips may contain any appropriate design information known in the art. Each image is supplied in parallel to each DL model (DL model 1 (304), ..., DL model N (310)), and their outputs (output of DL model 1 (312), ..., output of DL model N (318)) may be collected as a list of 2D arrays. In one embodiment, the computer subsystem can combine the outputs of individual DL models in step 320 using the following equation, where the DL models are represented by the variable "t" which takes values ​​from 1 to M, where "M" is the total size of the ensemble. The output of DL model "t" given input "x" is h t Let's denote it as (x). The final 2D image is calculated according to the following formula.

number

[0076] In another embodiment, the output of the DL models within the learned ensemble includes a 2D probability map output, and the generation of the runtime label involves calculating a final scalar decision function for a combination of the 2D probability map outputs of each DL model within the learned ensemble. For example, the computer subsystem can calculate the final scalar decision function for the synthesis of the 2D probability maps of the individual DL models generated in an output combination step 320 executed based on the outputs of the individual DL models in the defect detection step 322. This is in contrast to other considered approaches where the final decision function is a linear combination of the scalar outputs of the same function applied to the individual learners of the ensemble.

[0077] In one implementation of the inference phase, at inference time, h t and β t can be loaded into the memory of the computer subsystem for all 1 ≤ t ≤ M. Next, the computer subsystem iterates through the dataset and prepares a three-dimensional (3D) array where the last dimension represents the SEM detection channel and, optionally, the design channel. The computer subsystem can implement a loop that loads and iterates through the data and prepares the DL model input "x" according to the above procedure. After loading "x", a copy of "x" is input into each DL model, and the computer subsystem collects "M" results z t = h t (x). Next, the pair (z t , β tThe inputs are collected and substituted into the function. Multiple copies of each DL model may be loaded into the computer subsystem. In this case, many inputs can be processed in parallel. This is another important difference from a single, relatively large neural network, where single inputs are processed sequentially. In contrast, an ensemble processes each input into a separate entity.

[0078] The embodiments described herein offer many significant advantages and improvements over currently used methods and systems for detecting defects in sample images. For example, the embodiments described herein can meet the high accuracy and high throughput requirements of HVM. In addition, the embodiments described herein can reduce the false positive rate by 2 × 10⁻⁶ in various datasets. -5 Improvements of less than 1 can be achieved. Furthermore, the embodiments can perform training based on a systematic procedure that has been theoretically proven to reduce classification errors as a polynomial function of increasing data and time. The performance of the embodiments described herein also does not depend heavily on tuning the parameters of the optimization algorithm, eliminating the expensive and necessary steps of existing methods. In other words, the embodiments described herein have the advantage of being relatively less sensitive to fine-tuning of the optimization algorithm. The performance of the embodiments described herein does not depend sensitively on the architecture of individual DL models. Moreover, the ensemble of DL models described herein is clearly robust to about a 10-fold increase in shot noise of the images input to the ensemble. In other words, the embodiments described herein have the advantage of being relatively less sensitive to shot noise.

[0079] Therefore, the embodiments described herein advantageously eliminate one of the highest risks in high-volume defect review in semiconductor chip manufacturing. In particular, as will be further described herein, high-volume defect review by SEM is an essential component used in high-throughput optical inspection, which is key to achieving economies of scale in the semiconductor industry. The embodiments described herein provide key techniques that can be used to enable the review of 5 million sites per hour.

[0080] Figure 4 shows how the ensemble improves results by combining the outputs of DL models according to a boosting procedure. In particular, Figure 4 shows an example of detecting defects using SEM images from the backend layer. The dots represent the results for individual DL models in the ensemble, and the final weighted results are represented by star-shaped markers. The gray trajectory shows how performance improves as M increases from 1 to the maximum size of the ensemble. The size of the dots represents the weight, and the shading indicates the position in the sequence, i.e., "t". The gray area represents the desired performance that the ensemble achieves.

[0081] In this figure, the number of false positives is the number out of 5 million, and the false negative rate is equal to the number of known defects missed divided by the total number of known defects in the dataset. The target performance of the HVM is within the gray area. While none of the individual DL models achieve high performance, the combined result is significantly improved because each is trained on a sequence that learns the errors of the preceding models. In some cases, the embodiments described herein may offer the advantage of scaling the false positive error nearly linearly with respect to the size of the ensemble.

[0082] In one embodiment, training does not involve hyperparameter tuning of the optimization algorithm for any DL model in the ensemble. Figure 5 illustrates how hyperparameter tuning of the optimization algorithm is unnecessary for the ensemble described herein. In particular, Figure 5 shows the sensitivity of performance to hyperparameter tuning for a single neural network (left) and an ensemble of neural networks (right). Five hyperparameter sets are consistently shown on the two plots. It is observed that the variability of false negatives and false positives in the single neural network is significantly larger. In the ensemble, there are also zero false positives in sets 2 and 4, but the variability between sets is much smaller otherwise. The lines for each set are drawn by changing the detection threshold.

[0083] Hyperparameter tuning of an optimization algorithm is a very costly step because it requires many iterations of the complete training loop. For a single network, the difference is clearly significant, making it necessary to explore the optimal hyperparameters. Figure 5 shows the change in false negative and false positive rates for five different hyperparameter sets. The lines in the figure are drawn by changing the detection threshold. For ensembles, only weak dependencies exist. This indicates that any reasonable hyperparameter set of an optimization algorithm can be a good starting point.

[0084] As an alternative to the embodiments described herein, a relatively large neural network can be constructed with a number of parameters equal to the total number of parameters in the entire ensemble. This network has the same capacity as the ensemble and, in principle, can achieve the same performance. However, unlike the hypothesis boosting mechanism in the ensemble approach of the embodiments described herein, there is no systematic procedure for training this relatively large network with probabilistic guarantees regarding training errors.

[0085] The computer subsystem may also be configured to generate results that include information about detected defects in the image of the sample, which may include any of the results or information described herein. The results may be generated by the computer subsystem in any suitable manner. All embodiments described herein may be configured to store the results of one or more steps of the embodiment on a computer-readable storage medium. The results may include any of the results described herein and may be stored in any manner well known in the art. The results may have any suitable format or type, such as a standard file type. The storage medium may include any storage medium described herein or any other suitable storage medium well known in the art.

[0086] After the results are saved, they are accessed on a storage medium and used by any of the methods or systems described herein, formatted for display to a user, and used by another software module, method, or system, etc., to perform one or more functions on the sample or another sample of the same type. For example, the results generated by the computer subsystem described herein may include information about any defects detected on the sample, such as the location of the bounding box of the detected defect, information about defect classification such as detection score, class label or ID, any defect attributes determined from any image, or any other suitable information known in the art. This information may be used by the computer subsystem, or other systems or methods, to perform additional functions on the sample and / or detected defects, such as sampling of defects for defect analysis, identifying the root cause of defects, etc.

[0087] Such functionality includes, but is not limited to, modifying processes such as manufacturing processes or steps that have been or will be performed on a sample, in a feedback or feedforward manner. For example, a computer subsystem may be configured to determine one or more changes to processes performed on and / or that will be performed on a sample, based on defect detection results. Changes to processes may include any appropriate changes to one or more parameters of the process. In such an example, the computer subsystem preferably determines these changes so that defects can be compensated for in another process performed on a sample, so that defects can be corrected or eliminated on the sample in another process performed on the sample, so that defects can be mitigated or prevented in other samples on which the modified process is performed. The computer subsystem may determine such changes in any appropriate method known in the art.

[0088] These changes may then be transmitted to a semiconductor manufacturing system (not shown) or a storage medium (not shown) accessible to both the computer subsystem and the semiconductor manufacturing system. The semiconductor manufacturing system may or may not be part of the embodiments of the system described herein. For example, the imaging subsystem and / or computer subsystem described herein may be coupled to the semiconductor manufacturing system via one or more common elements such as a housing, power supply, sample processing device or mechanism. The semiconductor manufacturing system may include any semiconductor manufacturing system well known in the art, such as lithography tools, etching tools, chemical mechanical polishing (CMP) tools, and deposition tools.

[0089] Each of the embodiments described above can be combined as a single embodiment. In other words, unless otherwise specified herein, no embodiment is mutually exclusive with any other embodiment.

[0090] Another embodiment relates to a computer implementation method for detecting defects in an image of a sample. This method includes training and detection steps, which are further described herein. These steps are performed by a computer system, which may be configured according to any of the embodiments described herein. Each step of the method may be performed as further described herein. The method may include any other optional steps that may be performed by an ensemble of imaging subsystems, computer subsystems, and / or DL ​​models, as described herein. In addition, the above method may be performed by any of the system embodiments described herein.

[0091] Additional embodiments relate to a non-temporary computer-readable medium storing program instructions executable on a computer system for performing a computer implementation method for detecting defects in an image of a sample. One such embodiment is shown in Figure 6. In particular, as shown in Figure 6, the non-temporary computer-readable medium 600 includes program instructions 602 executable on a computer system 604. The computer implementation method may include any step of any method described herein.

[0092] Program instructions 602 implementing a method like that described herein may be stored in a computer-readable medium 600. The computer-readable medium may be a storage medium such as a magnetic or optical disk, a magnetic tape, or any other suitable non-temporary computer-readable medium known in the art.

[0093] Program instructions can be implemented in any of the following ways, including, in particular, procedure-based techniques, component-based techniques, and / or object-oriented techniques. For example, program instructions can be implemented using ActiveX controls, C++ objects, JavaBeans, Microsoft Foundation Classes ("MFC"), SSE (Streaming SIMD Extensions), Python, Tensorflow, or other techniques or methodologies as needed.

[0094] The computer system 604 may be configured according to any of the embodiments described herein.

[0095] Further modifications and alternative embodiments of various aspects of the present invention will be apparent to those skilled in the art in consideration of this description. For example, a method and system for detecting defects in an image of a sample are provided. Therefore, this description should be interpreted as merely illustrative and is intended to teach those skilled in the art a general way of carrying out the present invention. It should be understood that the forms of the present invention shown and described herein should be taken as preferred embodiments at present. Elements and materials may be replaced with those illustrated and described herein, parts and processes may be reversed, and certain attributes of the present invention may be used independently, all of which will be apparent to those skilled in the art in this description of the present invention. Modifications may be made in elements described herein without departing from the spirit and scope of the present invention as set forth in the following claims.

Claims

1. A system configured to detect defects in an image of a sample, Computer subsystem, and One or more components executed by the computer subsystem, each including an ensemble of deep learning models and a pseudo-loss function based on the output generated by the ensemble of deep learning models, Equipped with, Here, the computer subsystem is Training the ensemble using a training dataset that includes training sample images and training labels indicating whether defects were detected in the training sample images, wherein one or more parameters of the ensemble are modified until the pseudo-loss function determined based on the output of the ensemble is approximately equal to but less than or equal to 0.5, and Defects in a runtime sample image are detected by inputting the runtime sample image into an ensemble of trained deep learning models, and generating a runtime label for the runtime sample image indicating whether a defect was detected in the runtime sample image based on the output of the deep learning models in the trained ensemble. Composed for, system.

2. The system according to claim 1, wherein the output of each of the deep learning models in the trained ensemble is the same type of information relating to the runtime sample image.

3. The system according to claim 1, wherein the training sample images include fewer than 150 positive and negative samples of defects.

4. The system according to claim 1, wherein each of the deep learning models comprises six convolutional layers and approximately 100,000 parameters.

5. The system according to claim 1, wherein the sequence of weight matrices and bias matrices represents each layer of the deep learning model.

6. The system according to claim 1, further comprising determining the pseudo-loss function for each of the deep learning models in the ensemble and for each sample of the training data.

7. The system according to claim 1, wherein the pseudo-loss function is configured such that false positive defect detection continuously increases the value of the pseudo-loss function, and true defect detection decreases the value of the pseudo-loss function in proportion to the maximum size of the true defect detection.

8. The system according to claim 1, wherein the pseudo-loss function is configured such that weak true detections and false positive detections affect the pseudo-loss function on a continuous scale.

9. The system according to claim 1, wherein the computer subsystem is further configured to calculate a probability weight for each of the multiple samples in the training dataset and to select the multiple samples to be used for training based on the calculated probability weight for each of the multiple samples.

10. The system according to claim 1, wherein the output of the deep learning model in the trained ensemble includes a two-dimensional probability map output, and the generation of runtime labels includes generating a two-dimensional weighted average map from the two-dimensional probability map output and assigning the runtime labels based on whether the two-dimensional weighted average map includes one or more detections that satisfy predetermined threshold and size criteria.

11. The system according to claim 1, wherein the output of the deep learning models in the trained ensemble includes a two-dimensional probability map output, and the generation of runtime labels includes computing a final scalar decision function for each combination of the two-dimensional probability map outputs of the deep learning models in the trained ensemble.

12. The system according to claim 1, wherein the training does not involve hyperparameter tuning of the optimization algorithm for any of the deep learning models in the ensemble.

13. The system according to claim 1, wherein at least one of the deep learning models in the ensemble is configured as a weak learning algorithm.

14. The system according to claim 1, wherein the training further includes hypothesis boosting.

15. The system according to claim 1, wherein the runtime sample image is generated during a defect review process performed on the sample in a mass production process.

16. The system according to claim 1, wherein the trained sample image and the runtime sample image are generated by the imaging subsystem of the defect review tool.

17. The system according to claim 1, wherein the trained sample image and the runtime sample image are generated by an electron beam-based imaging subsystem.

18. The system according to claim 1, wherein the runtime sample image input to the ensemble of trained deep learning models for any one position on the sample includes an image generated using a plurality of detectors of the imaging subsystem.

19. A non-temporary computer-readable medium that stores program instructions for executing a computer implementation method for detecting defects in an image of a sample, which is executable on a computer system, wherein the computer implementation method is Training an ensemble of deep learning models using a training dataset comprising training sample images and training labels indicating whether defects are detected in the training sample images, wherein the training includes modifying one or more parameters of the ensemble until a pseudo-loss function determined based on the output of the ensemble is approximately equal to but less than or equal to 0.5, wherein one or more components performed by the computer system include training the ensemble of deep learning models and the pseudo-loss function. Defects in a runtime sample image are detected by inputting the runtime sample image into an ensemble of trained deep learning models, and generating a runtime label for the runtime sample image indicating whether a defect was detected in the runtime sample image based on the output of the deep learning models in the trained ensemble. Non-temporary computer-readable media, including [specific examples of such media].

20. A computer implementation method for detecting defects in an image of a sample, Training an ensemble of deep learning models using a training dataset comprising training sample images and training labels indicating whether defects are detected in the training sample images, wherein the training includes modifying one or more parameters of the ensemble until a pseudo-loss function determined based on the output of the ensemble is approximately equal to but less than or equal to 0.5, wherein one or more components performed by the computer system include training the ensemble of deep learning models and the pseudo-loss function. The detection of defects in a runtime sample image is performed by inputting the runtime sample image into an ensemble of trained deep learning models, and generating a runtime label for the runtime sample image indicating whether a defect was detected in the runtime sample image based on the output of the deep learning models in the trained ensemble, wherein the training and the detection are performed by the computer system. Computer implementation methods, including those mentioned above.

Images