A generative adversarial network (GAN) for simulating sample images

Abstract

Methods and systems for generating simulated images of samples are provided. A system includes one or more computer subsystems and one or more components executed by the one or more computer subsystems. The one or more components include a generative adversarial network (GAN), such as a conditional GAN (cGAN), trained using a training set that includes portions of design data of one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs. The one or more computer subsystems are configured to generate a simulated image of a sample by inputting a portion of design data of the sample into the GAN.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to methods and systems for generating simulated images of samples using generative adversarial networks (GANs). BACKGROUND

[0002] The following description and examples do not admit to being prior art merely by virtue of their inclusion in this section.

[0003] Fabricating semiconductor devices such as logic and memory devices typically includes processing a substrate such as a semiconductor wafer using a large number of semiconductor manufacturing processes to form various features and multiple levels of the semiconductor devices on the substrate. For example, lithography, which involves transferring a pattern from a reticle to a resist arranged on a semiconductor wafer, is a semiconductor manufacturing process. Additional examples of semiconductor manufacturing processes include, but are not limited to, chemical-mechanical polishing (CMP), etch, deposition, and ion implantation. Multiple semiconductor devices can be fabricated in an arrangement on a single semiconductor wafer and then separated into individual semiconductor devices.

[0004] Inspection processes are used during various steps in the semiconductor manufacturing process to detect defects on wafers to promote higher yield in the manufacturing process and thus higher profits. Inspection has always been an important part of fabricating semiconductor devices. However, as the dimensions of semiconductor devices decrease, inspection becomes even more important to successfully fabricate acceptable semiconductor devices because smaller defects can cause device failure.

[0005] One significant hurdle in setting up most inspection processes is identifying a sufficient number of defects of interest (DOIs) that can then be used to set various parameters of the inspection process. Such DOIs are typically necessary to set both hardware parameters such as optical or other imaging parameters and software-type parameters such as defect classification settings, nuisance filter parameters, and the like. If sufficient DOI instances are not found on one or several setup samples, then the resulting inspection recipe can be suboptimal for detecting and identifying the DOIs on other samples.

[0006] While setting such hardware and software parameters also requires nuisance instances, nuisance instances tend to be used liberally and overwhelming. For example, when setting up a new inspection process, a hot scan can be performed on a setup sample where the threshold for defect detection is set at, near, or even within the noise floor of the system output. Thus, this scan can generate many more nuisance instances than needed, and the nuisances can make it particularly difficult to identify DOI instances because it needs to be separated from all detection events, most of which are nuisances.

[0007] Another difficulty in setting up inspection processes is that the set-up sample can not contain any instances of one or more DOI types, which can result in an inspection process that cannot detect this DOI type(s). This difficulty can also generate in maintaining the performance of an already set-up inspection process. For example, if the DOI type on the inspection sample changes for some reason, the already set-up inspection process can not be able to detect the new DOI type. This inspection process can even become unusable to require a completely new inspection process to be set up using a new set-up sample.

[0008] The lack of DOI instances causes particular problems for performing interference reduction on optical or other tools using sophisticated techniques such as deep learning (DL). Therefore, the inspection recipe can be limited to manual tuning or a determination tree based on random forests. However, even a random forest type determination tree requires a certain number of DOI instances for training that are sometimes not available. Therefore, one current drawback in setting up inspection and other recipes is that sophisticated techniques cannot be used for interference reduction in other inspection or quality control type processes when not enough instances of DOIs are available.

[0009] Therefore, it would be advantageous to develop systems and methods for generating simulated images of samples that do not have one or more of the above disadvantages. SUMMARY

[0010] The following description of various embodiments does not in any way limit the subject matter of the appended claims.

[0011] One embodiment relates to a system configured to generate a simulated image of a sample. The system includes one or more computer subsystems and one or more components executed by the one or more computer subsystems. The one or more components include a generative adversarial network (GAN) trained using a training set including portions of design data of one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs. The one or more computer subsystems are configured to generate a simulated image of a sample by inputting a portion of design data of the sample into the GAN. The system can be further configured as described herein.

[0012] Another embodiment relates to a computer-implemented method for generating a simulated image of a sample. The method includes generating a simulated image of a sample by inputting a portion of design data of the sample into a GAN. The input is performed by one or more computer subsystems. One or more components are executed by the one or more computer subsystems. The one or more components include the GAN. The GAN is trained using a training set including portions of design data of one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs.

[0013] Each of the steps of the method can be further performed as further described herein. The method can include any other steps of any other method described herein. The method can be performed by any of the systems described herein.

[0014] Another embodiment relates to a non-transitory computer readable medium storing program instructions executable on one or more computer systems to perform a computer-implemented method for generating simulated images of samples. The computer-implemented method includes the steps of the above-described method. The computer readable medium can be further configured as described herein. The steps of the computer-implemented method can be performed as further described herein. Additionally, the computer-implemented method that the program instructions are executable to perform can include any other steps of any other method described herein. BRIEF DESCRIPTION OF DRAWINGS

[0015] Those skilled in the art will appreciate the other advantages of the application upon reading the following detailed description of the preferred embodiments, taken in conjunction with the drawings, in which:

[0016] Figure 1 and 1a is a schematic illustration of a side view of an embodiment of a system configured as described herein;

[0017] Figures 2 to 4 is a flowchart illustrating steps that can be performed by embodiments described herein;

[0018] Figure 5 is a schematic illustration of one instance of a generator that can be included in an embodiment of a generative adversarial network (GAN);

[0019] Figure 6 is a schematic illustration of one instance of a discriminator that can be included in an embodiment of a GAN; and

[0020] Figure 7 is a block diagram illustrating one embodiment of a non-transitory computer readable medium storing program instructions for causing a computer system to perform a computer-implemented method described herein.

[0021] While the application is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and are herein described in detail. The drawings can not be to scale. It should be understood that the drawings and detailed description thereto are not intended to limit the application to the particular form disclosed but on the contrary the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the application as defined by the appended claims. DETAILED DESCRIPTION

[0022] The terms "design," "design data," and "design information," used interchangeably herein, generally refer to the physical design (layout) of an IC or other semiconductor device and data derived from the physical design by complex simulation or simple geometric and Boolean operations. The design can include any other design data or design data agent described in commonly owned U.S. Patent No. 7,570,796, issued August 4, 2009, to Zafar et al., and U.S. Patent No. 7,676,077, issued March 9, 2010, to Kulkarni et al., both of which are incorporated herein by reference as if fully set forth. Additionally, the design data can be standard cell library data, integrated layout data, design data for one or more layers, derivatives of the design data, and full or partial chip design data. Furthermore, "design," "design data," and "design information" described herein refer to information and data generated by a semiconductor device designer during the design process and thus are available for use in the embodiments described herein prior to the design being printed on any physical sample such as reticles and wafers.

[0023] The term "interference," which can sometimes be used interchangeably with "interference defect" or "interference event," as used herein, is generally defined as a defect that is not of interest to the user and / or an event detected on the sample that is not in fact an actual defect on the sample. Interference that is not in fact a defect can be detected as an event due to non-defect noise sources on the sample (such as a grain in a metal line on the sample, a signal from underlying layers or materials on the sample, line edge roughness (LER), relatively small critical dimension (CD) variations of patterned features, thickness variations, etc.) and / or due to the edge cases of the inspection system itself or its configuration for inspection.

[0024] The term "defect of interest (DOI)," as used herein, is defined as a defect detected on the sample that is in fact an actual defect on the sample. Thus, a DOI is of interest to the user because the user generally is interested in how many actual defects and what kinds of actual defects are on the inspected sample. In some contexts, the term "DOI" is used to refer to a subset of all actual defects on the sample that includes only the actual defects of interest to the user. For example, there can be multiple types of DOIs on any given sample, and the user can be more interested in one or more of them than one or more other types. However, in the context of the embodiments described herein, the term "DOI" is used to refer to any and all actual defects on the sample.

[0025] Turning now to the drawings, it is noted that the figures are not drawn to scale. In particular, the proportions of some of the elements of the figures have been significantly exaggerated to emphasize features of the elements. It is also noted that the figures are not drawn to the same scale. Elements shown in more than one figure that can be similarly configured have been indicated using the same element number. Unless otherwise indicated herein, any elements described and shown can include any suitable commercially available elements.

[0026] In general, embodiments described herein include a generative adversarial network (GAN) or a type of GAN (e.g., conditional GAN (cGAN)) for improving optical defect inspection execution and other applications. As described further herein, some embodiments create a new way of performing defect augmentation using a GAN or cGAN. As such, such embodiments are able to use learning techniques that are based on and / or require large numbers of defect instances. Embodiments described herein can also or alternatively be configured and used to generate reference images for any test images of applications such as die-to-database type inspections.

[0027] In some embodiments, the sample is a wafer. The wafer can include any wafer known in semiconductor technology. Although some embodiments can be described herein with respect to one or several wafers, embodiments are not limited to the samples that can be used therewith. For example, embodiments described herein can be used for samples such as reticles, flat panels, personal computer (PC) boards, and other semiconductor samples.

[0028] One embodiment relates to a system configured to generate a simulated image of a sample. Figure 1 One embodiment of such a system is shown in FIG. 1. The system includes one or more computer subsystems 102 and one or more components 104 executed by the one or more computer subsystems. In some embodiments, the system includes an inspection system coupled to the one or more computer subsystems. For example, in Figure 1 In particular, the system includes an inspection system 100 coupled to the computer subsystems 102. In Figure 1 In the embodiment shown, the inspection system is configured as a light-based inspection system. However, in other embodiments described herein, the inspection system is configured as an electron beam or charged particle beam inspection system.

[0029] In general, the inspection systems described herein include at least an energy source, a detector, and a scanning subsystem. The energy source is configured to generate energy directed by the inspection system to a sample. The detector is configured to detect energy from the sample and generate an output in response to the detected energy. The scanning subsystem is configured to change a position on the sample at which energy is directed thereto and from which energy is detected.

[0030] In a light-based inspection system, the energy directed to the sample includes light, and the energy detected from the sample includes light. In Figure 1In the illustrated embodiment of the system, the inspection system includes an illumination subsystem configured to direct light to the sample 14. The illumination subsystem includes at least one light source, such as the light source 16 illustrated in Figure 1 The illumination subsystem can be configured to direct light to the sample at one or more angles of incidence that can include one or more oblique angles and / or one or more normal angles. For example, as illustrated in Figure 1 The oblique angle of incidence can include any suitable oblique angle of incidence, which can vary depending on, for example, the characteristics of the sample and the defects to be detected on the sample.

[0031] The illumination subsystem can be configured to direct light to the sample at different angles of incidence at different times. For example, the inspection system can be configured to alter one or more characteristics of one or more elements of the illumination subsystem so that light can be directed to the sample at an angle of incidence that is different from the angle of incidence illustrated in Figure 1 In one such example, the inspection system can be configured to move the light source 16, the optical element 18, and the lens 20 so that light is directed to the sample at a different oblique angle of incidence or a normal (or near-normal) angle of incidence.

[0032] In some instances, the inspection system can be configured to direct light to the sample at more than one angle of incidence at the same time. For example, the illumination subsystem can include more than one illumination channel, one of which can include the light source 16, the optical element 18, and the lens 20 (as illustrated in Figure 1 The other (not illustrated) can include similar elements, which can be in different or the same configuration or can include at least one light source and possibly one or more other components (such as other components described further herein). If such light is directed to the sample at the same time as the other light, one or more characteristics of the light directed to the sample at different angles of incidence (e.g., wavelength, polarization, etc.) can be different so that light from the illumination of the sample at different angles of incidence can be distinguished from each other at the detector.

[0033] In another example, the illumination subsystem can include only one light source (e.g., Figure 1The light source 16 shown in FIG. 1 can be any suitable light source. For example, the light source 16 can be a broadband plasma (BBP) light source. In this manner, the light generated by the light source and directed to the sample can include broadband light. However, the light source can include any other suitable light source, such as a laser. The laser can include any suitable laser known in the art and can be configured to generate light at any suitable wavelength known in the art. The laser can be configured to generate monochromatic or nearly monochromatic light. In this manner, the laser can be a narrowband laser. The light source can also include a polychromatic light source that generates light at multiple discrete wavelengths or wavebands.

[0034] The light source 16 can include a broadband plasma (BBP) light source. In this manner, the light generated by the light source and directed to the sample can include broadband light. However, the light source can include any other suitable light source, such as a laser. The laser can include any suitable laser known in the art and can be configured to generate light at any suitable wavelength known in the art. The laser can be configured to generate monochromatic or nearly monochromatic light. In this manner, the laser can be a narrowband laser. The light source can also include a polychromatic light source that generates light at multiple discrete wavelengths or wavebands.

[0035] The light from the optical element 18 can be focused onto the sample 14 by a lens 20. Although the lens 20 is shown in FIG. 1 as a single refractive optical element, in practice, the lens 20 can include several refractive and / or reflective optical elements that combine to focus the light from the optical element to the sample. Figure 1 The lens 20 can be any suitable lens known in the art. For example, the lens 20 can be a single refractive optical element. In this manner, the lens 20 can be a single lens. However, the lens 20 can include any other suitable lens known in the art. For example, the lens 20 can include a plurality of lenses that combine to focus the light from the optical element to the sample. In this manner, the lens 20 can be a multi-lens system. Figure 1 The illumination subsystem shown in FIG. 1 and described herein can include any other suitable optical elements (not shown in the figures). Examples of such optical elements include, but are not limited to, polarization components, spectral filters, spatial filters, reflective optical elements, apodizers, beam splitters, stops, and the like, which can include any such suitable optical elements known in the art. Additionally, the system can be configured to alter one or more of the elements of the illumination subsystem based on the type of illumination used for inspection.

[0036] The inspection system also includes a scanning subsystem configured to change the position of light directed to and from the sample it is inspecting, and may cause the light to scan across the sample. For example, the inspection system may include a stage 22 on which the sample 14 is placed during inspection. The scanning subsystem may include any suitable mechanical and / or robotic assembly (including the stage 22) configured to move the sample such that light can be directed to and detected from different positions on the sample. Alternatively or additionally, the inspection system may be configured such that one or more optical elements of the inspection system perform some scans of light across the sample, such that light can be directed to and detected from different positions on the sample. In the example where light scans across the sample, the light may scan across the sample in any suitable manner, such as along a serpentine or spiral path.

[0037] The inspection system further includes one or more detection channels. At least one of the detection channels includes a detector configured to detect light from the sample attributable to system illumination of the sample and to generate an output in response to the detected light. For example, Figure 1 The inspection system shown includes two detection channels: one detection channel is formed by a light collector 24, an element 26, and a detector 28, and the other detection channel is formed by a light collector 30, an element 32, and a detector 34. For example... Figure 1 As shown, two detection channels are configured to collect and detect light at different collection angles. In some examples, the two detection channels are configured to detect scattered light, and the detection channel is configured to detect light scattered from the sample at different angles. However, one or more of the detection channels may be configured to detect another type of light (e.g., reflected light) from the sample.

[0038] like Figure 1 The diagram further illustrates that the two detection channels are positioned within the plane of the paper, and the illumination subsystem is also positioned within the plane of the paper. Therefore, in this embodiment, the two detection channels are positioned (e.g., centered) within the incident plane. However, one or more of the detection channels may be positioned outside the incident plane. For example, the detection channel formed by the light collector 30, element 32, and detector 34 may be configured to collect and detect light scattered from the incident plane. Therefore, this detection channel may generally be referred to as a "side" channel, and this side channel may be centered in a plane substantially perpendicular to the incident plane.

[0039] although Figure 1An embodiment of an inspection system comprising two detection channels is shown, but the inspection system may comprise a different number of detection channels (e.g., only one detection channel or two or more detection channels). In one example, the detection channel formed by the light collector 30, element 32, and detector 34 may form one side channel as described above, and the inspection system may include an additional detection channel (not shown) formed as another side channel positioned on the opposite side of the incident surface. Thus, the inspection system may comprise a detection channel having a light collector 24, element 26, and detector 28, centered in the incident surface, and configured to collect and detect light at a scattering angle perpendicular to or nearly perpendicular to the sample surface. Therefore, this detection channel may generally be referred to as the "top" channel, and the inspection system may also include two or more side channels configured as described above. Thus, the inspection system may comprise at least three channels (i.e., one top channel and two side channels), and each of the at least three channels has its own light collector, wherein each is configured to collect light at a scattering angle different from that of each of the other light collectors.

[0040] As further described above, each of the detection channels included in the inspection system can be configured to detect scattered light. Therefore, Figure 1 The testing system shown herein can be configured for dark-field (DF) testing of samples. However, the testing system may also, or alternatively, include a detection channel configured for bright-field (BF) testing of samples. In other words, the testing system may include at least one detection channel configured to detect light reflected from the specular surface of the sample. Therefore, the testing system described herein can be configured for DF only, BF only, or both DF and BF testing. Although each of the light collectors is in Figure 1 The image is shown as a single refractive optical element, but it should be understood that each of the light collectors may contain one or more refractive optical elements and / or one or more reflective optical elements.

[0041] One or more detection channels may contain any suitable detector known in the art, such as a light multiplier tube (PMT), a charge-coupled device (CCD), or a time-delay integration (TDI) camera. The detector may also contain non-imaging detectors or imaging detectors. If the detector is a non-imaging detector, each of the detectors may be configured to detect certain properties of light (e.g., intensity) but cannot be configured to detect properties that vary depending on position within the imaging plane. Therefore, the output generated by each of the detectors contained in each of the detection channels may be a signal or data, but not an image signal or image data. In such examples, the computer subsystem of the inspection system, such as computer subsystem 36, may be configured to generate an image of the sample from the non-imaging output of the detector. However, in other examples, the detector may be configured as an imaging detector configured to generate an imaging signal or image data. Therefore, the inspection system may be configured to generate images in various ways.

[0042] It should be noted that the descriptions provided herein Figure 1a A configuration of an inspection system that can be included in the system embodiments described herein is generally illustrated. It is apparent that the inspection system configuration described herein can be altered to optimize performance of the inspection system, as is typically performed when designing a commercial inspection system. In addition, the systems described herein can use existing inspection system implementations (e.g., by adding the functionality described herein to an existing inspection system), such as the 29xx / 39xx series tools available from KLACorp., Milpitas, Calif. For some such systems, the methods described herein can be provided as optional functionality of the inspection system (e.g., in addition to other functionality of the inspection system). Alternatively, the inspection systems described herein can be designed "from scratch" to provide a completely new inspection system.

[0043] The computer subsystem 36 can be coupled to the detector of the inspection system in any suitable manner (e.g., via one or more transmission media that can include "wired" and / or "wireless" transmission media) such that the computer subsystem can receive the output generated by the detector. The computer subsystem 36 can be configured to perform a number of functions using the output of the detector. For example, the computer subsystem can be configured to detect events on the sample using the output of the detector. Detecting events on the sample can be performed by applying some defect detection algorithm and / or method to the output generated by the detector, which can include any suitable algorithm and / or method known in the art. For example, the computer subsystem can compare the output of the detector to a threshold. Any output having a value above the threshold can be identified as an event (e.g., a potential defect), while any output having a value below the threshold can not be identified as an event.

[0044] The computer subsystem of the inspection system can be further configured as described herein. For example, the computer subsystem 36 can be part of or can be configured as one or more of the computer subsystems described herein. In particular, the computer subsystem 36 can be configured to perform the steps described herein. As such, the steps described herein can be performed "on-tool" by a computer system or subsystem that is part of the inspection system.

[0045] The computer subsystems of the inspection system (as well as other computer subsystems described herein) can also be referred to herein as computer systems. Each of the computer subsystems or systems described herein can take various forms, including a personal computer system, image computer, mainframe computer system, workstation, network appliance, Internet appliance, or other device. In general, the term "computer system" can be broadly defined to encompass any device having one or more processors, which executes instructions from a memory medium. The computer subsystems or systems can also include any suitable processor known in the art, such as a parallel processor. Additionally, the computer subsystems or systems can include a computer platform having high speed processing and software, either as a stand-alone or networked appliance.

[0046] If the system includes more than one computer subsystem, the different computer subsystems can be coupled to one another such that images, data, information, instructions, etc. can be sent between the computer subsystems. For example, computer subsystem 36 can be coupled to computer subsystem 102 by any suitable transmission medium, which can include any suitable wired and / or wireless transmission medium known in the art, as shown by the dashed lines in FIG. 1. Two or more such computer subsystems can also be coupled, in fact, by a shared computer-readable storage medium (not shown in the figures). Figure 1

[0047] Although the inspection system is described above as an optical inspection system or light-based inspection system, in another embodiment, the inspection system is configured as an electron beam inspection system. In an electron beam inspection system, the energy directed to the sample includes electrons, and the energy detected from the sample includes electrons. In one such embodiment, the inspection system includes an electron column 122, and the system includes a computer subsystem 124 coupled to the inspection system. The computer subsystem 124 can be configured as described above. Additionally, this inspection system can be coupled to one or more additional computer subsystems in the same manner as described above and shown in FIG. 1. Figure 1a Figure 1a

[0048] Also as shown in FIG. 1, the electron column includes an electron beam source 126 configured to generate electrons that are focused to the sample 128 by one or more elements 130. The electron beam source can include, for example, a cathode source or a tip, and the one or more elements 130 can include, for example, a gun lens, an anode, a limiting aperture, a gate valve, a beam current selection aperture, an objective lens, and a scanning subsystem, all of which can include any such suitable elements known in the art. Figure 1a

[0049] The electrons returned from the sample, such as secondary electrons, can be focused to a detector 134 by one or more elements 132. The one or more elements 132 can include, for example, a scanning subsystem, which can be the same scanning subsystem included in elements 130.​​​​

[0050] The electron column can include any other suitable elements known in the art. Additionally, the electron column can be further configured as described in U.S. Patent Nos. 8,664,594, issued April 4, 2014 to Jiang et al., 8,692,204, issued April 8, 2014 to Kojima et al., 8,698,093, issued April 15, 2014 to Gubbens et al., and 8,716,662, issued May 6, 2014 to MacDonald et al., which are incorporated by reference herein as if fully set forth.

[0051] Although the electron column is shown in Figure 1a as being configured such that electrons are directed to the sample at one oblique angle of incidence and scattered from the sample at another oblique angle, the electron beam can be directed to the sample and scattered from the sample at any suitable angles. Additionally, the electron beam inspection system can be configured to generate output of the sample described further herein using multiple modes (e.g., with different angles of illumination, collection angles, etc.). The multiple modes of the electron beam inspection system can differ in any output generation parameter of the inspection system.

[0052] The computer subsystem 124 can be coupled to the detector 134 as described above. The detector can detect electrons returned from the surface of the sample, thereby forming an electron beam image (or other output) of the sample. The electron beam image can include any suitable electron beam image. The computer subsystem 124 can be configured to detect events on the sample using the output generated by the detector 134, which can be performed as described above or in any other suitable manner. The computer subsystem 124 can be configured to perform any additional steps described herein. Including Figure 1a The system of the inspection system shown in

[0053] It should be noted that the Figure 1 to generally describe configurations of electron beam inspection systems that can be included in embodiments described herein. As with the optical inspection systems described above, the electron beam inspection system configurations described herein can be altered to optimize performance of the inspection system, which is typically performed when designing a commercial inspection system. Additionally, the systems described herein can be implemented using existing inspection system implementations (e.g., by adding the functionality described herein to an existing inspection system), such as tools available from KLA. For some such systems, the methods described herein can be provided as optional functionality of the system (e.g., in addition to other functionality of the system). Alternatively, the systems described herein can be designed "from scratch" to provide a brand new system.

[0054] Although the inspection system is described above as a light or electron beam inspection system, the inspection system can be an ion beam inspection system. This inspection system can be configured as shown in Figure 1a , except that the electron beam source can be replaced with any suitable ion beam source known in the art. Additionally, the inspection system can include any other suitable ion beam imaging system, such as those included in commercially available focused ion beam (FIB) systems, helium ion microscope (HIM) systems, and secondary ion mass spectrometer (SIMS) systems.

[0055] As further mentioned above, the inspection system can be configured to have multiple modes. Generally, a "mode" is defined by the values of the parameters of the inspection system used to generate an output of a sample. Thus, different modes can differ in the values of at least one of the optical or electron beam parameters of the inspection system (in addition to the location on the sample where the output is generated). For example, for a light-based inspection system, different modes can use different wavelengths of light. The modes can differ in the wavelength of the light directed to the sample, as further described herein for different modes (e.g., by using different light sources, different spectral filters, etc.). In another embodiment, different modes can use different illumination channels. For example, as mentioned above, the inspection system can include more than one illumination channel. Thus, different illumination channels can be used for different modes.

[0056] The multiple modes can also differ in the illumination and / or collection / detection. For example, as further described above, the inspection system can include multiple detectors. Thus, one of the detectors can be used for one mode, while another of the detectors can be used for another mode. Moreover, the modes can differ from one another in more than one way as described herein (e.g., different modes can have one or more different illumination parameters and one or more different detection parameters). The inspection system can be configured to scan the sample using different modes in the same scan or in different scans, e.g., depending on the ability to simultaneously scan the sample using multiple modes.

[0057] The systems described herein can be configured as another type of semiconductor-related process / quality control type system, such as a defect review system and a metrology system. For example, the embodiments of the systems shown in Figure 1 and 1a The embodiments of the systems shown in Figure 1 may be modified to provide different imaging capabilities depending on the application for which they will be used. In one embodiment, the electron beam inspection system configuration described herein can be modified to configure as an electron beam defect review system. For example, the electron beam inspection system configuration shown in Figure 1 may be configured to have a higher resolution when it is used for defect review or metrology rather than for inspection. In other words, the electron beam inspection system configuration shown in 1aThe embodiments of systems shown in the figures describe some general and various configurations of systems that can be adapted in various ways as will be apparent to those skilled in the art to produce systems with different imaging capabilities that are well suited to different applications.

[0058] As mentioned above, the inspection system can be configured to direct energy (e.g., light, electrons) to and / or scan over the physical version of the specimen, thereby generating actual images of the physical version of the specimen. In this way, the inspection system can be configured as an "actual" imaging system rather than a "virtual" system. Figure 2 The storage media (not shown in the figures) and computer subsystems 102 shown in the figures can be configured as a "virtual" system. Systems and methods configured as "virtual" inspection systems are described in commonly-assigned U.S. Patent No. 8,126,255 to Bhaskar et al., issued February 28, 2012, and U.S. Patent No. 9,222,895 to Duffy et al., issued December 29, 2015, both of which are incorporated herein by reference as if set forth in their entirety. Embodiments described herein can be further configured as described in these patents.

[0059] The one or more components executed by the one or more computer subsystems include Figure 2 The generative adversarial network (GAN) 106 shown in the figures. The GAN is trained using a training set that includes portions of design data for one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs. The GAN can or can not be trained by the one or more computer subsystems and / or by one of the components executed by the computer subsystems. For example, another method or system can train the GAN, which can then be stored for use as a component executed by the computer subsystems.

[0060] Figure 2 Some steps shown in the figures can be performed by the one or more computer subsystems to train the GAN. Although Figure 2 Steps shown can be performed in a method for data augmentation using the GAN, but Figure 3 Some steps shown in the figures can also be performed for other applications. For example, steps 200, 202, and 204 can be performed to train the GAN, regardless of which application the GAN will be used for as described herein.

[0061] As shown in step 200, the computer subsystems can align the optical images with the design using pixel-to-design alignment (PDA). If the GAN is trained for different kinds of images, then the PDA can be performed for each kind of image. Figure 3Another type of image (e.g., e-beam) is used in place of the optical image in the steps shown. The PDA can be performed in any suitable manner known in the art, some examples of which are described in the above-cited patent by Kulkarni et al. The PDA can be performed on any one or more alignment targets on the sample to thereby determine image-to-design offsets, which can then be used by the computer subsystem to identify which portions of the design of the sample correspond to images generated for the sample and / or events detected on the sample.

[0062] As shown in step 202, the computer subsystem can run a hot scan to obtain pairs of optical images and design segments. The term "design segment" as used herein is generally defined as a relatively small portion of the entire design of the sample. The term "design segment" can be used interchangeably herein with the phrase "portion of design data." The hot scan can be performed in any suitable manner known in the art. Generally, a "hot" scan involves scanning the sample and detecting events on the sample using a "hot" threshold (e.g., a threshold at, near, or even within the noise floor of the output generated by the scan). For any events detected in the hot scan, the computer subsystem can create pairs of optical images and their corresponding design segments. Pairs can be created for only some or all of the events detected in the hot scan. Generally, pairs can be created for less than all events because a hot scan generally detects many more events than are needed for any application. The computer subsystem can sample or select in any suitable manner which events to use to create pairs of images and design segments. Pairs of images and design segments can also be created for locations in which no events were detected. In this way, the training set can include pairs of images and design segments for "good" locations on the sample. In some such embodiments, the hot scan can not detect events on the sample at all, but rather can only collect images that can then be paired with corresponding design segments. Whether or not for events, the image-to-design offsets described above can be used to identify and pair corresponding images and design segments.

[0063] Next, in step 204 in which a GAN (i.e., a generator-discriminator network (NW)) is trained using the designs and optical images, the image and design segment pairs generated by step 202 can be used as a training set, with the design segments designated as training inputs and the corresponding images designated as training outputs. Training of the GAN can be performed as further described herein. In some embodiments, a computer subsystem can be configured to perform steps 200, 202, and 204. As another alternative, different computer subsystems or systems can be configured to perform steps 200, 202, and 204. For example, a first computer subsystem can be configured to generate the training set by performing steps 200 and 202, a second computer subsystem can be configured to train the GAN in step 204, and the first and second computer subsystems can or can not be included in the computer subsystems of the embodiments described herein.

[0064] Training can include inputting the training inputs into the GAN and altering one or more parameters of the GAN until the output produced by the GAN matches (or substantially matches) the training outputs. Training can include altering any one or more trainable parameters of the GAN. For example, the one or more parameters of the GAN trained by the embodiments described herein can include one or more weights of any layer of the GAN having trainable weights. In one such example, the weights can include weights of a convolutional layer rather than a pooling layer.

[0065] In one embodiment, the GAN is configured as a conditional GAN (cGAN). Generally, a GAN consists of two adversarial models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a given sample came from the training data rather than G. G and D can be multi-layer perceptrons, i.e., non-linear mapping functions. The generator builds a mapping function from a prior noise distribution p z (z) to data space G(z; θ g ) to learn a generator distribution p g over data x, where G is a differentiable function represented by a multi-layer perceptron with parameters θ g . The generator is trained to produce images that cannot be distinguished from real images. The discriminator, trained adversarially, is trained to detect fakes created by the generator. Both the generator and the discriminator are trained as well as possible so that the generator produces very good “fake” images.

[0066] When extending the GAN to a conditional model (cGAN), both the discriminator and the generator are conditioned on some extra information y by feeding it into the discriminator and the generator as an extra input layer.

[0067] GANs are inspired by game theory, where the generator G and the critic (i.e., the discriminator D) compete with each other to make each other stronger. The goal is to make the generated data distribution p gThe real sample distribution p is evaluated by the Jensen-Shannon divergence as much as possible r :

[0068]

[0069] where D KL is the Kullback-Leibler divergence defined as:

[0070] and

[0071]

[0072] The generator outputs synthetic samples given a random noise variable input z. The generator is trained over time to capture the real data distribution by having the discriminator reject images it deems to be poor fakes.

[0073] For conditional GANs, the generator and discriminator are conditioned on some additional information y for the design snippet in the embodiments described herein. The following min-max game for the objective L(D, G) describes the setup:

[0074]

[0075] This means that G is trained to increase the chances that D produces high probabilities for fake instances, thus minimizing We also need to ensure that the discriminator's decision is accurate based on real data, and given a fake sample G(z), the discriminator is expected to output a probability close to zero by maximizing

[0076] Figure 5 cGAN training is shown to be performed by providing the generator with a design image. In particular, as Figure 5 ​​As shown in the figure, additional information y 300 (a design image or a conditional image) is input to the generator network 302 along with the latent space vector z 301. Thus, the generator network 302 outputs a generated patch image 304. In this case, the original patch image x 306 is an actual optical image of the sample at the locations on the sample where the parts of the design shown in the design image 300 are formed. In this way, the additional information y 300, e.g., a design image, can be a training input in a training set, and the original patch image x 306 can be its corresponding image of the sample designated as a training output in the training set. The generated patch image 304 and the additional information y 300 combine as a first input to the discriminator network 310. The original patch image x 306 and the additional information y 308 (design image) combine as a second input to the discriminator network 310. In this case, the additional information y 300 and 308 are merely different instances of the same part of the same design. The discriminator network can generate an output P(True) 312, which is a probability that the generated patch image is true or a good “fake” image. P(True) can be input to a loss function 314.

[0077] Each of the design image, original patch image, and generated patch image shown in the figure is not meant to illustrate any particular sample or its characteristics for which the embodiments described herein can be used. In a similar manner, each of the original patch image and generated patch image shown in the figure is not meant to illustrate any particular actual or simulated image that can be generated for a sample. Rather, the design image, original patch image, and generated patch image shown in the figure are merely intended to facilitate an understanding of the embodiments described herein. The images actually input from and output by the generator will vary depending on the sample and its characteristics related to its design, and the configuration of the imaging system that generates actual images of the sample, which is used to train the GAN, to thereby affect the simulated images generated by the GAN.

[0078] The loss function 314 can be any suitable loss function known in the art, such as the loss function disclosed by Isola et al. in “Image-to-Image Translation with Conditional Adversarial Networks,” The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1125-1134, which is incorporated herein by reference as if set forth in its entirety. The embodiments described herein can be further configured as described in this reference.

[0079] Figure 5The image shows an example of a generator that can be included in an embodiment of a GAN configured as described herein. Generally, a GAN may comprise an encoder-decoder network that progressively downsamples up to a bottleneck layer and then reverses the process. For example... Figure 5 As shown, the generator may include encoder 500 and decoder 502. Each of blocks 506, 508, and 510 shown in encoder 500 represents an instance of the final output layer size after repeated convolutions, batch normalization, and rectified linear unit (ReLU) activation, with maximum set applied at the end of each segment. Although encoder 500... Figure 5 The encoder is shown as containing three blocks, but it can contain any suitable number of blocks, determined according to any suitable method known in the art. Furthermore, each of the blocks, convolutional layers, batch normalization layers, ReLU layers, and aggregation layers can have any suitable configuration known in the art. Input 504 (which is part of the design data in the embodiments described herein) can be input to block 506, the output of block 506 can be input to block 508, and so on. The encoder can generate feature layer 512.

[0080] The decoder may also contain multiple blocks that perform different functions on the feature layer 512 input to the decoder. Each of blocks 514, 516, and 518 in the decoder represents an instance of the final output layer size after repeated upsampling (transposed convolution) and ReLU activation. Although decoder 502 in Figure 6 The decoder is shown as containing three blocks, but it can contain any suitable number of blocks, determined according to any suitable method known in the field. Each of the blocks included in the decoder, as well as the upsampling and ReLU layers, can have any suitable configuration known in the field. The feature layer 512 generated by the encoder can be input to block 514, the output of block 514 can be input to block 516, and so on. The output 520 of the decoder can be any of the analog images described herein.

[0081] In some examples, a GAN may include skip connections 522 between corresponding blocks in the encoder and decoder (e.g., between blocks 506 and 518, between blocks 508 and 516, and between blocks 510 and 514). Skip connections can be used to transfer low-level information already learned between blocks. Skip connections can have any suitable configuration determined according to any suitable method known in the relevant domain. Figure 6 The numbers below the inputs and outputs in the encoder indicate the size of the input and output, respectively. The number below the block in the encoder indicates the size of the block's output, and the number below the block in the decoder indicates the size of the input for each block.

[0082] Figure 6The image shows an example of a discriminator that can be included in an embodiment of a GAN configured as described herein. The input 600 of the discriminator may contain two images ( Figure 6 (Not shown): The image generated by the GAN's generator and the original image. The discriminator may contain several layers with layers 602, 604, 606, 608, and 610, each of which may contain some combination of convolutional layers, ReLU layers, and maximum-size layers. The convolutional layers, ReLU layers, and maximum-size layers may have any suitable configuration known in the relevant domain. The output 612 of the discriminator may be P(x), the probability that the simulated image matches the original image well or the probability that the simulated image is a good "fake" image or a bad "fake" image. Figure 4 The number below the input in the discriminator indicates the size of the input. The number below the layer in the discriminator indicates the size of the layer's output. Although the discriminator... Figure 4 The discriminator in the GAN included in the embodiments described herein may have any suitable number of layers as determined by any suitable manner known in the art.

[0083] Additional descriptions of the general architecture and configuration of GANs and cGANs can be found in Goodfellow et al.'s "Generative Adversarial Networks" (arXiv:1406.2661, June 10, 2014, p. 9) and Mirza et al.'s "Conditional Generative Adversarial Networks" (arXiv:1411.1784, November 6, 2014, p. 7), which are incorporated herein by reference as if cited in their entirety. The embodiments described herein can be further configured as described in these references.

[0084] The one or more computer subsystems are configured to generate a simulated image of the specimen by inputting the portion of the design data of the specimen into the GAN. The computer subsystems can be configured to input the portion of the design data of the specimen into the GAN in any suitable manner known in the art. Although some embodiments are described herein (for clarity and simplicity only) as generating a "simulated image" of a specimen, the GANs described herein can be used to generate any number of simulated images limited only by the portion of the design data input to the GAN. Additionally, although some embodiments are described herein (also for clarity and simplicity only) as generating a simulated image of a "specimen," the embodiments described herein are not limited to generating a simulated image of only one specimen. In some embodiments, generating a simulated image of only one specimen can be all that is required for some applications, such as when the simulated image can be used to verify a reference image for a plurality of specimens or when the simulated image is used to train an augmented defect image for another deep learning (DL) or machine learning (ML) model or network.

[0085] In one embodiment, the simulated image is a simulated optical image. In this way, the simulated image illustrates how the specimen would appear in one or more actual images generated by an optical imaging system, such as an optical inspection system. As such, the simulated image can represent (e.g., correspond to, simulate, or approximate) an image of the specimen that can be generated by the optical imaging system. Such simulated optical images can be used for inspection applications in various ways further described herein. Thus, one novel feature of the embodiments described herein is that they provide a method for improved optical defect inspection using a GAN or cGAN to artificially create optical patch images.

[0086] In another embodiment, the specimen for which the simulated image is generated is not included in the one or more specimens. For example, the one or more specimens used to generate training data (which is then used to train the GAN) can be different than the specimen for which a simulated image is generated using the GAN. In this way, one or more training specimens can be used to generate training data, and simulated images can be generated for "run-time" specimens. The training specimens and the run-time specimens can be the same type of specimen, e.g., they can have the same design and can be processed using the same manufacturing process steps, but as further described herein, this need not always be the case. Additionally, a trained GAN can be used to generate simulated images for more than one run-time specimen.

[0087] In another embodiment, the design of the sample for which the simulated image is generated is different from the design(s) of the one or more samples. For example, as described above, the sample for which the simulated image is generated can not be the same as (or included in) the one or more samples used to train the GAN. In general, the one or more samples used to train the GAN and the sample for which the simulated image is generated can have the same design and can have been processed in the same process (and thus belong to the same "tier"). Training the GAN in this way will ensure that the simulated image is most similar to an actual image.

[0088] But in some cases, the first sample can have enough characteristics (e.g., patterned features, materials, etc.) similar to the second sample that even though the first and second samples do not have the same design, the GAN trained on the first sample can be used to generate a simulated image of the second sample. In such cases, the simulated image generated by the trained GAN is applied in an optical mode that is the same as the optical mode in which it was trained. In other words, as described further herein, a GAN trained to generate a simulated image produced by one imaging mode is not necessarily suitable for generating a simulated image produced by another imaging mode. Thus, if two samples have at least some similarity in their design, if they are or will be imaged in the same way, then one of the samples can be used to train a GAN, and the trained GAN can be used to generate a simulated image of the other of the samples.

[0089] In this way, the trained GAN can be slightly modified to generate simulated images of samples for which it was not necessarily trained. In one such example, if two different samples having two different designs have at least some common patterned features in portions of the designs formed from similar materials and having the same or similar dimensions (e.g., similar memory array regions), then a GAN trained for one of the samples is able to produce simulated images of synthetic defects in the portions of the designs for the other of the samples. Even though the GAN trained for one sample is not able to produce simulated images for the other sample having a different design, if there is some similarity between the samples, the trained GAN can be used as a starting configuration for retraining for the other sample to create a different GAN. This retraining can be performed as described herein.

[0090] In one embodiment, the simulated image comprises an amplified defect image of the synthetic defect. In this way, the simulated image illustrates how the synthetic defect appears in one or more actual images generated by tools such as a testing system. Thus, the simulated image can represent (e.g., correspond to, simulate, or approximate) an image of a synthetic defect that can be generated by a testing system or other tools. The term "synthetic" defect, as used herein, is broadly defined as a defect artificially created in the design of a sample in one of the several ways described herein. Therefore, a "synthetic" defect is not a defect that has already been detected on the sample, but its characteristics can be determined based on this detected defect. Instead, this "synthetic" defect is created in the design of the sample by intentionally manipulating originally defect-free parts of the design. When this manipulated part of the design is input into a trained GAN as described herein, the resulting simulated image is referred to as an "amplified" defect image because the simulated sample image has been amplified with a defect that does not actually exist on the sample.

[0091] Figure 4 This illustrates one embodiment of the steps that can be performed on the generation of artificial images using a GAN. Specifically, as... Figure 2 As shown, one or more computer subsystems can input a design image 400 (conditional image) with added defect 402 into a trained generator network 404. Added or synthesized defects can be created in the portion of design data shown in the design image 400 (as further described herein), which can be performed by one or more computer subsystems or another system or method. The trained generator network can output a generated patch image 406 showing defect 408. In this way, a synthesized defect is an added defect 402 in the portion of design data shown in the design image 400, which is input by one or more computer subsystems into the GAN to generate the simulated image 406.

[0092] Therefore, as Figure 2 As demonstrated, once a GAN is trained, it can be used to generate simulated optics and other images. In this embodiment, the trained generator network is used to create artificially introduced patches of defects in a design fragment, creating seemingly realistic images. These generated patch images can then be used for data augmentation, such as augmenting other non-simulated patch images and information to train another ML network, which can be performed as further described herein.

[0093] In one embodiment, the number of actual defects that can be detected by the inspection system on one or more samples and on the samples is insufficient for training an ML model. For example, the embodiments described herein can be used to generate simulated images that become amplified defect images as synthetic defects (which can then be used to amplify training data), which is particularly advantageous when the number of DOI events is limited (a very common situation). Thus, another advantage of the embodiments described herein is that they can use ML techniques such as DL, random forests, etc., even when the original number of DOI instances is substantially low. The embodiments described herein also advantageously enable the use of ML techniques when no DOI instances are available; for example, as further described herein, when there is no optical image of a DOI at all, a simulated image can be generated by the embodiments described herein based on knowledge of how a DOI should be viewed in the design space within a design fragment, and the simulated image can then be used to train an ML model.

[0094] In another embodiment, one or more computer subsystems are configured to generate portions of the design data for samples input to the GAN by using synthetic defects to modify the original design data of the samples. For example, as Figure 2 As shown in step 206, one or more computer subsystems can select DOI candidates from thermal scans examined by scanning electron microscopy (SEM) and locate them in the design. In one such example, if as Figure 2 As shown in the example, a hot scan is performed on the GAN training. Events detected by the hot scan are then sampled and subsequently (for example,) sent to a SEM or another suitable defect inspection tool for defect inspection. Defect inspection is performed to determine which sampled events are actual defects and which are interfering, and to determine the defect type or classification of the actual defects. In this way, defect inspection can determine which detected events are DOIs and identify the type or classification of the DOIs (when different types of DOIs may exist on the sample). Step 206 is optional because in some cases, the user will know what type of DOI they are looking for. For example, chip manufacturers typically know what defects they are looking for.

[0095] Next, as Figure 7 As shown in step 208, one or more computer subsystems can introduce the same defect type from the thermal scan into the design fragment. If the samples for which amplified defect images are generated have the same design as the one or more samples used to generate the training set, then the design fragments used in steps 200, 202, and 204 can also be used in step 208. If the samples for which amplified defect images are generated have a different design than the one or more samples used to generate the training set, then the design fragments used in steps 200, 202, and 204 may be different from the design fragments used in step 208.

[0096] However, in either case, the DOI selected in step 206 can be used in step 208 to create a modified design fragment that includes the DOI. For example, if a bridge-type DOI is selected in step 206, the original design data for the specimen can be modified by the computer subsystem to include a bridge-type structure at a different location in the design than the location at which the bridge-type DOI was detected. The location in the design at which the DOI is created can be selected based on the type of patterning feature located at or near the detected DOI, for example, so that the modified design fragment and the original design fragment contain the same or at least similar patterning structures (shape, size, spacing relative to each other, orientation, etc.) at the location of the detected DOI. In one example, if a bridge-type DOI is detected between two patterning features having a particular orientation and spacing relative to each other, a bridge-type patterning feature can be created at any one or more other locations in the design that contain the same two patterning features having the same relative orientation and spacing. Such locations can be identified in any suitable manner, for example, by pattern searching the original design based on the image of the detected DOI or the original, defect-free design fragment at the location of the detected DOI.

[0097] The original design data for the specimen can be modified as described herein using electronic design automation (EDA) tools. The EDA tools can include any suitable commercially available EDA tools. Additionally, modifying the original design data as described herein can be automated using a programmable / graphical EDA editor that can include any suitable EDA software, hardware, system, or method. In some such embodiments, one or more computer subsystems described herein, such as computer subsystem 102, can be configured as or can be computer subsystems included in EDA tools.

[0098] If the original design data for the specimen (e.g., computer aided design (CAD)) is available, "legal" synthetic defect instances can be injected directly. For example, DOIs such as open circuits, short circuits, "mouse bites," "pops," etc. can be presented (drawn) in various sizes, which can be automated based on the description of the DOI. Using EDA tools, these presented DOIs can be located in "legal" locations of the geometry rather than random locations. In one example, a short circuit is a metal connection between two copper lines. For this DOI, we can just add a small stub at the strategic pinch point in the design. When the user, such as a chip manufacturer, knows what defect it is looking for, it can also draw it in the design file manually.

[0099] As described herein, using synthetic defect modification of original design data for a sample can also be performed using a starting module configured to alter a design to create synthetic defects in the design. For example, a neural network can be trained by a defect hallucination system such as the defect hallucination system proposed by GoogLeNet starting for natural scene images. Then, the conventional neural network pre-trained for defects can participate in these reversals to create new defect types on other geometries. An example of a system and method for performing GoogLeNet starting can be found in Szegedy et al., "Going Deeper with Convolutions," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, 9 pages, which is incorporated herein by reference as if set forth in its entirety. Embodiments described herein can be further configured as described in this reference.

[0100] Next, as shown in step 210, one or more computer subsystems can process the design segment generated by step 208 through a previously trained GAN. Then, the trained GAN can output an artificially generated image, as shown in step 212. In addition to this, the image output by the GAN in step 212 can be used as input data to train other learning algorithms, as shown in step 214, which can be performed as further described herein.

[0101] In some such embodiments, the one or more computer subsystems are configured to determine one or more characteristics of the synthetic defects based on one or more defects detected on the one or more samples. For example, as described above, in some instances events can be detected on the one or more samples used to generate the training set for training the GAN, and DOIs can be identified and selected from the events. Information for the DOIs can then be used to determine one or more characteristics of the synthetic defects. Such information can include any information determined for or determinable for a DOI, such as location, size, shape, orientation, texture or roughness, a patterned feature on which the DOI is located, a patterned feature in which the DOI is located, patterned features located near the DOI, and so forth. Such information can be determined or generated by the inspection tool that detected the events, the review tool that re-inspected the events and identified one or more of them as a DOI, the metrology tool that measured one or more characteristics of the identified DOI, or some combination thereof. For example, as described above, a pattern search can be performed based on the defect image or its corresponding design fragment to find other similar instances of the pattern in the design, and then the design of the other instances can be modified as described herein to create a DOI at the other instances. Other DOI characteristics, such as those described above, can also be used to create the modified design (e.g., in the case of a bridge-type DOI, the size, orientation, roughness, and so forth of the bridge-type structure). Thus, the original design of the sample can be modified based on one or more characteristics of one or more DOIs detected on the sample used to generate the training set.

[0102] In another such embodiment, the one or more computer subsystems are configured to determine one or more characteristics of the synthetic defects without information of one or more actual defects detected on the one or more samples or sample. For example, even if no DOI instances exist at all, the one or more computer subsystems can modify a design fragment of a sample by introducing a defect into the sample and input the modified design fragment into the GAN to create a corresponding patch image that can then be used to train a neural network for an application such as nuisance filtering. Thus, one novel feature of the embodiments described herein is that they provide a method for improved optical defect inspection using a GAN or cGAN by modifying a design file by introducing a defect into the design file and generating a corresponding optical patch image even if no defect instances exist at all.

[0103] When no instances of a defect exist at all, one or more characteristics of a synthetic defect can be determined in various possible ways. For example, as described above, in many instances in which embodiments described herein can be implemented, a user can know which types of DOIs they are interested in. This knowledge can be based on prior knowledge about the types of DOIs typically seen on samples produced in the same or similar processes as the sample for which simulated images are being generated. This prior knowledge can be acquired through experimentation, heuristics, theory, and the like. For example, even if a sample's design is substantially different, particular patterned structures that can be located in every one of the design can be susceptible to a particular type of DOI. In one particular instance, an alternating line / space pattern in which the widths of the lines and spaces are relatively small can be susceptible to bridge-type defects formed between the lines. Thus, when a user sees this group of patterns in a new design, they can surmise that bridge-type defects can exist in such patterns and thus can create such defects in the design of the sample. The artificially created design segment for such defects can then be input by the computer subsystems described herein to the GAN to thereby generate simulated images of the defects.

[0104] In another such embodiment, the synthetic defect is a DOI of a first type, the one or more computer subsystems are configured to generate an additional portion of the design data for the sample by modifying the original design data for the sample using an additional synthetic defect, the additional synthetic defect is a DOI of a second type that is different from the first type, and the one or more computer subsystems are configured to generate an additional simulated image of the sample by inputting the additional portion of the design data into the GAN. For example, embodiments described herein can be advantageously used to easily generate simulated images of instances of different DOI types. In other words, the GAN described herein can generate simulated images of different types of DOIs in the same design of the same sample.

[0105] Many of the samples described herein can have multiple types of DOIs present on them, and a user can be interested in more than one (or even all) of the multiple types of DOIs. Different DOIs can have substantially different characteristics. For example, bridge-type defects can have substantially different characteristics than pinhole-type defects, but both types of defects can be present on a sample and can be of interest to a user. Different types of DOIs can be located in different portions of a design or in the same portion of a design. Thus, a design segment modified to include a synthetic defect as described herein can include a single synthetic defect or multiple synthetic defects. Regardless of the number of synthetic defects in a modified design segment and / or the number of modified design segments (which can be used for multiple instances of the same DOI type and / or different DOI types), the GAN trained as described herein will be able to produce substantially high-quality simulated images for each of the DOI types and each of the DOI instances.

[0106] In an additional such embodiment, one or more computer subsystems are configured to determine one or more characteristics of synthetic defects based on one or more defects detected on one or more additional samples, the one or more additional samples being formed in one or more process steps that are altered prior to the one or more process steps being used to form the sample, the ML model is trained to perform one or more functions on the one or more additional samples, and the one or more computer subsystems are configured to retrain the ML model using the simulated images. In this way, one advantage of the embodiments described herein is that they can be used to mitigate process variation by generating images with instances of DOIs from a first wafer and instances of images from a second wafer.

[0107] In one such instance, assume that a pseudo-positive filter, often referred to as an nuisance event filter or NEF, has been set on wafer 1. In some instances, the process can change and thus the defect attributes can change for a later wafer, here simply referred to as wafer 2. The change in defect attributes can mean that the filter set on wafer 1 is not useful for wafer 2, but also since wafer 2 is not the set wafer, there can not be any instances of defects for retraining the filter. Here it is implied that it is not possible to train the NEF classifier, which can be, for example, a random forest model. However, the system embodiments described herein can solve this problem by, for example, using an inspection system, such as the inspection systems described herein, to collect some image data on wafer 2. In this case, no new DOI information is needed. Then, one or more computer subsystems can use these images from wafer 2 to train a new GAN or retrain a previously trained GAN, which can be used as described herein and as shown in Figure 7 and 3 In one such instance, assume that a pseudo-positive filter, often referred to as an nuisance event filter or NEF, has been set on wafer 1. In some instances, the process can change and thus the defect attributes can change for a later wafer, here simply referred to as wafer 2. The change in defect attributes can mean that the filter set on wafer 1 is not useful for wafer 2, but also since wafer 2 is not the set wafer, there can not be any instances of defects for retraining the filter. Here it is implied that it is not possible to train the NEF classifier, which can be, for example, a random forest model. However, the system embodiments described herein can solve this problem by, for example, using an inspection system, such as the inspection systems described herein, to collect some image data on wafer 2. In this case, no new DOI information is needed. Then, one or more computer subsystems can use these images from wafer 2 to train a new GAN or retrain a previously trained GAN, which can be used as described herein and as shown in

[0108] In one embodiment, corresponding images of the one or more samples are generated by a first mode of an imaging system, the one or more components include an additional GAN trained using an additional training set, the additional training set including portions of design data of the one or more samples designated as additional training inputs and corresponding additional images of the one or more samples designated as additional training outputs, the corresponding additional images generated by a second mode of the imaging system different from the first mode, and the one or more computer subsystems are configured to generate additional simulated images of the sample by inputting the portions of design data of the sample into the additional GAN.

[0109] Thus, in such embodiments, different GANs can be trained to produce different modes of images or other outputs. In particular, in most cases, different modes of an inspection or other imaging tool will produce images and / or outputs that differ from one another in one of several possible ways, such as noise level, contrast, resolution, image type (e.g., DF vs. BF, optical vs. e-beam, etc.), and the like. Thus, if a GAN is trained to produce outputs or images of a mode of a tool, there is a chance that it will not be suitable to train to produce outputs or simulated images of another mode of the tool. As such, multiple GANs can be trained separately and independently, one for each mode of interest. However, the same pre-trained GAN network can be used for each mode, but this is not necessary. Each trained GAN can then be used to generate a mode-specific dataset. However, the input to each differently trained GAN can be the same design of the same design fragment or different portions, as the design of the specimen does not change with mode.

[0110] If GANs are trained or used to generate simulated defect images, the simulated images generated by the trained GANs for different modes can be used to train one or more DL or ML models or networks. For example, if the embodiments described herein use simulated images generated by GANs to generate or augment a training dataset, the training dataset can be used to train neural networks for single and multiple optical (or other) modes, which can be performed as described in U.S. Patent No. 10,115,040 to Brauer, issued October 30, 2018, and U.S. Patent No. 10,360,477 to Bhaskar et al., issued July 23, 2019, which are incorporated by reference herein as if fully set forth. The embodiments described herein can be further configured as described in these patents. Training one or more DL or ML models or networks using a training set can depend on the intended use of the DL or ML model or network and what its input is. For example, one DL or ML model or network can be configured to perform interference filtering or defect classification using multiple mode images as input. Thus, simulated images generated for multiple modes can be used to train the model or network. In another example, one DL or ML model or network can be configured to perform interference filtering using images from a first mode as input, and a different DL or ML model or network can be configured to perform defect classification using images from a second mode as input. In this way, simulated images generated for the first mode can be used to train one DL or ML model or network, and simulated images generated for the second mode can be used to train another DL or ML model or network.

[0111] If a GAN is trained or used to generate simulated reference images, different simulated reference images can be generated and stored as further described herein for different modes. For example, if defect detection is performed in a die-to-database manner, each mode itself requires a reference image to be subtracted from the test images generated in each mode. Thus, when different GANs are trained as described herein for different modes, different GANs can be used to generate different reference images, each of which can be used in the same inspection process but for different modes.

[0112] In some embodiments, one or more computer subsystems are configured to train an ML model using simulated images. In some such embodiments, additional samples (e.g., simulated images) can be used to augment a training dataset or by adding to other augmentation-based defect instances generated by augmentation techniques such as described in U.S. Patent No. 10,402,688 issued to Brauer et al. on September 3, 2019, which is incorporated herein by reference in its entirety. Embodiments described herein can be further configured as described in this patent. Simulated images can be added to a training set and / or used to create a new training set in any suitable manner known in the art. An ML model trained using simulated images described herein can have any suitable configuration known in the art. Training an ML model using simulated images generated as described herein can be performed as described herein or in any other suitable manner known in the art.

[0113] In one such embodiment, the ML model is configured for interference filtering. One advantage of embodiments described herein is that the methods described herein or derivatives thereof allow for reduction of interference, which in turn allows semiconductor manufacturers to make more reliable process decisions and thus not waste money using incorrect process conditions. In another such embodiment, the ML model is configured for defect classification. In this way, the ML model can identify different types of defects and assign them to different classes or groups. An ML model trained using simulated images described herein can have any suitable configuration known in the art, whether it is configured for interference filtering or defect classification.

[0114] In another embodiment, the simulated images are reference images, and the system includes an inspection system configured to detect defects on a specimen by subtracting the reference images from images of the specimen generated by the inspection system. For example, if simulated images are generated for die-to-database type inspection, one or more computer subsystems can input each given design fragment or full frame design or even full die design into a trained GAN to thereby generate a reference image. This artificial image can then be used as a reference image for defect inspection. In this way, the embodiments described herein can be used to generate artificial optical (or other) reference images to perform die-to-database inspection. Thus, one novel feature of the embodiments described herein is that they provide a method for using GANs or cGANs for improved optical defect inspection to artificially create optical reference patch images that can be used for optical die-to-database inspection. Another novel feature of the embodiments described herein is that they provide a method for using GANs or cGANs for improved optical defect inspection to implement an unsupervised DL method for optical reference image generation.

[0115] The computer subsystems can be configured to store various information, images, etc. generated by the embodiments described herein. For example, the computer subsystems can be configured to store simulated augmented defect images generated by the GANs for training ML or DL models or networks, such as the ML or DL models or networks described herein. In one such example, the computer subsystems can store the simulated defect images in a data structure or file containing a training set augmented by the simulated defect images. The simulated defect images can be stored in any suitable manner in any of the computer-readable storage media described herein.

[0116] In another example, the computer subsystems can be configured to store simulated reference images and / or ML or DL models or networks trained using one or more simulated images generated by the embodiments described herein for inspecting a specimen or another specimen of the same type. The computer subsystems can be configured to store such image(s) and / or model(s) or network(s) in a recipe or by generating a recipe for an inspection in which the reference image(s) and / or model(s) or network(s) will be used. The term "recipe" as used herein is defined as a set of instructions that can be used by a tool to perform a process on a specimen. In this way, generating a recipe can include generating information on how to perform a process, which can then be used to generate instructions for performing the process. The computer subsystems can also store any information that can be used to identify, access, and / or use the reference image(s) and / or model(s) or network(s) (e.g., a file name and its storage location, for example). The information stored for the model(s) or network(s) can also include program code, instructions, algorithms, etc. of the model(s) or network(s). The reference image(s), model(s) or network(s), or information thereof, can be stored in any of the computer-readable storage media described herein in any suitable manner.

[0117] The images, models, or networks and information thereof can be stored with any other results described herein and can be stored in any manner known in the art. The storage medium can include any storage medium described herein or any other suitable storage medium known in the art. After the information is stored, the information can be accessed in the storage medium and used by any of the method or system embodiments described herein, formatted for display to a user, used by another software module, method, or system, etc. For example, the embodiments described herein can generate an inspection recipe described above. The inspection recipe can then be stored and used by a system or method (or another system or method) to inspect a sample or other sample to thereby generate information (e.g., defect information) for the sample or other sample.

[0118] Results and information generated by performing an inspection on a sample or other sample of the same type can be used in various ways by the embodiments described herein and / or other systems and methods. Such functions include, but are not limited to, altering a process, e.g., a manufacturing process or step, that has been or will be performed on the inspected sample or another sample, in a feedback or feedforward manner. For example, the computer subsystems described herein can be configured to determine one or more changes to a process performed on a sample inspected as described herein and / or to a process that will be performed on the sample based on detected defects. The changes to the process can include any suitable change to one or more parameters of the process. The computer subsystems described herein preferably determine the changes such that defects can be reduced or prevented on other samples on which a revised process is performed, can be corrected or eliminated on the sample in another process performed on the sample, can be compensated for in another process performed on the sample, etc. The computer subsystems described herein can determine such changes in any suitable manner known in the art.

[0119] Those changes can then be sent to a semiconductor manufacturing system (not shown) or a storage medium (not shown) accessible to the computer subsystems and semiconductor manufacturing system. The semiconductor manufacturing system can or can not be part of the system embodiments described herein. For example, the computer subsystems and / or inspection systems described herein can be coupled to a semiconductor manufacturing system, e.g., via one or more common elements (e.g., a housing, a power supply, sample handling devices or mechanisms, etc.). The semiconductor manufacturing system can include any semiconductor manufacturing system known in the art, such as a lithography tool, an etch tool, a chemical mechanical polishing (CMP) tool, a deposition tool, and the like.

[0120] Thus, as described herein, embodiments can be used to set up a new inspection process or recipe. Embodiments can also be used to modify an existing inspection process or recipe, whether it is for a sample or an inspection process or recipe created for one sample and adapted for use with another sample.

[0121] The embodiments described herein are not limited to inspection recipe or process creation or modification. For example, the embodiments described herein can also be used to set up or modify recipes or processes for metrology, defect review, etc. in a similar manner. In particular, the GANs described herein can be trained depending on the process being set up or modified (e.g., to generate simulated outputs that simulate the actual outputs that would be generated by the process). Then, depending on the process or recipe being set up or modified, the simulated outputs can be used to set up a recipe for the process, whether it stores simulated reference images for the process or trains a DL or ML model or network for the process. Such output processing methods can include, for example, defect re-detection methods for re-detecting defects in outputs generated by a defect review system.

[0122] In a similar manner, the embodiments described herein can be used not only to select output processing parameters and methods, but also to select output acquisition parameters or modes that detect light, electrons, ions, etc. from a sample using, for example, an inspection system, a metrology system, or a defect review system. This output acquisition parameter selection can include training and using different GANs to generate simulated images of different output acquisition parameters or modes, which can be performed as further described herein. Then, the generated simulated images can be compared and evaluated to select the mode or parameter that is best suited for any process. Thus, the embodiments described herein can be used not only to set up or modify an inspection process, but also to set up or modify any quality control type process performed on the samples described herein and any parameters of such a process.

[0123] Each of the embodiments of each of the above-described systems can be combined together into one single embodiment.

[0124] Another embodiment relates to a computer-implemented method for generating a simulated image of a sample. The method includes generating a simulated image of a sample by inputting a portion of design data of the sample into a GAN. The input is performed by one or more computer subsystems. One or more components are performed by the one or more computer subsystems. The one or more components include the GAN. The GAN is trained using a training set that includes a portion of design data of one or more samples designated as training input and corresponding images of the one or more samples designated as training output.

[0125] Each of the steps of the method can be performed as further described herein. The method can also include any other steps that can be performed by the systems, computer subsystems, components, and / or GANs described herein. The computer subsystems can be configured according to any of the embodiments described herein, such as the computer subsystem 102. The one or more components and the GAN can also be configured according to any of the embodiments described herein. The method can be performed by any of the system embodiments described herein.

[0126] Additional embodiments relate to a non-transitory computer readable medium storing program instructions executable on one or more computer systems to perform a computer-implemented method for generating simulated images of a sample. ​ One such embodiment is shown in FIG. 7. In particular, as shown in FIG. 7, a non-transitory computer readable medium 700 includes program instructions 702 executable on a computer system 704. The computer-implemented method can include any step of any method described herein. ​ One such embodiment is shown in FIG. 7. In particular, as shown in FIG. 7, a non-transitory computer readable medium 700 includes program instructions 702 executable on a computer system 704. The computer-implemented method can include any step of any method described herein.

[0127] The program instructions 702 implementing a method, such as the methods described herein, can be stored on the computer readable medium 700. The computer readable medium can be a storage medium such as a magnetic or optical disk, magnetic tape or any other suitable non-transitory computer readable medium known in the art.

[0128] The program instructions can be implemented in any of various ways including, for example, as program-based techniques, component-based techniques, and / or object-oriented techniques. For example, the program instructions can be implemented using ActiveX controls, C++ objects, JavaBeans, Microsoft Foundation Classes ("MFC"), SSE (Streaming SIMD Extensions), or other techniques or methods, as desired.

[0129] The computer system 704 can be configured according to any of the embodiments described herein.

[0130] Various modifications and alternative embodiments of the aspects of the application will be apparent to those skilled in the art in view of this description, which is to be construed as illustrative only. For example, methods and systems for generating simulated images of a sample are provided. Therefore, this description should not be taken as limiting in scope, but rather as merely describing one general way of implementing the application. It is to be understood that the form of the application herein shown and described is to be taken merely as a preferred example of the same, and it is not intended to limit the scope of the application as described in the appended claims. Changes can be made in the elements and materials as well as in the steps of the methods described herein without departing from the spirit and scope of the application as described in the following claims.

Claims

1. A system configured to generate simulated images of samples, the system comprising: One or more computer subsystems; and One or more components, executed by one or more computer subsystems, wherein the one or more components include a generative adversarial network trained using a training set, the training set including portions of design data for one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs. and The one or more computer subsystems are configured to: A portion of the design data for the samples input to the generative adversarial network is generated by modifying the original design data of the samples using synthetic defects. as well as A simulated image of the sample is generated by inputting a portion of the design data of the sample into the generative adversarial network, wherein the simulated image includes an amplified defect image of the synthetic defect, wherein the synthetic defect is a first type of defect of concern, wherein one or more computer subsystems are further configured to generate an additional portion of the design data of the sample by modifying the original design data of the sample using an additional synthetic defect, wherein the additional synthetic defect is a second type of defect of concern different from the first type, and wherein the one or more computer subsystems are further configured to generate an additional simulated image of the sample by inputting the additional portion of the design data into the generative adversarial network.

2. The system of claim 1, wherein the generative adversarial network is configured as a conditional generative adversarial network.

3. The system of claim 1, wherein the number of actual defects that can be detected by the inspection system on the one or more samples and on the samples are insufficient to train the machine learning model.

4. The system of claim 1, wherein the one or more computer subsystems are further configured to determine one or more characteristics of the synthetic defect based on one or more defects detected on the one or more samples.

5. The system of claim 1, wherein the one or more computer subsystems are further configured to determine one or more characteristics of the synthetic defect without requiring information about one or more actual defects detected on the one or more samples or on the samples.

6. The system of claim 1, wherein the one or more computer subsystems are further configured to determine one or more characteristics of the synthetic defect based on one or more defects detected on one or more additional samples, wherein the one or more additional samples are formed in one or more process steps, wherein a change to the one or more process steps occurs before the samples are formed using the one or more process steps, wherein a machine learning model is trained to perform one or more functions on the one or more additional samples, and wherein the one or more computer subsystems are further configured to retrain the machine learning model using the simulated images.

7. The system of claim 1, wherein the corresponding images of the one or more samples are generated by a first mode of the imaging system, wherein the one or more components further include an additional generative adversarial network trained using an additional training set, the additional training set including a portion of the design data of the one or more samples designated as additional training input and corresponding additional images of the one or more samples designated as additional training output, wherein the corresponding additional images are generated by a second mode of the imaging system different from the first mode, and wherein the one or more computer subsystems are further configured to generate additional simulated images of the samples by inputting the portion of the design data of the samples into the additional generative adversarial network.

8. The system of claim 1, wherein the sample for generating the simulated image is not included in the one or more samples.

9. The system of claim 1, wherein the design of the sample for generating the simulated image differs from one or more designs of the one or more samples.

10. The system of claim 1, wherein the one or more computer subsystems are further configured to use the simulated images to train a machine learning model.

11. The system of claim 10, wherein the machine learning model is configured for interference filtering.

12. The system of claim 10, wherein the machine learning model is configured for defect classification.

13. The system of claim 1, wherein the simulated image is a reference image, and wherein the system further comprises an inspection system configured to detect defects on the sample by subtracting the reference image from an image of the sample generated by the inspection system.

14. The system of claim 1, wherein the simulated image is a simulated optical image.

15. The system of claim 1, wherein the sample is a wafer.

16. A non-transitory computer-readable medium storing program instructions executable on one or more computer systems to perform a computer-implemented method for generating simulated images of samples, wherein the computer-implemented method includes: A portion of the design data for the sample input to the generative adversarial network is generated by modifying the original design data of the sample using synthetic defects. as well as A simulated image of the sample is generated by inputting a portion of the design data of the sample into the generative adversarial network, wherein the input is performed by one or more computer systems, wherein the simulated image includes an amplified defect image of the synthetic defect, wherein the synthetic defect is a first type of defect of concern, wherein the one or more computer subsystems are further configured to generate an additional portion of the design data of the sample by modifying the original design data of the sample using an additional synthetic defect, wherein the additional synthetic defect is a second type of defect of concern different from the first type, and wherein the one or more computer subsystems are further configured to generate an additional simulated image of the sample by inputting the additional portion of the design data into the generative adversarial network; One or more components are executed by the one or more computer systems, wherein the one or more components include the generative adversarial network; and The generative adversarial network is trained using a training set, which includes portions of design data for one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs.

17. A computer-implemented method for generating simulated images of samples, comprising: A portion of the design data for the sample input to the generative adversarial network is generated by modifying the original design data of the sample using synthetic defects. as well as A simulated image of the sample is generated by inputting a portion of the design data of the sample into the generative adversarial network, wherein the input is performed by one or more computer subsystems, wherein the simulated image includes an amplified defect image of the synthetic defect, wherein the synthetic defect is a first type of defect of concern, wherein the one or more computer subsystems are further configured to generate an additional portion of the design data of the sample by modifying the original design data of the sample using an additional synthetic defect, wherein the additional synthetic defect is a second type of defect of concern different from the first type, and wherein the one or more computer subsystems are further configured to generate an additional simulated image of the sample by inputting the additional portion of the design data into the generative adversarial network; One or more components are executed by the one or more computer subsystems, wherein the one or more components include the generative adversarial network; and The generative adversarial network is trained using a training set, which includes a portion of design data for one or more samples designated as training inputs and corresponding images of the one or more samples designated as training outputs.

Images