Generating a training set usable for inspecting a semiconductor sample

Abstract

A system and method of generating a training set for training a deep neural network usable for sample inspection is provided. The method includes, for each given training image in a set: i) generating a first batch of training patches, including: cropping the given training image into a first plurality of original patches; and augmenting at least a portion of the first plurality of original patches so as to simulate changes resulting from physical processes of the sample; and ii) generating a second batch of training patches, including: shifting a plurality of first locations on the given training image to obtain a second plurality of original patches, and repeating the augmenting of the second plurality of original patches to generate a second plurality of augmented patches; and including at least the first second batch of training patches corresponding to each given training image in the training set.

Description

[0001] This application is a divisional application of Chinese patent application No. 202110291440.7, filed on March 18, 2021, entitled “Generating a training set that can be used to examine semiconductor samples”. Technical Field

[0002] The currently published topics generally relate to the field of sample inspection, and more specifically to automating sample inspection. Background Technology

[0003] Current demands for high density and performance associated with the ultra-large-scale integration of manufactured devices require submicron features, increased transistor and circuit speeds, and improved reliability. These demands necessitate the formation of device features with high precision and uniformity, which in turn requires careful monitoring of the manufacturing process, including automated inspection of the device while it is still in the form of a semiconductor wafer.

[0004] The term “sample” as used in this specification should be broadly interpreted to encompass any kind of wafer, mask, and other structure, and combinations and / or portions thereof, used in the manufacture of semiconductor integrated circuits, magnetic heads, flat panel displays, and other articles of semiconductor manufacturing.

[0005] The term "inspection" as used in this specification should be broadly interpreted to encompass any kind of metrology-related operation, as well as operations relating to the detection and / or classification of defects in a sample during sample fabrication. Inspection is provided by using non-destructive inspection tools during or after the fabrication of the sample to be inspected. As a non-limiting example, the inspection process may include runtime scanning (in single or multiple scans), sampling, examination, measurement, classification, and / or other operations relating to the sample or portions thereof using the same or different inspection tools. Similarly, inspection may be provided prior to the fabrication of the sample to be inspected and may include, for example, generating inspection formulations(s) and / or other setup operations. It should be noted that, unless specifically stated otherwise, the term "inspection" or its derivatives as used in this specification are not limited with respect to the resolution or size of the area to be inspected. As a non-limiting example, various non-destructive inspection tools include scanning electron microscopes, atomic force microscopes, optical inspection tools, etc.

[0006] As a non-limiting example, runtime inspection can employ a two-stage procedure, for instance, inspecting the sample and then examining the sampling locations of potential defects. During the first stage, the surface of the sample is inspected at high speed and relatively low resolution. In this first stage, a defect map is generated to indicate suspected locations on the sample with a high probability of defect. During the second stage, at least some of the suspected locations are analyzed more thoroughly at relatively high resolution. In some cases, both stages can be implemented using the same inspection tool, and in other cases, the two stages are implemented using different inspection tools.

[0007] Inspection processes are used at various steps during semiconductor manufacturing to detect and classify defects on samples, as well as to perform metrology-related operations. The effectiveness of inspection can be improved through the automation of processes(s), such as Automatic Defect Classification (ADC), Automatic Defect Review (ADR), etc. Summary of the Invention

[0008] According to certain aspects of the currently disclosed subject matter, a method is provided for generating a training set for training a deep neural network (DNN) that can be used for sample inspection, the method being performed by a processor and memory circuitry (PMC), the method comprising: obtaining a set of training images, each training image representing at least a portion of a sample; for each given training image in the set: i) generating a first batch of training patches, comprising: cropping the given training image into a first plurality of original patches according to a plurality of first positions on the given training image; and expanding at least a portion of the first plurality of original patches to simulate changes caused by physical processes of the sample, thereby generating a first plurality of expanded patches constituting the first batch of training patches; and ii) generating a second batch of training patches, comprising: shifting the plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches, thereby generating a second batch of training patches; and including at least the first batch of training patches and the second batch of training patches corresponding to each given training image into the training set.

[0009] In addition to the features described above, this aspect of the method according to the currently disclosed subject matter may include one or more of the features (i) to (viii) listed below in any technically possible desired combination or arrangement:

[0010] (i) The training set can be used to train a DNN that can be used for at least one inspection process selected from the group consisting of: automatic defect detection based on runtime images, automatic defect review, automatic defect classification, automatic segmentation, and automatic measurement.

[0011] (ii) The method may further include repeatedly generating a second batch of training patches one or more times until the completion criteria are met, generating one or more batches of training patches, and including said one or more batches in the training set.

[0012] (iii) The method may further include including a first plurality of original facets and / or a second plurality of original facets in the training set.

[0013] (iv) The predetermined location can be based on a grid.

[0014] (v) Shifting can be performed with random offsets in random directions.

[0015] (vi) The simulated changes may include process changes and / or gray level changes.

[0016] (vii) The method may further include including a first batch of training patches corresponding to each training image in a first epoch for training the DNN, and including a second batch of training patches corresponding to each training image in a second epoch for training the DNN.

[0017] (viii) Training images can be high-resolution images obtained from review tools.

[0018] According to other aspects of the currently disclosed subject matter, a system is provided for generating a training set for training a deep neural network (DNN) that can be used for sample inspection, the system comprising a processor and memory circuitry (PMC) configured to: obtain a set of training images, each training image representing at least a portion of a sample; for each given training image in the set: i) generate a first batch of training patches, comprising: cropping the given training image into a first plurality of original patches according to a plurality of first positions on the given training image; and expanding at least a portion of the first plurality of original patches to simulate changes caused by physical processes of the sample, thereby generating a first plurality of expanded patches constituting the first batch of training patches; and ii) generate a second batch of training patches, comprising: randomly shifting the plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches, thereby generating a second batch of training patches; and including at least the first batch of training patches and the second batch of training patches corresponding to each given training image into the training set.

[0019] This aspect of the disclosed subject matter may include one or more of the features (i) to (viii) listed above with respect to the method in any technically possible combination or arrangement, plus necessary modifications.

[0020] According to other aspects of the currently disclosed subject matter, a non-transitory computer-readable medium containing instructions, which, when executed by a computer, cause the computer to perform a method for generating a training set for training a deep neural network (DNN) that can be used for sample inspection, the method comprising: obtaining a set of training images, each training image representing at least a portion of a sample; for each given training image in the set: i) generating a first batch of training patches, comprising: cropping the given training image into a first plurality of original patches according to a plurality of first positions on the given training image; and expanding at least a portion of the first plurality of original patches to simulate changes caused by physical processes of the sample, thereby generating a first plurality of expanded patches constituting the first batch of training patches; and ii) generating a second batch of training patches, comprising: shifting the plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches, thereby generating a second batch of training patches; and including at least the first batch of training patches and the second batch of training patches corresponding to each given training image into the training set.

[0021] This aspect of the disclosed subject matter may include one or more of the features (i) to (viii) listed above with respect to the method in any technically possible combination or arrangement, plus necessary modifications. Attached Figure Description

[0022] To understand this disclosure and how it can be practiced, embodiments will now be described by way of non-limiting example only, with reference to the accompanying drawings, in which:

[0023] Figure 1 A generalized block diagram of an inspection system according to certain embodiments of the currently disclosed subject matter is shown.

[0024] Figure 2 A generalized flowchart illustrating the generation of a training set according to certain embodiments of the currently disclosed subject matter is shown, the training set being used to train a DNN that can be used to examine semiconductor samples.

[0025] Figure 3 A generalized flowchart is shown, illustrating the use of a generated training set to train a DNN according to certain embodiments of the currently disclosed subject matter.

[0026] Figure 4 The above reference illustrates implementations of certain embodiments of the currently disclosed subject matter. Figure 3 Examples of techniques for FP image segmentation are described in detail.

[0027] Figure 5 Examples of a second plurality of original facets obtained after initial position shifting are shown, according to certain embodiments of the currently disclosed subject matter.

[0028] Figure 6 Examples of a pair of original and extended facets are shown according to certain embodiments of the currently disclosed subject matter.

[0029] Figure 7 Examples of FP images and corresponding segmented diagrams generated according to certain embodiments of the currently disclosed subject matter are shown. Detailed Implementation

[0030] Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the contents of this disclosure. However, those skilled in the art will understand that the subject matter disclosed herein can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the subject matter of this disclosure.

[0031] Unless otherwise specifically stated, as will be apparent from the following discussion, it should be understood that the use of terms such as “generate,” “train,” “obtain,” “cut,” “expand,” “shift,” “repeat,” “include,” etc., throughout this specification refers to multiple actions and / or processes of a computer that manipulate and / or transform data into other data, which represent physical (such as electronic) quantities and / or physical objects. The term “computer” should be broadly interpreted to encompass any kind of hardware-based electronic device with data processing capabilities, including, by way of non-limiting example, the FPEI (Process Inspection Information) system and its corresponding portions disclosed in this application.

[0032] The terms “non-transitory memory” and “non-transitory storage medium” as used herein should be broadly interpreted to cover any volatile or non-volatile computer memory applicable to the subject matter currently disclosed.

[0033] The term “defect” as used in this specification should be interpreted broadly to encompass any kind of abnormality or undesirable feature that forms on or within the sample.

[0034] The term "design data" as used in this specification should be broadly interpreted to encompass any data that includes the layered physical design (layout) of the indicated sample. Design data may be provided by the respective designer and / or may be derived from the physical design (e.g., through complex simulations, simple geometric operations, and Boolean operations). As a non-limiting example, design data may be provided in various formats, such as GDSII format, OASIS format, etc. Design data may exist in vector format, grayscale image format, or other formats.

[0035] It should be understood that, unless otherwise specifically stated, certain features of the currently disclosed subject matter described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the currently disclosed subject matter described in the context of a single embodiment may also be provided separately or in any suitable sub-combination. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the methods and apparatus.

[0036] In view of this, note Figure 1 , Figure 1 A functional block diagram of an inspection system according to certain embodiments of the currently disclosed subject matter is shown.

[0037] Figure 1 The inspection system 100 shown can be used to inspect semiconductor samples (e.g., wafers and / or portions of wafers) as part of a sample manufacturing process. The illustrated inspection system 100 includes a computer-based system 101 capable of automatically determining metrology-related and / or defect-related information using images acquired during sample manufacturing (hereinafter referred to as process (FP) images). System 101 is referred to hereinafter as an FPEI (Process Inspection Information) system. According to certain embodiments of the currently disclosed subject matter, the FPEI system 101 can be configured to perform segmentation of images representing at least a portion of the sample, as will be referenced below. Figure 3 Described in more detail. System 101 can be operatively connected to one or more inspection tools 120. The inspection tools are configured to capture FP images and / or review (multiple) captured FP images and / or enable or provide measurements on (multiple) captured images. The FPEI system can be further operatively connected to design data server 110 and storage unit 122.

[0038] For example, FP images can be selected from images of samples (e.g., wafers or portions of wafers) captured during the manufacturing process, derivatives of captured images obtained through various preprocessing stages (e.g., images of a portion of a wafer or photomask captured by a SEM or optical inspection system; SEM images generally centered on defects to be classified by ADC; SEM images of larger areas where defects are to be located by ADR; registration images corresponding to different inspection modes at the same mask location; segmented images; height map images, etc.), and images based on computer-generated design data.

[0039] The term “(multiple) inspection tools” as used herein should be broadly interpreted to encompass any tool that may be used in inspection-related processes, including, by way of non-limiting example, imaging, scanning (in single or multiple scans), sampling, examination, measurement, classification, and / or other processes relating to a sample or a portion thereof. One or more inspection tools 120 may include one or more inspection tools and / or one or more review tools. In some cases, at least one of the inspection tools 120 may be an inspection tool configured to (typically, at relatively high speed and / or low resolution) scan a sample (e.g., an entire wafer, an entire die, or a portion of a die) to capture inspection images for the detection of potential defects. In some cases, at least one of the inspection tools 120 may be a review tool configured to capture review images of at least some of the defects detected by the inspection tool for the confirmation that a potential defect is indeed a defect. Such review tools are typically configured to (typically, at relatively low speed and / or high resolution) inspect segments of the die one at a time. Inspection tools and review tools may be different tools located in the same or different locations, or a single tool operating in two different modes. In some cases, at least one inspection tool may have metrological capabilities.

[0040] Without limiting the scope of this disclosure in any way, it should also be noted that the inspection tool 120 can be implemented as various types of inspection machines, such as optical imaging machines, electron beam inspection machines, etc. In some cases, the same inspection tool can provide both low-resolution and high-resolution image data.

[0041] The FPEI system 101 includes a processor and memory circuitry (PMC) 102, which is operatively connected to a hardware-based I / O interface 126. (See reference...) Figure 2Further details are provided. PMC 102 is configured to provide all the processing necessary to operate the FPEI system and includes a processor (not shown separately) and memory (not shown separately). The processor of PMC 102 can be configured to execute several functional modules according to computer-readable instructions implemented on non-transitory computer-readable memory contained within the PMC. Such functional modules are referred to below as being contained within the PMC. The functional modules contained within PMC 102 include training set generator 104.

[0042] According to some embodiments, the training set generator 104 can be configured to: obtain a set of training images, each training image representing at least a portion of a sample, and for each given training image in the set, generate a first batch of training patches and a second batch of training patches, including at least the first batch of training patches and the second batch of training patches corresponding to each given training image in the training set. The first batch can be generated by cropping a given training image into a first plurality of original patches based on a plurality of first positions on the given training image, and expanding at least some / parts of the first plurality of original patches to simulate changes caused by the physical processing of the sample, thereby producing a first plurality of expanded patches constituting the first batch of training patches. The second batch can be generated by shifting a plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches, thereby producing the second batch of training patches. References below. Figure 2 Describe the details of the generation process.

[0043] In some embodiments, the functional module may further include a deep neural network (DNN) 106. The DNN 106 may be configured to enable data processing using the deep neural network(s) to output application-related data based on manufacturing (FP) input data. Optionally, the PMC 102 may further include a pre-DNN module (not shown separately) and / or a post-DNN module (not shown separately), the pre-DNN module being configured to provide preprocessing before forwarding input data to the DNN, and the post-DNN module being configured to provide post-processed data generated by the DNN. (Refer to...) Figure 2 Further details on the operation of FPEI system 101, PMC102 and its functional modules.

[0044] DNN 106 may comprise a supervised or unsupervised DNN model, which includes layers organized according to a corresponding DNN architecture. As a non-limiting example, the layers of the DNN may be organized according to a convolutional neural network (CNN) architecture, a recurrent neural network (RNN) architecture, a recursive neural network (RNN) architecture, a generative adversarial network (GAN) architecture, or otherwise. Optionally, at least some of the layers may be organized into multiple DNN sub-networks. Each layer of the DNN may include multiple basic computational elements (CEs), commonly referred to in the art as dimensions, neurons, or nodes.

[0045] Typically, a computational element in a given layer can be connected to a computational element (CE) in a previous and / or subsequent layer. Each connection between a CE in a previous layer and a CE in a subsequent layer is associated with a weighted value. A given CE can receive input from a CE in a previous layer via a corresponding connection, each given connection being associated with a weighted value that can be applied to the input of that given connection. The weighted values ​​can determine the relative strength of the connection and thus the relative effect of the corresponding input on the output of the given CE. A given CE can be configured to compute activation values ​​(e.g., a weighted sum of inputs) and further derive the output by applying an activation function to the computed activation. For example, the activation function can be an identity function, a deterministic function (e.g., linear, sigmoid, threshold, etc.), a random function, or other suitable function. The output from a given CE can be sent to a CE in a subsequent layer via a corresponding connection. Similarly, as described above, each connection at the output of a CE can be associated with a weighted value that can be applied to the output of the CE before the output of the CE is received as input to a CE in a subsequent layer. Further for the weighted values, there can be thresholds (including limiting functions) associated with the connections and CEs.

[0046] The weights and / or thresholds of the deep neural network can be initially selected before training and can be further iteratively adjusted or modified during training to achieve the optimal set of weights and / or thresholds in the trained DNN. After each iteration, the difference between the actual output generated by the DNN module and the target output associated with the corresponding training data set can be determined. This difference can be referred to as the error value. Training can be determined to be complete when the loss / cost function indicating the error value is less than a predetermined value, or when a finite change in performance is achieved between iterations. Optionally, at least a portion of the DNN subnetwork (if any) can be trained separately before training the entire DNN.

[0047] The DNN input dataset used to adjust the weights / thresholds of the deep neural network is referred to as the training set below.

[0048] It should be noted that the teaching of the currently published topics is not limited to the specific architecture of DNN 106.

[0049] Alternatively or in addition to DNN 106, PMC 102 may include one or more inspection modules, such as, for example, a segmentation module and / or a defect detection module and / or an automated defect review module (ADR) and / or an automated defect classification module (ADC) and / or a metrology-related module and / or other inspection modules that can be used to inspect semiconductor samples. Optionally, one or more inspection modules may include DNN 106. Optionally, DNN 106 may be shared among inspection modules, or alternatively, each of the one or more inspection modules may include its own DNN 106.

[0050] According to some embodiments, system 101 may include storage unit 122. Storage unit 122 may be configured to store any data necessary for operating system 101 (e.g., data regarding inputs and outputs of system 101), as well as intermediate processing results generated by system 101. For example, storage unit 122 may be configured to store images and / or image derivatives generated by inspection tool 120. Thus, one or more images can be retrieved from storage unit 122 and provided to PMC 102 for further processing.

[0051] In some embodiments, system 101 may optionally include a computer-based graphical user interface (GUI) 124 configured to implement user-specified input in relation to system 101. For example, a visual representation of a sample (e.g., via a display forming part of GUI 124) may be provided to the user, the visual representation including image data of the sample. Options for defining certain operational parameters may be provided to the user via the GUI. The user may also view the results of operations on the GUI.

[0052] For reference Figure 2As further detailed, system 101 is configured to receive FP input data via I / O interface 126. FP input data may include data generated by inspection tool 120 (and / or its derivatives and / or associated metadata) and / or data stored in one or more data warehouses. It should be noted that FP input data may include image data (e.g., captured images, images derived from captured images, analog images, synthetic images, etc.) and associated digital data (e.g., metadata, hand-crafted attributes, etc.). It should be further noted that image data may include data associated with the layer of interest and / or data associated with one or more other layers of the sample. In some embodiments of this disclosure, for training purposes, FP input data may include a set of training images for preparing a training set that can be used to train a DNN. It should be noted that image data and associated metadata (e.g., pixel size, textual description of defect types, parameters of the image capture process, etc.) may be received and processed together.

[0053] The FPEI system is further configured to process at least a portion of the received FP input data and send the results (or a portion of the results) via I / O interface 126 to storage unit 122, and / or (a plurality of) inspection tools 120, and / or GUI 124 (for presenting the results) and / or external systems (e.g., FAB's production management system (YMS)).

[0054] Those skilled in the art will readily understand that the teachings of the currently disclosed subject matter are not limited to Figure 1 The system shown; equivalent and / or modified functions may be combined or separated in another manner and may be implemented in any appropriate combination of software and firmware and / or hardware.

[0055] It should be noted that Figure 1 The inspection system shown can be implemented in a distributed computing environment, in which the previously mentioned Figure 1 The illustrated functional modules can be distributed across several local and / or remote devices and can be linked via a communication network. It should be further noted that, in another embodiment, at least some of the inspection tool 120, storage unit 122, and / or GUI 124 can operate externally to the inspection system 100 and communicate data with the system 101 via I / O interface 126. The system 101 can be implemented as a standalone computer(s) used in conjunction with the inspection tool(s). Alternatively, the corresponding functions of the system 101 can be at least partially integrated with one or more inspection tools 120, thereby facilitating and enhancing the functionality of the inspection tool(s) 120 in the inspection-related process.

[0056] For illustrative purposes only, the following description is provided to generate a training set for training a DNN that can be used to inspect semiconductor samples. Those skilled in the art will readily understand that the teachings of the currently disclosed subject matter can be applied to various types of machine learning models (such as, for example, support vector machines (SVMs) and various inspections (such as, for example, segmentation, defect detection, ADR, ADC, automatic navigation modules, automatic segmentation modules, metrology-related modules, etc.).

[0057] See Figure 2 , Figure 2 A generalized flowchart illustrating the generation of a training set according to certain embodiments of the currently disclosed subject matter is shown, the training set being used to train a DNN that can be used to examine semiconductor samples.

[0058] A set of training images (210) can be obtained (e.g., via training set generator 104 through I / O interface 126), each training image representing at least a portion of the samples.

[0059] Depending on the application, the training images can be “real-world” images of the sample obtained during the sample manufacturing process. As a non-limiting example, images can be obtained by inspecting the sample using one or more low-resolution inspection machines (e.g., optical inspection systems, low-resolution SEMs, etc.). Alternatively or additionally, images can be obtained by inspecting the sample using high-resolution machines (e.g., by examining a subset of potential defect locations selected for examination using scanning electron microscopy (SEM) or atomic force microscopy (AFM), etc.).

[0060] In some embodiments, ground truth data associated with the training images may also be obtained. For example, ground truth data may include images and / or labels indicating the application of specific information, such as, for example, defect size, orientation, background segments, etc. Ground truth data may be generated synthetically (e.g., CAD-based images), practically (e.g., captured images), generated through machine learning annotation (e.g., labels based on feature extraction and analysis), generated through manual annotation, or a combination of the above. In one embodiment, a training set, such as that generated based on currently disclosed topics, may be used to train a DNN that can be used for segmented FP images, and the ground truth data associated with the training set is labels indicating segments on the training images. For example, segments may represent structural elements as presented in FP images.

[0061] It is recognized that large training databases are generally required to adequately train deep neural networks (DNNs). However, obtaining sufficient training data is a known challenge when training DNNs that can be used to examine semiconductor samples, as the available training images are very limited and sometimes difficult to obtain. In particular, obtaining a sufficient amount of fully annotated image data as needed is very difficult in supervised learning. Therefore, based on certain embodiments of the currently disclosed subject matter, a method for generating training sets based on a limited amount of image data is proposed.

[0062] Specifically, for each given training image in the group, multiple batches of image patches can be generated. For example, the first batch of training patches can be generated (220) (e.g., via training set generator 104) by: cropping (222) a given training image into a first plurality of original patches according to a plurality of first locations on the training image; and expanding (224) the first plurality of original patches to simulate one or more changes caused by physical processes of the samples, thereby producing a first plurality of expanded patches. The first plurality of expanded patches constitute the first batch of training patches. According to some embodiments, physical processes may refer to the sample manufacturing process and / or inspection process.

[0063] In some embodiments, to generate the first batch of training patches, the plurality of first locations can be multiple predetermined locations defined according to a grid, and a given training image can be cropped into a first plurality of original patches according to the predetermined first locations. For example, anchor points in the grid (e.g., intersections within the grid) can be used as predetermined first locations, and patches can be generated centered on these anchor points, having a predefined size / dimension and stride (i.e., the amount of overlap between adjacent patches). For example, the training image can be 1000×1000 pixels in size, and the training patches can be cropped to sizes such as 100×100 pixels, 150×150 pixels, or 200×200 pixels, wherein there is a stride of half-patch overlap between adjacent patches.

[0064] In some embodiments, multiple first positions can be obtained by shifting multiple predetermined positions, such as the positions of anchor points. For example, the shift can be performed randomly for each predetermined position according to a probability function. For instance, for a first anchor point in the grid, the probability function can determine whether to shift it (e.g., shift it to a random direction with a random offset), while for a second anchor point, the probability function can determine whether to shift it at all. The anchor points to be shifted can be randomly shifted around the intersections in the grid (e.g., each anchor point is shifted to a random direction with a random offset), and the shifted positions are used as the first positions for cropping image patches. As another example, the shift can be performed according to a predefined list defining how each predetermined position is shifted individually. By shifting the positions of the patches, the diversity of the coverage of the generated patches can be increased. The term probability function used herein can refer to a probability distribution function that gives the probability of different possible outcomes of an event. For example, the probability function can be a uniform probability distribution function.

[0065] Then, to simulate possible variations caused by the physical processes of the samples, the original patches can be augmented. This is because the number of training images is finite, so not all common physical phenomena of the manufacturing and / or inspection processes will be encountered during training. By performing augmentation on the patches and including the augmented patches in the training set, the diversity of the training set can be greatly increased, and the robustness of training to these variations can be enhanced.

[0066] Generally, image augmentation can be achieved through various image processing techniques, including: adding noise, blurring, geometric transformations (e.g., rotation, stretching, simulating different angles, cropping, scaling, etc.), tone mapping, and altering the vector information of one or more pixels of the image (e.g., adding and / or modifying the acquired viewpoint or channel, etc.).

[0067] In some embodiments, augmentation techniques can be applied to training images in an application-independent manner. For example, for segmentation-related applications, and / or defect detection-related applications, and / or classification-related applications, the simulated changes may include grayscale changes and / or process changes (PV), and one or more of the following augmentation processes can be applied to simulate the effects of these changes.

[0068] According to some embodiments, grayscale level variations can be caused by physical effects of the sample inspection process. For example, even when images are captured at the same location on the same wafer at the same resolution, the grayscale levels of images taken by different inspection tools can differ by a factor. This can be caused by, for example, different physical configurations and calibrations of the inspection tools / detectors. Additionally, the grayscale levels of images taken from different locations on the wafer can also vary. Therefore, training a DNN based on images captured by tools other than those used to capture FP images and / or based on images captured for different locations on the wafer can degrade the generalization ability of the DNN.

[0069] Therefore, the intensity of gray levels in image patches can be adjusted to simulate this effect. For example, for each image patch, a histogram of pixel values ​​for the image patch can be generated, and the number of gray level biases can be randomly selected within a bias range and applied to the histogram. For instance, the bias range can be determined as a predefined percentage of the range of pixel values ​​(or a portion of the range of pixel values) in the image, and the number of biases to be applied to each image patch can be selected based on a uniform distribution within said range. A training set containing such augmented images will enable increased robustness to tool variations and reduce the need to collect training data from multiple tools.

[0070] As another example, the grayscale intensity of an image patch can also be adjusted by changing its contrast. For instance, the pixel value histogram of an image patch can be stretched to obtain better contrast.

[0071] According to a further embodiment, grayscale level variations can include color variations (CV). Color variations can occur within a single image (e.g., due to layer thickness variations) or between a defect and a reference image. CV can originate from different tool calibrations and can be characterized by localized changes in the GL within the image. To address such effects, patches of the original image can be augmented with additional variations of different desired levels. Therefore, a training set containing such augmented images will enable increased robustness to color variations in FP images.

[0072] According to some embodiments, pattern variations may appear in different images, which may be due to the effects of process variation (PV). Process variation can refer to changes caused by alterations in the sample's manufacturing process. For example, the manufacturing process may cause slight shifts / scaling / distortions of certain structures / patterns between different test images, resulting in pattern variations in the images. As another example, the manufacturing process may cause variations in the sample's thickness, which affects reflectivity and thus, consequently, the grayscale level of the resulting test images. For instance, variations in material thickness from die to die can lead to different reflectivities between two dies, resulting in different background grayscale values ​​for the images of the two dies.

[0073] To address this effect, in some cases, augmentation techniques can be applied to image patches or at least a portion thereof to adjust pattern variations, such as scaling, flipping, etc.

[0074] In some cases, other possible effects of the physical process may include, but are not limited to: noise (e.g., each gray level intensity contains noise intensity at different scales), focusing error (e.g., some patterns are more sensitive to focusing error than others), charging effect (which can introduce artificial differences between the FP and the reference image and can substantially affect the inspection process), and calibration of the four detectors (which may be suboptimal, producing images with overlap in different viewpoints), etc.

[0075] Turn now Figure 6 This illustration shows an example of a pair of original and expanded patches according to certain embodiments of the currently disclosed subject matter. As shown, the original patch 602 undergoes one or more expansion processes as described above to address variations caused by, for example, PV, grayscale changes, and / or CV, and generates a corresponding expanded patch 604. For example, the expanded patch 604 in the current example is synthesized to simulate the effects caused by PV, scaling, and grayscale changes. For example, it can be seen that, compared to the original patch 602, the contours of the polygons, the size of the polygons, and the overall grayscale intensity in the expanded patch 604 are adjusted.

[0076] According to certain embodiments of the currently disclosed subject matter, at least some / parts of a first plurality of original patches can be augmented. For example, for each of the first plurality of original patches, it can be randomly determined, for instance, according to a probability function, whether to augment the original patch. Thus, by using the probability function, at least a portion of the patches will be determined to be augmented. For each image patch of at least a portion of the first plurality of original patches, one or more of the aforementioned disturbance variations / effects can be simulated by using a corresponding image augmentation technique. In some cases, at least a portion of the first plurality of original patches is augmented with a corresponding augmentation intensity determined according to the probability function. As another example, a decision regarding whether to augment each patch, and / or a specific augmentation technique to be applied and / or a specific augmentation intensity to be applied, can be determined according to a corresponding predetermined list. For example, the predetermined decision list may include which original patches to augment, and the augmentation list may include which augmentation techniques and / or specific intensities to be applied to each patch.

[0077] One advantage of certain embodiments of the augmentation process (including those shown above) is that it reduces the impact of interference effects on the trained DNN, and thus improves the robustness and effectiveness of the trained DNN for different applications (such as, for example, segmentation, ADC, ADR, matching, measurement and other inspection tasks).

[0078] As previously mentioned, augmentation techniques can be applied to training images in an application-independent manner. For example, for defect detection applications, augmentation techniques can be applied instead of or in addition to the techniques described above, such as adding noise, blurring, tone mapping, implanting synthetic defects, and modifying defect tones. For classification applications, augmentation techniques can be applied instead of or in addition to the techniques described above, such as geometric transformations, tone mapping, implanting synthetic defects, and modifying defect tones.

[0079] Alternatively or additionally, synthetic data (e.g., defect-related data, simulated connectors or other objects, implants from other images, etc.) can be used to augment images. As a non-limiting example, available images (and / or parameters) with known defect types can be used to implant new defects into an image, amplify the defectiveness of pre-existing defects in an image, remove defects from an image, masquerade defects in an image (making them more difficult to detect), etc.

[0080] Alternatively or additionally, the image can be augmented using segmentation, defect contour extraction, and / or height map calculation, and / or obtained by processing it together with a corresponding CAD-based image.

[0081] After generating the first batch of training patches, as described in reference box 220 above, a second batch of training patches can be generated (230) (e.g., via training set generator 104) for each given training image in the group. This includes shifting (232) multiple first positions to multiple second positions on the given training image to obtain a second plurality of original patches corresponding to the multiple second positions, and repeating (234) the expansion described above on the second plurality of original patches to generate a second plurality of expanded patches, thereby producing a second batch of training patches. Once the first batch of training patches and the second batch of training patches have been generated for each training image in the group, both batches of all training images can be included in the training set to be used to train the DNN.

[0082] According to some embodiments, the above shifting and expansion can be repeated once or multiple times to generate one or more additional batches of training patches. These additional batches may also be included in the training set. In some cases, a first plurality of original patches and a second plurality of original patches may also be included in the training set.

[0083] In some embodiments, shifting can be performed for each first position in a manner similar to that described above in generating the first batch of patches. Whether to shift each first position can be determined randomly based on a probability function. For example, each initial position (or at least some of the initial positions) can be shifted in a random direction with a random offset to obtain a second plurality of positions on the image. The image can be cropped based on the second plurality of positions to obtain a second plurality of original patches. Now turning to... Figure 5 This illustrates examples of a second plurality of original patches obtained after shifting an initial position, according to certain embodiments of the currently disclosed subject matter. As shown, patches can be cut according to different positions 502 marked by dashed squares in the figure. These positions are obtained after random shifting from corresponding initial positions. Assuming the initial positions are determined according to a predefined grid, as can be seen, each current position 502 has a corresponding randomization from the corresponding initial position (e.g., shifted with different offsets in different directions). In some cases, for example, when the offset is determined to be zero, some positions may not be shifted at all.

[0084] In some embodiments, generating a training set may further include obtaining augmented ground truth data about the augmented training patches and associating the augmented training patches with the augmented ground truth data. The generated training set (including multiple batches of augmented training patches associated with the augmented ground truth data) may be stored in the memory of the PMC 102. The generated augmented images may be added to the training set in association with the ground truth data. In some cases, the original image and the augmented image may be associated with the same ground truth data. Alternatively, the augmented image may be associated with augmented ground truth data (e.g., the augmented class distribution in the training set).

[0085] Similarly, for ground truth data associated with the original training images and training patches, augmented ground truth data can be provided by a person analyzing the augmented image patches, with or without the assistance of a computer system. Alternatively or additionally, augmented ground truth data can be generated by the FPEI system by processing the original ground truth data corresponding to the augmentation provided in the corresponding original patches when exporting the augmented training patches.

[0086] According to certain embodiments of the currently disclosed subject matter, the number of training image patches in the generated training set can be substantially greater than the number of "real-world" training samples. For example, a training image set may include dozens of training samples, while the training set generated according to certain embodiments of the currently disclosed subject matter may include at least thousands of training samples. It should be noted that capturing images and their annotations is a slow and potentially expensive process. Efficiency in DNN training is achieved by cropping images and expanding image patches to generate a sufficient number of training samples in the training set.

[0087] Furthermore, augmented and synthetic image generation techniques allow the generated training set to be customized to the requirements of a specific application. As a non-limiting example, the lack of FAB data related to a specific category of defect (e.g., minority category) can be compensated for by presenting synthetic and / or augmented images of the corresponding defect. As a further non-limiting example, the lack of appropriate FAB data related to a specific layer can be compensated for by synthetic images associated with said layer. Similarly, the lack of data for said tool required for tool variation training can be compensated for by synthetic images associated with a specific tool.

[0088] Turn now Figure 3 This illustrates a generalized flowchart of training a DNN using a generated training set, based on certain embodiments of the currently disclosed subject matter.

[0089] In training phase 310, after obtaining (312) as according to Figure 2 After generating the training set as described in the process, the FPEI system 101 can iteratively process the training set using a DNN 106 and provide an application-specific set based on the trained parameters (e.g., weights in the DNN), thereby obtaining an application-specific trained DNN (314). The trained parameters correspond to an application-specific cost function. For example, for segmentation-related applications, the cost function may involve segmentation accuracy (e.g., the correct segmented label map versus the predicted label map). As another example, for defect detection-related applications, the cost function may involve detection accuracy and penalties for false positives and over-detections. As a further example, for classification-related applications, the cost function may involve classification error. Optionally, using a DNN to process the training set may include preprocessing operations performed by the pre-DNN module (e.g., selecting input channels, resizing / clipping, etc.) and post-processing operations performed by the post-DNN module (e.g., performing spatial pyramid pooling, multi-scale pooling, Fisher vectors, etc.). In addition to applying specific optimization weights, training-based parameters can further include applying specific thresholds, applying specific preprocessing parameters, and applying specific postprocessing parameters.

[0090] It should be noted that in some embodiments, the training process shown may be cyclical and may be repeated several times until the DNN is sufficiently trained. For example, the training process may begin with a first batch of training patches. For instance, the first batch of training patches for each training image, generated according to the process described in box 220, may be used to perform the first epoch of training (e.g., entirely through the training set). Users can provide feedback on the results achieved by the DNN based on the initial training set (e.g., the first batch of training patches). For example, the feedback provided may include: changes in sensitivity, updates to ground truth segments and / or manually defining regions of interest (ROIs) for segments, manually reclassifying one or more pixels, regions and / or defects for classification applications, redefining masks / bounding boxes for defect detection applications, etc.

[0091] PMC can adjust the next training cycle based on the received feedback. Adjustments may include at least one of the following: updating the training set (e.g., updating ground truth data and / or augmenting the algorithm, obtaining additional augmented training samples, etc.), updating the cost function, updating the pre-DNN and / or post-DNN algorithm, etc. The next training cycle (i.e., the second epoch) can begin using a second batch of training patches corresponding to each training image, generated as described in box 230.

[0092] Once the training phase is complete, the trained DNN can be used for a specific application in runtime 320. The PMC102 can use the trained DNN to process (322) one or more runtime FP images together, thereby obtaining (324) application-specific inspection-related data. One or more FP images can be obtained using the same or different inspection modalities. It should be noted that the training data should correspond to the FP images to be used for the respective application.

[0093] As a non-limiting example, application-specific inspection-related data can represent per-pixel plots whose meaning depends on the applied values ​​(e.g., binary plots for defect detection; label plots for segmentation; discrete plots for indicating family type or general category of hindrance family prediction; discrete plots for defect type classification; continuous values ​​for cross-modal or die-to-model (D2M) regression, etc.). The per-pixel plot can be further correlated with per-pixel probabilities indicating the probability of the value obtained for a pixel. Figure 1 Together we obtained it.

[0094] Alternatively or additionally, application-specific inspection-related data may represent one or more values ​​summarizing the entire image content over a larger area (not per pixel), such as, for example, defect attributes, segmentation labels for certain structural elements and / or layers, defect bounding box candidates and associated defect probabilities for automated defect review applications, defect categories and classification probabilities for automated defect classification applications, etc. Alternatively or additionally, inspection-related data may be used for metrological purposes. For example, segmentation maps may be used to perform measurements on sample images.

[0095] refer to Figure 4 The above reference illustrates implementations of certain embodiments of the subject matter currently disclosed. Figure 3 Examples of techniques for segmenting FP images are detailed below. Unless otherwise specifically stated, the term "segmentation" as used herein should be broadly interpreted to encompass any process that divides an image into meaningful parts / segments (e.g., background and foreground, noisy and noise-free regions, structuring elements, defects and no defects, etc.) while providing per-pixel or per-region values ​​indicative of these segments. For example, such segmentation can be used by metrology tools to perform measurements on samples. As another example, the segmentation can also be used in constructing attributes (e.g., for defining whether a defect is on the main pattern, on the background, or both), in an ADR to apply segment-specific detection thresholds to each segment, etc.

[0096] The process includes a setup phase 410 for segmentation-specific training of the DNN and a runtime phase 420 for image segmentation using the trained DNN.

[0097] During setup 410, after obtaining a set of training images and ground-based data, the PMC 102 can use the reference above. Figure 2 The described technique is used to generate (412) a segmented training set, and the generated segmented training set is used to obtain (414) a trained segmented DNN characterized by segment-related training parameters.

[0098] The training image set may include previously captured low-resolution and / or high-resolution images, and optionally CAD-based images. For example, training images may be high-resolution review images obtained from a review tool (e.g., SEM). The obtained ground-based data provides information on segment-related data associated with the corresponding training images. As a non-limiting example, the segment-related data associated with a given training image may provide segment labels per pixel or per region, CAD polygons, ROIs, etc. Once the training set is generated, expanded training patches in the training set can be associated with expanded ground-based data, which provides segment-related information about the corresponding expanded training patches.

[0099] After generating the (412) segmented training set, PMC uses the training set to train the (414) DNN. The training process produces a trained DNN with segment-dependent training parameters.

[0100] During runtime 420, the PMC uses a trained segmentation DNN to process (422) one or more runtime images containing one or more captured FP images to be segmented, in order to provide (424) a segmentation map for each image. The resulting segmentation map can provide information indicating per-pixel or per-region segmentation labels for different segments on the image. For example, a polygon on one layer may have one segmentation label, while a polygon on another layer may have different segmentation labels, and the background may have a separate segmentation label.

[0101] Turn now Figure 7 Examples of FP images and corresponding segmented maps generated according to certain embodiments of the currently disclosed subject matter are shown. As shown, the FP image 702 is processed by a trained segmented DNN 701, and the output of the DNN is a segmented map 704, in which different colors represent different segments. For example, three segments are shown: horizontal element 706, vertical element 708, and background 710.

[0102] According to certain embodiments of the currently disclosed subject matter, segmentation DNNs are designed with specific architectures to improve segmentation performance. For example, the DNN can apply the U-Net network architecture, a convolutional network used for fast and accurate image segmentation. The U-Net architecture can be modified and extended to operate with fewer training images and produce more accurate segmentations.

[0103] In some embodiments, to overcome the vanishing gradient problem while preserving high-resolution information during training, piecewise DNNs can apply a U-Net-based architecture to address this issue. The vanishing gradient problem is a known challenge found in training neural networks using gradient-based learning methods and backpropagation. In such methods, in each iteration of training, the weights of each neural network receive updates proportional to the partial derivative of the error function with respect to the current weights. The problem is that in some cases, the gradient will become small (i.e., vanish), effectively preventing the weights from changing their values. In the worst case, this can completely prevent further training of the neural network. U-Net-based networks are designed to have better gradient flow and therefore can effectively solve this problem.

[0104] In some embodiments, during the training of the segmented DNN, in addition to the main loss function (e.g., cross-entropy loss) computed as the difference between the actual output generated by the DNN and the ground truth data associated with the corresponding training data, the segmented DNN may include one or more additional loss functions specifically designed to improve the sensitivity of the segments. The one or more additional loss functions may indicate physical knowledge of the samples and their images. According to some embodiments, one or more additional loss functions may be added to improve the smoothness and consistency of the segments.

[0105] It will be understood that this disclosure is not limited to the details set forth in the description contained herein or shown in the accompanying drawings.

[0106] It will also be understood that the system according to this disclosure can be implemented, at least in part, on a suitably programmed computer. Similarly, this disclosure contemplates methods for using a computer-readable computer program to execute this disclosure. This disclosure further contemplates a non-transitory computer-readable storage container that tangibly embodies instructions of a program, which are executable by a computer to perform methods of executing this disclosure.

[0107] This disclosure can have other embodiments and can be practiced and performed in various ways. Therefore, it will be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. Consequently, those skilled in the art will understand that the concepts on which this disclosure is based can be readily used as a basis for designing other structures, methods, and systems to achieve several of the objectives of the subject matter currently disclosed.

[0108] Those skilled in the art will readily understand that various modifications and alterations can be applied to embodiments of the disclosure described above without departing from the scope of the disclosure as defined in and by the appended claims.

Claims

1. A computerized method for runtime inspection of samples, the method comprising: Obtain one or more runtime images of the sample; as well as The one or more runtime images are processed using a deep neural network (DNN) to obtain application-specific inspection-related data as the output of the DNN. The DNN is pre-trained for a specific application using a training set, which includes at least a first batch of training patches and a second batch of training patches corresponding to a given training image. The first batch of training patches is generated by: cropping the given training image into a first plurality of original patches according to a plurality of first positions on the given training image, and expanding at least a portion of the first plurality of original patches to simulate changes caused by physical processes of the sample. The second batch of training patches is generated by the following operations: shifting the plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches.

2. The computerized method of claim 1, wherein the meaning of the application-specific inspection-related data depends on the value per pixel of the application-specific graph.

3. The computerized method of claim 2, wherein the per-pixel map is one of the following: a binary map for defect detection applications, a label map for segmentation applications, a discrete map for predicting interference families indicating family types or general categories, a discrete map for defect type classification applications, and a map of continuous values ​​for cross-modal or die-to-model D2M regression applications.

4. The computerized method of claim 1, wherein the application-specific inspection-related data represents one or more values ​​summarizing the entire image content of a large region in a runtime image.

5. The computerized method of claim 4, wherein the one or more values ​​represent one of the following: defect attributes, segmentation labels for structural elements and / or layers, defect bounding box candidates and associated defect probabilities for an automated defect review (ADR) application, and defect categories and category probabilities for an automated defect classification (ADC) application.

6. The computerized method of claim 1, wherein the specific application is a segmentation application for dividing an image into meaningful segments, the meaningful segments representing: background and foreground, noisy regions and noise-free regions, structural elements, or defects and no defects.

7. The computerized method of claim 6, wherein the given training image is a high-resolution review image with ground-based data obtained from a review tool, the ground-based data providing information on segment-related data associated with the given training image, and, when generating the training set, the first batch of training patches and the second batch of training patches are associated with augmented ground-based data, the augmented ground-based data providing information on segment-related information about the corresponding training patches.

8. The computerized method of claim 7, wherein the segment-related data associated with the given training image provides segment labels per pixel or per region, computer-aided design CAD polygons, or region of interest (ROI) information.

9. The computerized method of claim 6, wherein the application-specific inspection-related data is a segmented map that provides information indicating per-pixel or per-region segment labels for different segments on a runtime image.

10. The computerized method of claim 6, wherein the DNN may include a main loss function representing the difference between the actual output generated by the DNN and ground-based data associated with the corresponding training data, and one or more additional loss functions specifically designed to improve the sensitivity of the segment.

11. A computerized system for runtime inspection of samples, the system comprising a processor and memory circuitry configured to: Obtain one or more runtime images of the sample; and The one or more runtime images are processed using a deep neural network (DNN) to obtain application-specific inspection-related data as the output of the DNN. in, The DNN is pre-trained for a specific application using a training set, which includes at least a first batch of training patches and a second batch of training patches corresponding to a given training image. The first batch of training patches is generated by: cropping the given training image into a first plurality of original patches according to a plurality of first positions on the given training image, and expanding at least a portion of the first plurality of original patches to simulate changes caused by physical processes of the sample. The second batch of training patches is generated by the following operations: shifting the plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches.

12. The computerized system of claim 11, wherein the application-specific inspection-related data represents a per-pixel map, the per-pixel map being one of the following: a binary map for defect detection applications, a label map for segmentation applications, a discrete map for predicting hindrance families indicating family types or general categories, a discrete map for defect type classification applications, and a map of continuous values ​​for cross-modal or die-to-model D2M regression applications.

13. The computerized system of claim 11, wherein the application-specific inspection-related data represents one or more values ​​summarizing the entire image content of a large region in a runtime image.

14. The computerized system of claim 13, wherein the one or more values ​​represent one of the following: defect attributes, segmentation labels for structural elements and / or layers, defect bounding box candidates and associated defect probabilities for an automated defect review (ADR) application, and defect categories and category probabilities for an automated defect classification (ADC) application.

15. The computerized system of claim 11, wherein the particular application is a segmentation application for dividing an image into meaningful segments, the meaningful segments representing: background and foreground, noisy regions and noise-free regions, structural elements, or defects and no defects.

16. The computerized system of claim 15, wherein the given training image is a high-resolution review image with ground-based data obtained from a review tool, the ground-based data providing information on segment-related data associated with the given training image, and, when generating the training set, the first batch of training patches and the second batch of training patches are associated with augmented ground-based data, the augmented ground-based data providing information on segment-related information about the respective training patches.

17. The computerized system of claim 16, wherein the segment-related data associated with the given training image provides segment labels per pixel or per region, computer-aided design CAD polygons, or region of interest (ROI) information.

18. The computerized system of claim 15, wherein the DNN is designed with a U-Net network architecture to improve the performance of the segmentation.

19. The computerized system of claim 15, wherein the DNN may include a main loss function representing the difference between the actual output generated by the DNN and ground-based data associated with the corresponding training data, and one or more additional loss functions specifically designed to improve the sensitivity of the segmentation.

20. A non-transitory computer-readable storage medium tangibly embodying instructions of a program, the instructions causing the computer to perform a runtime check of a sample when executed by the computer, the method comprising: Obtain one or more runtime images of the sample; as well as The one or more runtime images are processed using a deep neural network (DNN) to obtain application-specific inspection-related data as the output of the DNN. The DNN is pre-trained for a specific application using a training set, which includes at least a first batch of training patches and a second batch of training patches corresponding to a given training image. The first batch of training patches is generated by: cropping the given training image into a first plurality of original patches according to a plurality of first positions on the given training image, and expanding at least a portion of the first plurality of original patches to simulate changes caused by physical processes of the sample. The second batch of training patches is generated by the following operations: shifting the plurality of first positions to a plurality of second positions on the given training image to obtain a second plurality of original patches corresponding to the plurality of second positions, and repeatedly expanding the second plurality of original patches to generate a second plurality of expanded patches.

Images