Generate training data that can be used to inspect semiconductor samples

Abstract

A system and method for generating training data for training a deep neural network that can be used for semiconductor sample inspection is provided. The method includes: obtaining a first training image and a first label associated with a group of pixels selected in each segment, extracting a feature set that characterizes the first training image, training a machine learning (ML) model using the first label, the value of the group of pixels, and the feature value of each feature of the feature set corresponding to the group of pixels, processing the first training image using the trained ML model to obtain a first segmentation map, and determining to include the first training image and the first segmentation map into DNN training data when a criterion is met, and repeating the extraction of a second feature, training, and processing when the criterion is not met.

Description

[0001] This application is a divisional application of a Chinese patent application with an application date of May 6, 2021, application number 202110491730.6, and name “Generating training data that can be used to inspect semiconductor samples”. Technical Field

[0002] The subject matter disclosed herein relates generally to the field of inspection of semiconductor samples, and more particularly to the generation of training data that can be used to inspect the samples. Background Art

[0003] The current demand for high density and performance associated with ultra-large scale integration of manufactured devices requires sub-micron features, increased transistor and circuit speeds, and improved reliability. As semiconductor processes advance, pattern dimensions such as line widths and other types of critical dimensions continue to shrink. Such demands require the formation of device features with high precision and uniformity, which in turn requires careful monitoring of the manufacturing process, including automated inspection of the device while it is still in the form of a semiconductor wafer.

[0004] As a non-limiting example, a run-time check may employ a two-stage procedure, e.g., inspecting a sample followed by reviewing the sampled locations for potential defects. Inspection generally involves generating some output (e.g., an image, a signal, etc.) for a sample by directing light or electrons to a wafer and detecting light or electrons from the wafer. During the first stage, the surface of the sample is inspected at high speed and relatively low resolution. Defect detection is typically performed by applying a missing part detection algorithm to the inspection output. A defect map is generated to illustrate locations on the sample suspected of having a high probability of defects. Most often, the purpose of the inspection is to provide high sensitivity for detecting defects of interest while suppressing detection of nuisance and noise on the wafer. During the second stage, at least some of the suspected locations are more thoroughly analyzed at a relatively high resolution. In some cases, both stages may be implemented by the same inspection tool, and in some other cases, the two stages are implemented by different inspection tools.

[0005] The inspection process may include multiple inspection steps. The inspection step may be performed multiple times during the manufacturing process, for example, after manufacturing or processing certain layers, etc. Additionally or alternatively, each inspection step may be repeated multiple times, for example for different wafer positions or for the same wafer position with different inspection settings.

[0006] Inspection processes are used at various steps during semiconductor manufacturing to detect and classify defects on samples, as well as to perform metrology-related operations. The effectiveness of inspection can be improved by automating the process(es), such as defect detection, automatic defect classification (ADC), automatic defect review (ADR), image segmentation, automatic metrology-related operations, etc. Summary of the invention

[0007] According to certain aspects of the subject matter disclosed herein, there is provided a computerized system for generating training data (DNN training data) for training a deep neural network that can be used to inspect a semiconductor sample, the system comprising a processor and memory circuit (PMC) configured to: obtain a first training image representing at least a portion of the semiconductor sample, and a first label associated with a set of pixels selected in each of one or more segments identified by a user from the first training image; extract a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features including a first feature providing information about a background relationship between the one or more segments in the first training image, and a first label provided in the first training image; The invention relates to a method for extracting a second feature of the image segmentation method, wherein the first label is used to extract a second feature of the image segmentation method, wherein the second feature is information about the distribution of pixels in the image segment relative to a statistical measurement value of a set of pixels in each segment; training a machine learning (ML) model using the first label, the value of a set of pixels selected in each segment associated with the first label, and the feature value of each feature of the feature set corresponding to the set of pixels in each segment, wherein the trained ML model is used for image segmentation; processing the first training image using the trained ML model to obtain a first segmentation map providing information of predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs; and determining to include a first training sample including the first training image and the first segmentation map into the DNN training data when a criterion is met, and repeating the extraction of the second feature, the training, and the processing when the criterion is not met.

[0008] In addition to the above features, the system according to this aspect of the herein disclosed subject matter may comprise one or more of the features (i) to (xi) listed below in any desired combination or arrangement that is technically possible:

[0009] (i). The training data may be used to train a deep neural network (DNN) for at least one inspection process selected from the group consisting of: automatic segmentation, automatic metrology, automatic defect detection, automatic defect review, and automatic defect classification based on runtime images.

[0010] (ii). The criterion is based on user feedback about the first segmentation map, and the PMC is configured to obtain an additional first label associated with an additional group of pixels in at least one segment when negative user feedback about the first segmentation map is received, the first label and the additional first label constitute aggregated label data, and repeatedly extracting the second feature, training and processing based on the aggregated label data until positive user feedback is received.

[0011] (iii). The PMC is configured to include the first training sample into the training data upon receiving positive user feedback on the first segmentation map.

[0012] (iv).PMC is also configured to obtain a second training image and a second label associated with a set of pixels respectively selected by the user from each of one or more segments identified by the user from the second training image, add the second label to the aggregated label data, extract a feature set that characterizes the second training image and includes a first feature and a second feature, train an ML model using the aggregated label data, pixel values ​​associated with the aggregated label data, and feature values ​​of each feature of the feature set corresponding to the pixels associated with the aggregated label data, and perform processing and determination based on the second training image.

[0013] (v). At least one of the one or more segments corresponds to one or more structural elements present in the first training image.

[0014] (vi). The first feature includes a texture feature extracted by the following operations: processing the first training image with a filter set to generate a set of feature responses, each feature response including a feature response value corresponding to a corresponding pixel of the first training image, wherein each pixel corresponds to a feature vector including its feature response value set; clustering the feature vectors corresponding to the pixels in the first training image into a plurality of clusters; assigning a feature value to each pixel according to the cluster to which its feature vector belongs, thereby generating a feature map including feature values ​​corresponding to the pixels of the first training image, the feature map being separable into a plurality of channels corresponding to the corresponding feature values; and sampling each channel of the feature map using a sampling filter to generate a feature map having a plurality of sampled channels.

[0015] (vii). The first feature further includes an intensity feature extracted by clustering the pixel values ​​of the first training image into a plurality of clusters, assigning a feature value to each pixel according to the cluster to which the pixel value belongs, thereby generating a feature map separable into a plurality of channels corresponding to the corresponding feature values, and sampling each channel of the feature map using a sampling filter, thereby generating a feature map having a plurality of sampled channels.

[0016] (viii). The filter set includes one or more of the following: derivative filter, Gabor filter and Laplace filter.

[0017] (ix). The second feature is extracted by calculating a statistical measurement value for a set of pixels in each segment, and for each pixel in the first training image, calculating a corresponding feature value as the distance between the pixel value and the statistical measurement value of each segment, thereby generating one or more feature maps each containing a feature value corresponding to the distance between the pixel value and the statistical measurement value of the corresponding segment.

[0018] (x). The training image contains multiple channels captured from different perspectives, and a feature set is extracted from the multiple channels.

[0019] (xi). The PMC is also configured to train the DNN using the DNN training data, and to validate the trained DNN using the validation set of images.

[0020] According to other aspects of the subject matter disclosed herein, there is provided a method for generating training data (DNN training data) for training a deep neural network that can be used to inspect a sample, the method being executed by a processor and memory circuit (PMC) and comprising: obtaining a first training image representing at least a portion of the semiconductor sample, and first labels respectively associated with a set of pixels selected in each of one or more segments identified by a user from the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the feature set including a first feature providing information about a background relationship between the one or more segments in the first training image, and a first feature providing information about a background relationship in the first training image relative to each segment. A method for extracting a second feature of a DNN comprises: extracting a second feature of information about a pixel distribution of a statistical measurement value of a group of pixels; training a machine learning (ML) model using a first label, a value of a group of pixels selected in each segment associated with the first label, and a feature value of each feature of a feature set corresponding to a group of pixels in each segment, wherein the trained ML model is used for image segmentation; processing a first training image using the trained ML model to obtain a first segmentation map providing information about predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs; and determining to include a first training sample including the first training image and the first segmentation map into DNN training data when a criterion is met, and repeating the extraction of the second feature, the training, and the processing when the criterion is not met.

[0021] This aspect of the disclosed subject matter may comprise one or more of the features (i) to (xi) listed above with respect to the system mutatis mutandis, in any desired combination or arrangement technically possible.

[0022] According to other aspects of the subject matter disclosed herein, there is provided a non-transitory computer-readable medium containing instructions, which, when executed by a computer, cause the computer to perform a method for generating training data (DNN training data) for training a deep neural network that can be used to inspect a sample, the method comprising: obtaining a first training image representing at least a portion of the semiconductor sample, and a first label associated with a group of pixels selected in each of one or more segments identified by a user from the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the feature set including a first feature providing information about a background relationship between the one or more segments in the first training image, and a first label providing information about a background relationship between the one or more segments in the first training image. A second feature of information about pixel distribution of statistical measurement values ​​for a set of pixels in each segment; training a machine learning (ML) model using a first label, values ​​of a set of pixels selected in each segment associated with the first label, and feature values ​​of each feature of a feature set corresponding to a set of pixels in each segment, wherein the trained ML model is used for image segmentation; processing a first training image using the trained ML model to obtain a first segmentation map providing information about predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs; and determining to include a first training sample including the first training image and the first segmentation map into DNN training data when a criterion is met, and repeating the extraction of the second feature, the training, and the processing when the criterion is not met.

[0023] This aspect of the disclosed subject matter may comprise one or more of the features (i) to (xi) listed above with respect to the method mutatis mutandis, in any desired combination or permutation technically possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to understand the present disclosure and to see how it may be carried out in practice, embodiments will now be described, purely as non-limiting examples, with reference to the accompanying drawings, in which:

[0025] Figure 1 A generalized block diagram of an inspection system is shown in accordance with certain embodiments of the subject matter disclosed herein.

[0026] Figure 2A A generalized flow chart for generating training data for training a DNN that can be used to inspect semiconductor samples is shown according to certain embodiments of the subject matter disclosed herein.

[0027] Figure 2B A generalized flow chart illustrating a continuous process of DNN training data generation upon receiving negative user feedback regarding a first segmentation map according to certain embodiments of the subject matter disclosed herein.

[0028] Figure 3A A generalized flow chart illustrating extraction of texture features from a first training image according to some embodiments of the subject matter disclosed herein.

[0029] Figure 3B A generalized flow chart illustrating extraction of statistical features from a first training image according to some embodiments of the subject matter disclosed herein.

[0030] Figure 4 A generalized flow chart for training a DNN using generated DNN training data according to certain embodiments of the subject matter disclosed herein is shown.

[0031] Figure 5 Examples of training images according to some embodiments of the subject matter disclosed herein are shown.

[0032] Figure 6 An example of partial user annotations on a training image is shown in accordance with some embodiments of the subject matter disclosed herein.

[0033] Figure 7 Examples of texture features and examples of intensity features are shown according to some embodiments of the subject matter disclosed herein.

[0034] Figure 8 An example of a segmentation map with negative user feedback is shown in accordance with some embodiments of the subject matter disclosed herein.

[0035] Fig. 9A An example of additional first tags provided by a user is shown according to some embodiments of the subject matter disclosed herein.

[0036] Fig. 9B An example of an updated segmentation map with positive user feedback is shown in accordance with some embodiments of the subject matter disclosed herein. DETAILED DESCRIPTION

[0037] In the following detailed description, many specific details are set forth to provide a thorough understanding of the present disclosure. However, those skilled in the art will appreciate that the subject matter disclosed herein can be practiced without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail to avoid confusing the subject matter disclosed herein.

[0038] Unless specifically stated otherwise, as will be apparent from the following discussion, it should be understood that discussions utilizing terms such as "generate," "train," "obtain," "extract," "process," "determine," "repeat," "include," "receive," "cluster," "distribute," "sample," "compute," "use," "validate," and the like throughout this specification refer to the actions and / or processes of a computer that manipulates and / or transforms data into other data, the data being represented physically, such as electronically, quantitatively, and / or the data representing physical objects. The term "computer" should be broadly interpreted to encompass any kind of hardware-based electronic device with data processing capabilities, including, as non-limiting examples, the inspection systems, training data generation systems, and corresponding portions thereof disclosed in this application.

[0039] The term "inspection" used in this specification should be broadly interpreted to cover any kind of metrology-related operations, as well as operations related to detecting and / or classifying defects in a sample during sample manufacturing. Inspection is provided by using a non-destructive inspection tool during or after manufacturing the sample to be inspected. As a non-limiting example, the inspection process may include using the same or different inspection tools to run time scans (single or multiple scans), sampling, review, measurement, classification and / or other operations provided with respect to the sample or part thereof. Similarly, inspection may be provided before manufacturing the sample to be inspected, and the inspection may include, for example, generating an inspection plan and / or other setup operations. Note that, unless specifically stated otherwise, the term "inspection" or its derivatives used in this specification are not limited to the resolution or size of the inspection area. As non-limiting examples, various non-destructive inspection tools include scanning electron microscopes, atomic force microscopes, optical inspection tools, and the like.

[0040] As used herein, the terms "non-transitory memory" and "non-transitory storage medium" should be broadly interpreted to cover any volatile or non-volatile computer memory suitable for use with the subject matter disclosed herein.

[0041] The term "sample" used in this specification should be broadly interpreted to cover any kind of wafers, masks, and other structures, combinations and / or portions thereof used to manufacture semiconductor integrated circuits, magnetic heads, flat panel displays, and other semiconductor manufactured articles.

[0042] The term "defect" as used in this specification should be interpreted broadly to cover any kind of abnormal or undesirable feature formed on or in a sample.

[0043] The term "design data" as used in this specification should be interpreted broadly to cover any data indicating the hierarchical physical design (layout) of a sample. The design data may be provided by a corresponding designer and / or may be derived from the physical design (e.g., through complex simulations, simple geometry and Boolean operations, etc.). As a non-limiting example, the design data may be provided in different formats such as GDSII format, OASIS format, etc. The design data may be presented in vector format, grayscale intensity image format, or other formats.

[0044] It will be appreciated that, unless specifically stated otherwise, certain features of the subject matter disclosed herein described in the context of different embodiments may also be provided in combination in a single embodiment. Conversely, various features of the subject matter disclosed herein described in the context of a single embodiment may also be provided separately or in any suitable sub-combination. In the following detailed description, many specific details are set forth to provide a thorough understanding of the method and apparatus.

[0045] Keeping this in mind, pay attention Figure 1 , in which a functional block diagram of an inspection system according to certain embodiments of the subject matter disclosed herein is shown.

[0046] Figure 1 The inspection system 100 shown in the figure can be used to inspect semiconductor samples (e.g., wafers and / or portions thereof) as part of a sample manufacturing process. The inspection system 100 shown includes a computer-based system 101 that is capable of automatically determining information about metrology and / or about defects using images obtained during sample manufacturing (hereinafter referred to as manufacturing process (FP) images). According to certain embodiments of the subject matter disclosed herein, the system 101 can be configured to generate training data for training a deep neural network (DNN) that can be used to inspect semiconductor samples. The system 101 is therefore also referred to as a training system or training data generation system in the present disclosure. The system 101 can be operably connected to one or more inspection tools 120. The inspection tool 120 is configured to capture FP images and / or review captured FP images and / or enable or provide measurements on captured images.

[0047] As an example, the FP image may be selected from an image of a sample (e.g., a wafer or portion thereof) captured during a manufacturing process, derivatives of the captured image obtained through various pre-processing stages (e.g., an image of a portion of a wafer or photomask captured by a scanning electron microscope (SEM) or an optical inspection system, an SEM image roughly centered on a defect to be classified by an ADC, an SEM image of a larger area in which a defect will be localized by an ADR, registered images of different inspection modalities corresponding to the same mask position, segmented images, height mapping images, etc.), and a computer-generated image based on design data. Note that in some cases, an image may include image data (e.g., a captured image, a processed image, etc.) and associated digital data (e.g., metadata, manual attributes, etc.). Note also that the image data may include data about the layer of interest and / or one or more other layers of the sample.

[0048] The term "inspection tool" as used herein should be broadly interpreted to cover any tool that can be used in the process of inspection, including as non-limiting examples imaging, scanning (single or multiple scans), sampling, review, measurement, classification and / or other processes provided with respect to a sample or a portion thereof. One or more inspection tools 120 may include one or more inspection tools and / or one or more review tools. In some cases, at least one inspection tool 120 may be an inspection tool configured to scan a sample (e.g., an entire wafer, an entire die or a portion thereof) to capture an inspection image (typically, at a relatively high speed and / or low resolution) for detecting potential defects. In some cases, at least one inspection tool 120 may be a review tool configured to capture a review image of at least some defects detected by the inspection tool for confirming whether the potential defect is actually a defect. Such a review tool is typically configured to inspect a fragment of a die one at a time (typically, at a relatively low speed and / or high resolution). The inspection tool and the review tool may be different tools located at the same or different locations, or a single tool operating in two different modes. In some cases, at least one inspection tool may have metrology capabilities and may be configured to perform metrology measurements on the FP images.

[0049] Without limiting the scope of the present disclosure in any way, it should also be noted that inspection tool 120 may be implemented as various types of inspection machines, such as optical imaging machines, electron beam inspection machines, etc. In some cases, the same inspection tool may provide both low resolution image data and high resolution image data.

[0050] System 101 includes a processor and memory circuit (PMC) 102 operatively connected to a hardware-based I / O interface 126. PMC 102 is configured as described in reference Figure 2A and Figure 2B The processing necessary to operate the system is further described in detail and includes a processor (not shown separately) and a memory (not shown separately). The processor of PMC 102 can be configured to execute several functional modules according to computer-readable instructions implemented on a non-transitory computer-readable memory contained in the PMC. Such functional modules are hereinafter referred to as being contained in the PMC.

[0051] According to some embodiments, the functional modules included in the PMC 102 may include a feature extractor 104, a training module 106, and a machine learning model 108. The PMC 102 may be configured to obtain a first training image representing at least a portion of a semiconductor sample, and a first label associated with a group of pixels selected in each of one or more segments identified by a user from the first training image, respectively, via an I / O interface 126. The feature extractor 104 may be configured to extract a feature set characterizing the first training image. Each feature in the set has a feature value corresponding to a pixel in the first training image. The feature set includes a first feature that provides information about a background relationship between one or more segments in the first training image, and a second feature that provides information about the distribution of pixels in the first training image relative to a statistical measurement value of a group of pixels in each segment. The training module 106 may be configured to train the machine learning model 108 using the first label, the value of a group of pixels selected in each segment associated with the first label, and the feature value of each feature of the feature set corresponding to a group of pixels selected in each segment. The trained machine learning model 108 can be used to process the first training image to obtain a first segmentation map, wherein the first segmentation map provides information of predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs. The training module 106 can also be configured to determine to include a first training sample including the first training image and the first segmentation map into the training data (i.e., training data for training a deep neural network (DNN), also referred to herein as DNN training data) when the criteria are met, and to repeat the extraction of the second feature, the training, and the processing when the criteria are not met. Figure 2A and Figure 2B Describe the details of the training data generation process.

[0052] In some embodiments, the DNN training data generated by the training system 101 can be used to train a DNN 110 that can be used to inspect semiconductor samples. Figure 1As shown, the DNN may be included in the PMC 102 of the system 101, or alternatively operably connected to the system 101. The DNN 110 may be configured to enable data processing using a deep neural network for outputting application-related data based on manufacturing input data. In one embodiment, the specific application may be image segmentation, and the output of the application-related may be a segmentation map corresponding to the input FP image. The operation of the system 101, the PMC 102, and the functional modules therein will be referred to in detail. Figure 2A and Figure 2B Further details.

[0053] DNN 110 may include a supervised or unsupervised DNN model comprising layers organized according to a corresponding DNN architecture. As a non-limiting example, the layers of the DNN may be organized according to a convolutional neural network (CNN) architecture, a recurrent neural network architecture, a recursive neural network architecture, a generative adversarial network (GAN) architecture, or others. Optionally, at least some of the layers may be organized into multiple DNN subnetworks. Each layer of the DNN may include multiple basic computing elements (CEs), commonly referred to in the art as dimensions, neurons, or nodes.

[0054] Typically, the computing elements of a given layer can be connected to the CE of the previous layer and / or the next layer. Each connection between the CE of the previous layer and the CE of the next layer is associated with a weighted value. A given CE can receive input from the CE of the previous layer via a corresponding connection, and each given connection is associated with a weighted value of the input that can be applied to the given connection. The weighted value can determine the relative strength of the connection and thus determine the relative influence of the corresponding input on the output of the given CE. A given CE can be configured to calculate an activation value (e.g., a weighted sum of inputs) and further derive an output by applying an activation function to the calculated activation. For example, the activation function can be an identity function, a deterministic function (e.g., linear, S-shaped, threshold, etc.), a random function, or other suitable function. The output from a given CE can be sent to the CE of the next layer via a corresponding connection. Similarly, as described above, each connection at the output of a CE can be associated with a weighted value, which can be applied to the output of the CE before being received as an input to the CE of the next layer. Further, for the weighted value, there can be a threshold (including a limit function) associated with the connection and the CE.

[0055] The weights and / or thresholds of the deep neural network can be initially selected before training, and can be further iteratively adjusted or modified during training to achieve an optimal set of weights and / or thresholds in the trained DNN. After each iteration, the difference between the actual output generated by the DNN module and the target output associated with the corresponding data training set can be determined. The difference can be referred to as an error value. When the loss / cost function indicating the error value is less than a predetermined value, or when a limited change in performance between iterations is achieved, the training can be determined to be complete.

[0056] The set of DNN input data used to adjust the weights / thresholds of the deep neural network is referred to as DNN training data hereinafter. As mentioned previously, the system 101 is configured to generate such a DNN training data that can be used to train the DNN.

[0057] Note that the teachings of the subject matter disclosed herein are not limited to the specific architecture of the DNN as described above.

[0058] In some embodiments, the DNN may be a segmentation DNN configured to perform image segmentation on the FP image. In some cases, in addition to the DNN 110, the inspection system 100 may also include one or more inspection modules, such as, for example, a defect detection module and / or an automatic defect review module (ADR) and / or an automatic defect classification module (ADC) and / or a module related to metrology and / or other inspection modules that can be used to inspect semiconductor samples. One or more inspection modules may be implemented as independent computers, or their functions (or at least part thereof) may be integrated with the inspection tool 120. In some cases, the DNN 110 may be included in one or more inspection modules. Optionally, the DNN 110 may be shared between inspection modules, or alternatively, each of the one or more inspection modules may contain its own DNN 110.

[0059] According to some embodiments, the system 101 may include a storage unit 122. The storage unit 122 may be configured to store any data necessary for operating the system 101, for example, data regarding inputs and outputs of the system 101, and intermediate processing results generated by the system 101. As an example, the storage unit 122 may be configured to store images and / or derivatives thereof generated by the inspection tool 120. Thus, one or more images may be retrieved from the storage unit 122 and provided to the PMC 102 for further processing.

[0060] In some embodiments, the system 101 may optionally include a computer-based graphical user interface (GUI) 124 configured to enable user-specified input regarding the system 101. For example, a user may be presented with a visual representation of a sample (e.g., via a display forming part of the GUI 124), including image data of the sample. The user may be provided with the option of defining certain operating parameters via the GUI. For example, the user may provide label data by manually annotating an image. The user may also observe the results of the operation on the GUI, such as, for example, a segmentation map.

[0061] If you will refer to Figure 2A and Figure 2B In further detail, the system 101 is configured to receive FP input data via the I / O interface 126. The FP input data may include data generated by the inspection tool 120 (and / or derivatives of the data and / or metadata associated with the data) and / or data stored in one or more data repositories. Note that in some cases, the FP input data may include image data (e.g., captured images, images derived from captured images, simulated images, synthetic images, etc.) and associated digital data (e.g., metadata, manual attributes, etc.). It is also noted that the image data may include data about one or more other layers of the layer of interest and / or sample. In some embodiments of the present disclosure, for training purposes, the FP input data may include one or more training images to generate training data that can be used to train the DNN.

[0062] The system 101 is also configured to process the received FP input data and send the results (or parts thereof) to the storage unit 122, and / or the DNN 110, and / or the GUI 124 (for presenting the results) via the I / O interface 126.

[0063] Those skilled in the art will readily appreciate that the teachings of the subject matter disclosed herein are not limited to Figure 1 The systems shown; equivalent and / or modified functionality may be combined or divided in another manner, and may be implemented in any suitable combination of software and firmware and / or hardware.

[0064] Notice that Figure 1 The inspection system shown can be implemented in a distributed computing environment, where the previously mentioned Figure 1The functional modules shown may be distributed over several local and / or remote devices and may be connected via a communication network. It is also noted that in other embodiments, at least some of the inspection tools 120, storage unit 122, and / or GUI 124 may be external to the inspection system 100 and operate in data communication with the system 101 via the I / O interface 126. The system 101 may be implemented as a stand-alone computer used in conjunction with the inspection tools. Alternatively, the corresponding functions of the system 101 may be at least partially integrated with one or more inspection tools 120, thereby facilitating and enhancing the functions of the inspection tools 120 in the process of the related inspection.

[0065] For purposes of illustration only, the following description is provided for generating DNN training data for training a DNN that can be used to inspect semiconductor samples. Those skilled in the art will readily appreciate that the teachings of the subject matter disclosed herein can be applied to various types of machine learning models and can be applied to various inspections (such as, for example, segmentation, defect detection, ADR, ADC, auto-navigation modules, modules related to metrology, etc.).

[0066] See also Figure 2A , Figure 2A A generalized flow chart for generating training data for training a DNN that can be used to inspect semiconductor samples is shown according to certain embodiments of the subject matter disclosed herein.

[0067] According to certain embodiments, training data for training a DNN (also referred to as DNN training data) may include a plurality of training samples, each training sample including a corresponding training image and corresponding ground truth data associated therewith. The ground truth data may include label data indicating application-specific information. As an example, for an application of image segmentation, each training sample may include a training image of a semiconductor sample and label data indicating one or more segments in the training image.

[0068] The training image may be a "real world" image of a semiconductor sample obtained during the manufacturing process of the semiconductor sample (e.g., an FP image as described above). As a non-limiting example, the image may be a test image obtained by inspecting the sample using one or more inspection tools for detecting potential defects (typically at a relatively high speed and / or a relatively low resolution). Such an inspection tool may be, for example, an optical inspection system, a low-resolution SEM, etc. Alternatively, the image may be a review image obtained by inspecting the sample at a subset of potential defect locations using one or more review tools (typically at a relatively low speed and / or a relatively high resolution) to confirm whether the potential defects detected by the inspection tool are actually defects. For example, such a review tool may be a scanning electron microscope (SEM), etc.

[0069] Ground truth data can be obtained in various ways. As an example, ground truth data can be generated by manual annotation, synthetically generated (e.g., CAD-based images), generated by machine learning annotation (e.g., labels based on feature extraction and analysis), or a combination of the above.

[0070] According to certain embodiments, a DNN is trained for a specific application of image segmentation on semiconductor sample images (e.g., FP images). As used herein, the term "segmentation" may refer to any process that divides an image into meaningful parts / segments (e.g., background and foreground, noisy and non-noisy regions, structural elements, defects and non-defects, etc.) while providing per-pixel or per-region values ​​indicative of these segments. DNN training data generated according to the presently disclosed subject matter is used to train a DNN that can be used for FP image segmentation.

[0071] As an example, the training image may be a SEM image or a derivative thereof, and the label data may be a segmentation map (also referred to as a segmentation mask) corresponding to the SEM image. The segmentation map provides information of predicted labels associated with corresponding pixels in the training image. Each predicted label indicates the segment to which the corresponding pixel in the image belongs.

[0072] Segmentation maps are traditionally generated by users manually annotating the entire training image. Given the size and resolution of the image, the manual annotation process is very time-consuming, and the results of the annotation are sometimes inaccurate and error-prone. This is not what users want, because the quality of the labeled data directly affects the training process and the performance of the trained DNN. Therefore, as described below, Figure 2A and Figure 2B As described, the present disclosure proposes a more efficient way to automatically generate accurately labeled data that can be used to train DNNs.

[0073] According to certain embodiments, a first training image representing at least a portion of a semiconductor sample may be obtained (202) (e.g., by PMC 102 via I / O interface 126) along with first labels respectively associated with a set of pixels in each of one or more segments selected by a user from the first training image. In some cases, at least one of the one or more segments corresponds to one or more structural elements present in the first training image.

[0074] Now turn to Figure 5, where examples of training images according to certain embodiments of the subject matter disclosed herein are shown. A training image 502 is illustrated as an SEM image captured by an SEM review tool and representing a portion of a die of a wafer. As shown, there are multiple structural elements 504 (shown as polygons, used to represent contact elements on the wafer) in the image. A structural element as used herein may refer to any primitive object on the image data, the primitive object having a geometric shape that is combined with other objects in some cases or having a geometric structure with an outline. For example, a structural element may be presented in the form of a polygon.

[0075] In some embodiments, the training images may contain multiple channels captured from different perspectives. Figure 5 In the example of , 502 may represent one channel of an image taken by a top detector of the inspection tool from a vertical perspective, and 506 may represent another channel of an image taken by a side detector of the inspection tool from a side perspective. In some cases, there may be more than one side detector from different angles, and thus the training image may contain multiple side channel images 506. In some cases, multiple side channel images may be combined into one combined side channel image.

[0076] In order to save the user the effort of fully annotating the entire image, partial label data associated with the training image can be provided by the user. As an example, the user can determine one or more segments he / she expects to identify in the segmentation map, select a set of pixels from each segment, and annotate the set of pixels to indicate the segment they belong to. Figure 6 An example of partial user annotations on a training image is shown in accordance with some embodiments of the subject matter disclosed herein.

[0077] exist Figure 6 A training image 602 is illustrated in FIG. The training image 602 captures Figure 5 The training images 502 and 506 in FIG. 5 are at the same wafer location and contain multiple structural elements that share the same design pattern. Such structural elements are also referred to as equivalent structural elements. As an example, the user may determine that the structural element may belong to one segment, while the background belongs to another segment. Thus, the user may select a few pixels from either segment and mark them as indicating the corresponding segment. As an example, Figure 6As shown in the circled area 604 in FIG. 2 (an enlarged view of 604 is shown on the right), a group of pixels 606 within the structural element are labeled by the user with a specific grayscale value indicating that they belong to one segment representing the structural element, and another group of pixels 608 in the background are labeled by the user with another grayscale value indicating that they belong to another segment representing the background area. As described below with respect to block 206, the user annotations on the pixels selected in each segment can be used as part of the label data for training the machine learning model.

[0078] It will be noted that the user can determine in the training image the different types of segmentation that he / she wants to include in the segmentation map. As an example, in some cases, the training image may include more than one family of equivalent structural elements. The user can determine that each family of equivalent structural elements belongs to a corresponding segment, or alternatively, he / she can determine that all structural elements belong to one segment, while the background belongs to a different segment. As another example, the user can decide the segments based on the grayscale changes in the image. For example, due to a specific structural contour, there may be grayscale contrast within a structural element, and the user may determine that there are two segments corresponding to the grayscale spacing within each structural element. The present disclosure is not limited by the specific manner of segmentation and the number of segments on the training image.

[0079] continue Figure 2A Based on the description of the first training image, a set of features characterizing the first training image can be extracted (204) (e.g., by feature extractor 104). Each feature has a feature value corresponding to a pixel in the first training image. The feature set can include a first feature indicating context information or representation in the first training image. For example, the first feature can provide information about contextual relationships / associations between one or more segments in the first training image. The feature set can also include a second feature that provides information about the distribution of pixels in the first training image relative to a statistical measure of a set of pixels in each segment.

[0080] The first feature is also referred to as background feature in this article. The term "background feature" should be broadly interpreted as representing the background (e.g., surrounding areas that may have different ranges) of a specific item / element (such as, for example, a pixel, a structural element, or a segment, etc.) in the image, and / or the global background relationship between such elements or segments in the image. Between elements or segments, the background relationship can be embodied in terms of relative texture, spatial layout, intensity, etc. For example, the pixels within the structural element share similar textures with the pixels around them (surrounding but still within the structural element), while the textures of the pixels outside the structural element are very different from the pixels within the structural element, indicating that the pixels within the structural element and the pixels outside the structural element may belong to different segments. In some embodiments, the first feature may include a texture feature. In some cases, the first feature may also include an intensity feature.

[0081] Texture features can be used to represent the perceived texture of an image. For example, such features can represent the spatial arrangement of structures or patterns that characterize different elements or regions in an image. Figure 3A , which shows a generalized flow chart for extracting texture features from a first training image according to some embodiments of the subject matter disclosed herein.

[0082] A first training image may be processed (302) with a set of filters (also referred to as a filter bank) to produce a set of feature responses corresponding to the set of filters. Each feature response comprises a feature response value corresponding to a corresponding pixel of the first training image. Each pixel corresponds to a feature vector comprising the set of feature response values. The feature vectors corresponding to the pixels in the image may be clustered (304) into a plurality of clusters. Each pixel may be assigned (306) a feature value according to the cluster to which its feature vector belongs, thereby producing a feature map comprising feature values ​​corresponding to the pixels of the first training image. The feature map may be separable into a plurality of channels corresponding to the corresponding feature values. Each channel of the feature map may be sampled (308) using a sampling filter, thereby producing a feature map having a plurality of sampled channels.

[0083] In some embodiments, texture features can be implemented as texton features. The term "texton" generally refers to basic microstructures in an image. Textons can provide compact representations for a range of different appearances of objects / elements and can effectively classify textures as well as general object classes.

[0084] As an example, the first training image can be convolved with a filter set (e.g., a multidimensional filter bank) to generate a multidimensional feature response for all pixels in the training image. A filter set can be selected to extract features representing semiconductor images. For example, the filter set can include various filter families, such as, for example, derivative filters, Gabor filters, and Laplace filters. Derivative filters can provide a quantitative measurement of the rate of change of pixel brightness information present in an image. By convolution with the image, derivative filters can generate derivatives of different directions and scales, depending on the choice of the kernel. The information obtained about the rate of change of brightness can be used to enhance contrast, detect edges and boundaries, and measure feature orientation. Gabor filters are linear filters that analyze whether there is any specific frequency content in a specific direction in a local area around a point or area of ​​interest in an image. The frequency and orientation representation of Gabor filters are similar to the frequency and orientation representation of the human visual system. Gabor filter sets with different frequencies and orientations can be used to extract features representing textures from the first training image. Laplace filters are also called Laplacian of Gaussian (LoG) filters. It is a measurement of the second spatial derivative of an image. The Laplacian of an image highlights areas where intensity changes rapidly and can be used for edge detection. For example, in areas of an image with constant intensity (i.e., areas where the intensity gradient is zero), the LoG response will be zero. However, near intensity changes, the LoG response will be positive on the darker side and negative on the lighter side.

[0085] In some embodiments of the present disclosure, three families of filters may be used in combination, and the present disclosure demonstrates that the combined filter set is more reliable when extracting features representing the texture and layout of structures in semiconductor images, which are typically noisy images with poor signal-to-noise ratios (SNR) and contrast-to-noise ratios (CNR). However, this is by no means intended to limit the present disclosure in any way. In addition to or in lieu of the foregoing, other filter families / types or filters with similar functionality may be used. Specifically, it will be noted that the specific filters in each family described above are shown for exemplary purposes. In addition to or in lieu of the foregoing, other suitable filters in the family may also be used. The present disclosure is not limited by the specific types of filters and the number of filters used in each family.

[0086] As an example, assuming that the dimension of the training image is M*N, and the filter set includes F filters, i.e., an F-dimensional filter bank, a multidimensional feature response will be generated, for example, in the form of a feature matrix with a dimension of M*N*F. That is, for each pixel in the training image, there are F feature responses that form a feature vector corresponding to the pixel. The multidimensional feature response (i.e., a feature matrix of M*N*F) can be regarded as containing M*N feature vectors corresponding to M*N pixels.

[0087] Once the multidimensional feature responses for all pixels in the training image are obtained, clustering (e.g., unsupervised clustering) can be performed on the multidimensional feature responses. As an example, a Euclidean distance K-means clustering algorithm can be performed on the M*N feature vectors to generate multiple clusters (e.g., K clusters) of feature vectors. Each pixel in the training image is clustered into one of the K clusters, and an eigenvalue corresponding to the cluster number is assigned to each pixel in the training image, thereby generating a feature map (e.g., a texture primitive map) containing eigenvalues ​​corresponding to pixels of the first training image. The texture primitive map T contains eigenvalues, where for each pixel i, the corresponding eigenvalue Ti∈{1,…,K}. The feature map (e.g., texture primitive map) can be divided into K channels corresponding to the corresponding eigenvalues ​​(e.g., one channel for each texture primitive), where for each channel, an independent channel map (e.g., an integral map) can be derived, which is similar to a histogram with a binary file for each texture primitive. The dimension of the independent channel map is M*N*K.

[0088] For example, sampling can be performed on each channel using a sampling filter. For example, K channel maps can be convolved with the sampling filter respectively, thereby generating the sampled channel map. As an example, the sampling filter can be implemented using averaging, Gaussian, or other types of sampling methods. When convolved with the channel map, it can effectively reflect the background of each pixel (e.g., the surrounding area of ​​each pixel) in the convolution value obtained in the sampled channel map, thereby increasing the sensitivity to the sampled map. K sampled channel maps (e.g., in the form of a feature matrix in the dimension of M*N*K) can be used as texture features corresponding to the first training image.

[0089] According to some embodiments, in addition to the texture feature, the first feature may also include an intensity feature. The intensity feature may represent the spatial arrangement of pixel intensities that characterize different elements or segments in the image. In some embodiments, the intensity feature may also be implemented as a texture primitive feature. As an example, clustering as described above may be performed on the pixel values ​​of the first training image. Similarly, each pixel in the training image is clustered into one of a plurality of clusters, and each pixel in the training image may be assigned a feature value corresponding to the cluster to which the pixel value belongs, thereby generating a feature map that can be separated into a plurality of channels. Each channel may be sampled, for example, by using a sampling filter, and the sampled channel map may be used as an intensity feature corresponding to the first training image.

[0090] Now turn to Figure 7 , which shows examples of texture features and examples of intensity features according to some embodiments of the subject matter disclosed herein. As shown, feature map 702 is a texture feature extracted from a first training image 502, and feature map 704 is an intensity feature extracted from the same image.

[0091] In addition to the first feature, the feature set may also include a second feature that provides information about the distribution of pixels in the first training image relative to a statistical measurement of a set of pixels in each segment. The second feature is also referred to as a statistical feature. The statistical feature mentioned herein may represent a quantitative measure of the statistical importance of a pixel, which indicates the likelihood that the pixel belongs to a particular segment. See Figure 3B , which shows a generalized flow chart for extracting statistical features from a first training image according to some embodiments of the subject matter disclosed herein.

[0092] A statistical measurement value may be calculated (312) for a selected set of pixels in each segment. As an example, the statistical measurement value may be an average value calculated based on the values ​​of the set of pixels in each segment. As another example, the statistical measurement value may be calculated based on the mean and a standard deviation (STD), the STD measuring the amount of change or difference in the values ​​of a set of pixels relative to the mean. For each pixel in the first training image, a corresponding feature value may be calculated (314) as the distance between the value of the pixel and the statistical measurement value for each segment, thereby generating one or more feature maps each containing a feature value corresponding to the distance between the pixel value and the statistical measurement value of the corresponding segment. For example, in Figure 6In the example of , statistical measurements may be calculated for the set of pixels 606 in the segment representing the structural element and the set of pixels 608 in the other segment representing the background area, respectively. Two feature maps may be generated for the first training image, the first feature map containing distance values ​​between each pixel value in the image 602 and the statistical measurement calculated for 606, and the second feature map containing distance values ​​between each pixel value in the image 602 and the statistical measurement calculated for 608. The feature maps generated in this manner thus indicate the distribution / variance of pixel values ​​in the image relative to the statistical measurement calculated for each segment (based on the set of pixels marked by the user), which indicates the likelihood that the pixel belongs to a particular segment.

[0093] continue Figure 2A As described above, once the first training image and the first label are obtained as described with reference to block 202 and the feature set is obtained as described with reference to block 204, a machine learning (ML) model (206) can be trained (e.g., by training module 106) using the first label, the values ​​of a set of pixels selected in each segment associated with the first label, and the feature values ​​of each feature of the feature set corresponding to the set of pixels in each segment. The ML model is trained for image segmentation.

[0094] In some embodiments, the ML model can be implemented as a classifier. The terms "classifier", "classifier model" or "classification model" mentioned herein should be broadly interpreted to cover any learning model that can identify which of a set of categories / classes a new instance belongs to based on a training data set. As an example, in the present disclosure, a classifier can be trained to classify pixel candidates into a set of segment classes defined by a user. For example, in Figure 6 In the example of , a classifier is trained based on a training set, the training set including: a first label indicating two segments (e.g., a user annotation on the set of pixels 606 representing one segment, and a user annotation on the set of pixels 608 representing another segment), pixel values ​​of the set of pixels 606 and the set of pixels 608 in the first training image, and feature values ​​corresponding to the set of pixels 606 and the set of pixels 608 (in each feature of a feature set). For example, assume that the feature set includes texture features as illustrated in 702, intensity features as illustrated in 704, and feature values ​​as illustrated in the above reference. Figure 3B The statistical features described, the feature values ​​in each of the three features (in the form of a feature map or a feature matrix) corresponding to the selected group of pixels 606 and the group of pixels 608 are used as part of the training set.

[0095] The trained classifier can be used for image segmentation, i.e., for providing a predicted label for each pixel in the image indicating the segment to which it belongs. It will be noted that the classifier can be implemented as various types of machine learning models, such as, for example, linear classifiers, support vector machines (SVMs), neural networks, decision trees, etc., and the present disclosure is not limited to specific models implemented with these.

[0096] The first training image can be processed (208) using the trained ML model to obtain a first segmentation map that provides information of predicted labels associated with corresponding pixels in the first training image. Each predicted label indicates a segment to which the corresponding pixel belongs. According to certain embodiments, the trained ML model (e.g., a classifier) ​​can be used to process at least the remaining pixels in the first training image (e.g., excluding the set of pixels selected in each segment and used to train the classifier). The output of the classifier is a segmentation map (also called a segmentation mask) that is the same size as the training image and contains the predicted labels for the corresponding pixels in the image.

[0097] Determine (210) (e.g., by the training module 106) to include a first training sample comprising a first training image and a first segmentation map into the DNN training data when a criterion is met, and to repeat the extraction of second features, training, and processing when the criterion is not met. In other words, as described above, whether to include the first training sample in the DNN training data or to repeat the processing can be determined based on the criterion. In some embodiments, the criterion can involve, for example, feedback by a user on the first segmentation map. As an example, the criterion can be based on user feedback. For example, the criterion can be positive user feedback about the first segmentation map, for example, the first segmentation map has a satisfactory quality. Therefore, once positive user feedback about the first segmentation map is received, the first training sample can be included in the DNN training data.

[0098] Once negative user feedback about the first segmentation map is received, for example, the quality of the first segmentation map is not satisfactory, this means that the currently trained ML model has not learned to segment the first training image as well as the user expected, so it is necessary to repeat the above reference. Figure 2A At least a portion of the operations described. Figure 8 An example of a segmentation map with negative user feedback is shown in accordance with some embodiments of the subject matter disclosed herein.

[0099] Generate and Figure 5 and Figure 6800 corresponding to the training image illustrated in . In this example, the segmentation map is illustrated as a binary map representing two segments, the first segment corresponding to the structural element in the image (shown as a white circle representing the contact element), and the second segment corresponding to the background area (shown as a black area). As shown in the figure, for the region 802 as the edge background area along the image boundary, it should be marked as black, but it is mistakenly marked as white. Therefore, region 802 is mistakenly predicted to belong to the first segment of the structural element. For region 804, a similar error occurs when the background area between the two structural elements is mistakenly predicted to belong to the first segment (illustrated in the figure as two elements connected to each other, so the background area is missing between them). According to the user's review, this segmentation map is not satisfactory due to the wrong segmented area. Therefore, the classifier needs to be retrained until it generates an output segmentation map with satisfactory quality. It will be noted that in some cases, the satisfactory quality of the segmentation map can be determined according to a quality condition. For example, the quality condition can be a range or threshold of the percentage of pixels with correct segmentation.

[0100] Now turn to Figure 2B , showing a generalized flow chart of a process for continuing DNN training data generation upon receiving negative user feedback regarding a first segmentation map according to certain embodiments of the subject matter disclosed herein.

[0101] Specifically, additional first labels associated with additional sets of pixels in at least one segment may be obtained (212) from the user. In this case, the additional first labels may be added as part of the label data along with the first labels previously provided by the user. The first labels and the additional first labels constitute aggregated label data. See Fig. 9A , which shows an example of additional first tags provided by a user according to some embodiments of the subject matter disclosed herein.

[0102] like Fig. 9A As shown in the first training image 602 in FIG. , in addition to the original first label (as shown in the circled area 604), additional groups of pixels (as shown in the circled areas 902 and 904) are labeled by the user with labels indicating the expected segments to which these pixels belong. For example, the user can specifically mark the areas in the first segmentation map where there are incorrect segmentations (e.g., in areas 802 and 804). The user can choose to mark additional pixels in one segment or in both segments. For example, in 902, the user marked pixels in both the structural element and the background, while in 904, the user marked pixels only in the background. Therefore, after the additional marking, the aggregated label data includes labels marked for those groups of pixels in areas 604, 902, and 904.

[0103] Once the aggregated label data is obtained, the second feature extraction, training, and processing can be repeated based on the aggregated label data until the criteria are met, such as receiving positive user feedback. Specifically, since the label data has been updated, the second feature (i.e., the statistical feature) calculated based on the set of pixels in each segment associated with the label should also be updated based on the aggregated label data (214). For example, the second feature can be re-extracted using a statistical measurement value calculated based on the pixel values ​​of the set of pixels and the additional set of pixels.

[0104] The classifier may be retrained (216) using the aggregated labels, the pixel values ​​associated with the aggregated label data, and the feature values ​​of each feature of the feature set corresponding to the pixel associated with the aggregated label data. The first training image may be processed again (218) using the retrained classifier to obtain an updated first segmentation map. Subsequently, it may be determined (220) whether to repeat the above process (refer to the operations of blocks 212-218) or to include the updated first segmentation map and the first training image as the first training sample in the DNN training data based on user feedback on the updated first segmentation map.

[0105] See also Fig. 9B , showing an example of an updated segmentation map with positive user feedback according to some embodiments of the subject matter disclosed herein.

[0106] Once you obtain Fig. 9A The additional first label shown, according to the reference Figure 2B The described operations generate a segmentation map 900. The segmentation map 900 is as follows Figure 8 800. As shown, in contrast to the segmentation map 800, the updated segmentation map 900 provides satisfactory segmentation results for the two segments. With the help of the additional first labels provided by the user and the retraining of the classifier, the segmentation in the previously mis-segmented regions 802 and 804 is fixed. The updated segmentation map is included in the DNN training data as a training sample together with the corresponding training image.

[0107] According to some embodiments, the following may be repeated for one or more training images: Figure 2A and Figure 2BThe process shown in order to generate one or more segmentation maps, which can be included in the DNN training data together with the training image. As an example, a second training image can be obtained together with a second label associated with a set of pixels in each of one or more segments selected by a user from the second training image. The second label is added to the aggregated label data. A feature set characterizing the second training image can be extracted, the feature set including a first feature and a second feature. The aggregated label data, the pixel values ​​associated with the aggregated label data, and the feature values ​​of each feature of the feature set corresponding to the pixels associated with the aggregated label data can be used to train a machine learning model. The processing and determination described above with reference to boxes 208 and 210 can be performed based on the second training image.

[0108] Now turn to Figure 4 , showing a generalized flow chart for training a DNN using generated DNN training data according to certain embodiments of the subject matter disclosed herein.

[0109] According to certain embodiments, a DNN is trained for a specific application of image segmentation on a semiconductor sample image (e.g., FP image). As an example, such segmentation can be used by a metrology tool to perform measurements on the sample. As another example, the segmentation can also be used for ADC in constructing attributes (e.g., to define whether a defect is on the main pattern, on the background, or both), ADR for applying a segment-specific detection threshold on each segment, etc.

[0110] The process includes a setup phase 410 of segmentation-specific training of a DNN and a runtime phase 420 of performing image segmentation using the trained DNN.

[0111] During the setting 410, it is possible to obtain (412) according to Figure 2A and Figure 2B DNN training data generated by the description of . The DNN training data includes one or more training samples, each training sample including a training image and a corresponding segmentation map. The DNN can be trained (414) using the DNN training data to obtain a trained DNN characterized by training parameters related to segmentation. In some embodiments, the training process can be cyclical and can be repeated several times until the DNN is fully trained. As an example, the DNN can be trained using a cost function related to segmentation accuracy (e.g., correct segmentation label map versus predicted label map). The trained DNN can be verified (416) using a verification set of images. The verification set of images can be a set of images different from the training image set and can include FP images selected for verification purposes. The user can provide feedback on the results achieved by the DNN during verification.

[0112] During runtime 420, the PMC uses the trained DNN to process (422) one or more runtime images including one or more captured FP images to be segmented to provide (424) a segmentation map for each image. The one or more FP images may be obtained by the same or different examination modalities. The obtained segmentation map may provide information indicating per-pixel or per-region segmentation labels for different sections on the image. As an example, polygons on one layer may have one segmentation label, and polygons on another layer may have a different segmentation label, while the background may have a separate segmentation label.

[0113] One of the advantages of certain embodiments of the training data generation process as described herein is that it provides an automatic means of generating accurate ground truth data that can be used to train DNNs, compared to the time-consuming and error-prone traditional manual annotation of entire training images. This process is automated at least by extracting meaningful features suitable for segmentation and interactive training of ML models based on user partial annotations and feedback.

[0114] One of the further advantages of certain embodiments of the training data generation process as described herein is that by extracting and using background features and statistical features that characterize the training images, the present disclosure seeks to provide a reliable representation of semiconductor images in terms of texture, layout, background, and statistical significance, so that the representative features can be used to train the ML model and generate segmentation maps with satisfactory results in an efficient manner.

[0115] It will be noted that the examples shown are described herein for illustration purposes only and should not be considered to limit the present disclosure in any way.Other suitable examples may be used in addition to or in place of the above.

[0116] It will be understood that the present disclosure is not limited in its application to the details set forth in the description contained herein or shown in the drawings.

[0117] It will also be understood that the system according to the present disclosure can be implemented at least in part on a suitably programmed computer. Likewise, the present disclosure contemplates a computer program readable by a computer for performing the method of the present disclosure. The present disclosure further contemplates a non-transitory computer-readable memory tangibly embodying a program of instructions that can be executed by a computer for performing the method of the present disclosure.

[0118] The present disclosure can have other embodiments and be practiced and executed in various ways. Therefore, it will be understood that the wording and terminology adopted herein are for descriptive purposes and should not be considered as limiting. Therefore, it will be understood by those skilled in the art that the concepts on which the present disclosure is based can be easily used as the basis for other structures, methods, and systems designed to achieve the several purposes of the subject matter disclosed herein.

[0119] It will be readily appreciated by those skilled in the art that various modifications and changes may be applied to the embodiments of the present disclosure as described above, without departing from the scope of the present disclosure as defined in and by the accompanying claims.

Claims

1. A computerized system for generating training data (DNN training data) for training a deep neural network that can be used for semiconductor sample inspection, the system comprising a processor and a memory circuit (PMC) configured to: obtaining a first training image of the semiconductor sample and first labels respectively associated with a group of pixels in each of one or more sections identified by a user from the first training image, the first labels being provided as partial label data of the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features comprising a first feature indicating background information in the first training image, and a second feature providing information on a distribution of pixels in the first training image relative to a statistical measurement of the set of pixels in each segment; training a machine learning (ML) model for image segmentation using the first label, the values ​​of the set of pixels selected in each segment, and the feature values ​​of each feature of the feature set corresponding to the set of pixels in each segment; Processing the first training image using the trained ML model to obtain a first segmentation map, the first segmentation map providing information of predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs; and It is determined that a first training sample including the first training image and the first segmentation map is included in the DNN training data when a criterion is met, wherein the first segmentation map is used as complete label data for the first training image.

2. The computerized system of claim 1, wherein the training data is usable to train a deep neural network (DNN) for at least one inspection process selected from the group consisting of: automatic segmentation based on runtime images, automatic metrology, automatic defect detection, automatic defect review, and automatic defect classification.

3. The computerized system of claim 1, wherein the first feature represents a context of one or more elements in the first training image, and a contextual relationship between the one or more elements.

4. A computerized system as described in claim 1, wherein the criterion is based on user feedback about the first segmentation map, and the PMC is configured to: obtain an additional first label associated with an additional group of pixels in at least one of the segments when negative user feedback about the first segmentation map is received, the first label and the additional first label constitute aggregated label data; and repeat the extracting of the second feature, the training and the processing based on the aggregated label data until positive user feedback is received. 5 . The computerized system of claim 4 , wherein the PMC is configured to include the first training sample into the training data upon receiving positive user feedback regarding the first segmentation map.

6. A computerized system as described in claim 5, wherein the PMC is further configured to: obtain a second training image, and a second label associated with a set of pixels respectively selected in each of one or more segments identified by a user from the second training image, and the second label is added to the aggregated label data; extract a feature set that characterizes the second training image and includes the first feature and the second feature; train the ML model using the aggregated label data, the pixel values ​​associated with the aggregated label data, and the feature value of each feature of the feature set corresponding to the pixel associated with the aggregated label data; and perform the processing and determination based on the second training image.

7. The computerized system of claim 1, wherein at least one of the one or more segments corresponds to one or more structural elements present in the first training image.

8. The computerized system of claim 1, wherein the first feature comprises a texture feature extracted by: The first training image is processed with a set of filters to generate a set of feature responses each containing feature response values ​​corresponding to a corresponding pixel of the first training image, wherein each pixel corresponds to a feature vector including Its characteristic response value set; clustering the feature vectors corresponding to the pixels in the first training image into a plurality of clusters; Assigning a feature value to each pixel according to the cluster to which its feature vector belongs, thereby generating a feature map containing feature values ​​corresponding to the pixels of the first training image, wherein the feature map can be separated into a plurality of channels corresponding to the respective feature values; as well as Each channel of the feature map is sampled using a sampling filter, thereby generating a feature map having a plurality of sampled channels.

9. The computerized system of claim 8, wherein the first feature further comprises an intensity feature extracted by: clustering the pixel values ​​of the first training image into a plurality of clusters; assigning a feature value to each pixel according to the cluster to which the pixel value belongs, thereby generating a feature map separable into a plurality of channels corresponding to the respective feature values; and sampling each channel of the feature map using a sampling filter, thereby generating a feature map having a plurality of sampled channels.

10. The computerized system of claim 8, wherein the filter set comprises one or more of the following: a derivative filter, a Gabor filter, and a Laplacian filter.

11. A computerized system as described in claim 1, wherein the second feature is extracted by the following steps: calculating a statistical measurement value for the set of pixels in each segment; and for each pixel in the first training image, calculating a corresponding feature value as the distance between the value of the pixel and the statistical measurement value for each segment, thereby generating one or more feature maps each containing a feature value, wherein the feature value corresponds to the distance between the pixel value and the statistical measurement value of the corresponding segment.

12. The computerized system of claim 1, wherein the PMC is further configured to train the DNN using the DNN training data and to validate the trained DNN using a validation set of images.

13. A computerized method for generating training data (DNN training data) for training a deep neural network that can be used for semiconductor sample inspection, the method being executed by a processor and memory circuit (PMC) and comprising: obtaining a first training image of the semiconductor sample and first labels respectively associated with a group of pixels in each of one or more sections identified by a user from the first training image, the first labels being provided as partial label data of the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features comprising a first feature indicating background information in the first training image, and a second feature providing information on a distribution of pixels in the first training image relative to a statistical measurement of the set of pixels in each segment; training a machine learning (ML) model for image segmentation using the first label, the values ​​of the set of pixels selected in each segment, and the feature values ​​of each feature of the feature set corresponding to the set of pixels in each segment; Processing the first training image using the trained ML model to obtain a first segmentation map, the first segmentation map providing information of predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs; and It is determined that a first training sample including the first training image and the first segmentation map is included in the DNN training data when a criterion is met, wherein the first segmentation map is used as complete label data for the first training image.

14. A computerized method as described in claim 13, wherein the criterion is based on user feedback about the first segmentation map, and the method further comprises: obtaining an additional first label associated with an additional group of pixels in at least one of the segments when negative user feedback about the first segmentation map is received, the first label and the additional first label constituting aggregated label data; and repeating the extracting of the second feature, the training and the processing based on the aggregated label data until positive user feedback is received.

15. The computerized method of claim 14, further comprising including the first training sample into the training data upon receiving positive user feedback regarding the first segmentation map.

16. The computerized method of claim 13, wherein the first feature comprises a texture feature extracted by: The first training image is processed with a set of filters to generate a set of feature responses each containing feature response values ​​corresponding to a corresponding pixel of the first training image, wherein each pixel corresponds to a feature vector including Its characteristic response value set; clustering the feature vectors corresponding to the pixels in the first training image into a plurality of clusters; Assigning a feature value to each pixel according to the cluster to which its feature vector belongs, thereby generating a feature map containing feature values ​​corresponding to the pixels of the first training image, wherein the feature map can be separated into a plurality of channels corresponding to the respective feature values; as well as Each channel of the feature map is sampled using a sampling filter, thereby generating a feature map having a plurality of sampled channels.

17. A computerized method as described in claim 16, wherein the first feature also includes an intensity feature extracted by the following steps: clustering the pixel values ​​of the first training image into multiple clusters; assigning a feature value to each pixel according to the cluster to which the pixel value belongs, thereby generating a feature map that can be separated into multiple channels corresponding to the corresponding feature values; and using a sampling filter to sample each channel of the feature map, thereby generating a feature map with multiple sampled channels.

18. A computerized method as described in claim 13, wherein the second feature is extracted by the following steps: calculating a statistical measurement value for the set of pixels in each segment; and for each pixel in the first training image, calculating a corresponding feature value as the distance between the value of the pixel and the statistical measurement value for each segment, thereby generating one or more feature maps each containing a feature value, wherein the feature value corresponds to the distance between the pixel value and the statistical measurement value of the corresponding segment.

19. The computerized method of claim 13, further comprising training the DNN using the DNN training data, and validating the trained DNN using a validation set of images.

20. A non-transitory computer-readable storage medium tangibly embodying a program of instructions that, when executed by a computer, cause the computer to perform a method of generating training data for training a deep neural network (DNN training data) that can be used for semiconductor sample inspection, the method comprising: obtaining a first training image of the semiconductor sample and first labels respectively associated with a group of pixels in each of one or more sections identified by a user from the first training image, the first labels being provided as partial label data of the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features comprising a first feature indicating background information in the first training image, and a second feature providing information on a distribution of pixels in the first training image relative to a statistical measurement of the set of pixels in each segment; training a machine learning (ML) model for image segmentation using the first label, the values ​​of the set of pixels selected in each segment, and the feature values ​​of each feature of the feature set corresponding to the set of pixels in each segment; Processing the first training image using the trained ML model to obtain a first segmentation map, the first segmentation map providing information of predicted labels associated with corresponding pixels in the first training image, each predicted label indicating a segment to which the corresponding pixel belongs; and It is determined that a first training sample including the first training image and the first segmentation map is included in the DNN training data when a criterion is met, wherein the first segmentation map is used as complete label data for the first training image.

Images