Generating training data that can be used to inspect semiconductor samples

The system addresses the inefficiencies of manual annotation by using automated feature extraction and iterative learning to generate accurate training data for deep neural networks, improving defect detection and segmentation in semiconductor inspection.

JP7579756B2Active Publication Date: 2024-11-08APPL MATERIALS ISRAEL LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021095076
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-07-29
Filing Date
2021-06-07
Publication Date
2024-11-08
Estimated Expiration
2041-06-07

AI Technical Summary

Technical Problem

Current semiconductor inspection processes face challenges in efficiently generating accurate training data for deep neural networks due to the time-consuming and error-prone nature of manual annotation of large semiconductor images, which affects the performance of defect detection and segmentation.

Method used

A system and method for generating training data using a computerized process that extracts contextual and statistical features from partial user annotations, iteratively refining the training data through machine learning models until satisfactory segmentation maps are achieved, thereby automating the generation of accurate training data for deep neural networks.

Benefits of technology

This approach reduces the time and errors associated with manual annotation, providing efficient and accurate training data for deep neural networks, enhancing the precision of defect detection and segmentation in semiconductor inspection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579756000001
    Figure 0007579756000001
  • Figure 0007579756000002
    Figure 0007579756000002
  • Figure 0007579756000003
    Figure 0007579756000003
Patent Text Reader

Abstract

To provide a system and a method for generating training data for training a deep neural network usable for inspecting a semiconductor sample.SOLUTION: A method includes the steps of: acquiring a first training image and a first label respectively associated with a group of pixels selected in each segment; extracting a set of features characterizing the first training image; training a machine learning (ML) model using the first label, values of the group of pixels, and feature values of each feature of the set of features corresponding to the group of pixels; processing the first training image using the trained ML model to acquire a first segmentation map; and determining to include the first training image and the first segmentation map into DNN training data when a criterion is satisfied, and to repeat the extraction, training, and processing of the second features when the criterion is not satisfied.SELECTED DRAWING: Figure 2A
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The subject matter of this disclosure relates generally to the field of inspection of semiconductor specimens, and more particularly to generating training data usable for inspection of the specimens. [Background technology]

[0002] Current demands for high density and performance associated with ultra-large scale integration of manufactured devices require sub-micron features, increased transistor and circuit speeds, and improved reliability. As semiconductor processes advance, pattern dimensions such as linewidths and other types of critical dimensions are continually shrinking. Such demands require the formation of device features with high precision and uniformity, which requires careful monitoring of the manufacturing process, including automated inspection of devices while they are still in the form of semiconductor wafers.

[0003] As a non-limiting example, runtime inspection can employ a two-stage procedure, e.g., inspection of the specimen followed by review of sampled locations of potential defects. Inspection generally involves directing light or electrons at the wafer and generating a particular output (e.g., image, signal, etc.) for the specimen by detecting the light or electrons from the wafer. During the first stage, the surface of the specimen is inspected at high speed and relatively low resolution. Defect detection is typically performed by applying defect detection algorithms to the inspection output. A defect map is generated to indicate suspect locations on the specimen that are likely to be defective. In most cases, the objective of the inspection is to provide high sensitivity to detect defects of interest while suppressing detection of nuisance and noise on the wafer. During the second stage, at least some of the suspect locations are more thoroughly analyzed at a relatively high resolution. In some cases, both stages can be performed on the same inspection tool, or these two stages can be performed on different inspection tools.

[0004] An inspection process can include multiple inspection steps. An inspection step can be performed multiple times during a manufacturing process, such as after manufacturing or processing of a particular layer. Additionally or alternatively, each inspection step can be repeated multiple times, for example, for different wafer locations or for the same wafer location with different inspection settings.

[0005] Inspection processes are used at various steps during semiconductor manufacturing to detect and classify defects on specimens and to perform metrology-related operations. The effectiveness of inspection can be increased by automating processes such as defect detection, automatic defect classification (ADC), automatic defect review (ADR), image segmentation, and automated metrology-related operations. Summary of the Invention

[0006] According to certain aspects of the subject matter of the present disclosure, a computerized system for generating training data for training a deep neural network (DNN training data) usable for inspection of a semiconductor specimen is provided, the system including: obtaining a first training image representing at least a portion of the semiconductor specimen and a first label associated with a selected set of pixels in each of one or more segments identified by a user from the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features including a first feature providing information of a contextual relationship between one or more segments in the first training image and a second feature providing information of a distribution of pixels in the first training image relative to a statistical measure of the pixels in each segment; and extracting the first label, the first label, and the first label. the first training image and the first segmentation map, the first training image being processed using the trained ML model to obtain a first segmentation map providing information of a predicted label associated with each pixel in the first training image, each predicted label indicating a segment to which the respective pixel belongs; and determining to include the first training sample comprising the first training image and the first segmentation map in the DNN training data if a criterion is met, and to repeat the extraction, training, and processing of the second feature if the criterion is not met.

[0007] In addition to the features described above, the system according to this aspect of the disclosed subject matter may include one or more of the features (i)-(xi) listed below, in any desired combination or permutation that is technically possible. (i) The training data may be usable to train a deep neural network (DNN) for at least one inspection process selected from the group including automatic segmentation, automatic metrology, automatic defect detection, automatic defect review, and automatic defect classification based on the runtime images. (ii) the criterion is based on user feedback on the first segmentation map, and the PMC is configured, upon receiving negative user feedback on the first segmentation map, to obtain additional first labels associated with additional groups of pixels in at least one of the segments, where the first label and the additional first labels constitute aggregated label data; and repeat the extraction, training, and processing of second features based on the aggregated label data until positive user feedback is received. (iii) the PMC is configured to include the first training sample in the training data upon receiving positive user feedback for the first segmentation map. (iv) the PMC is further configured to obtain second training images and second labels respectively associated with selected groups of pixels in each of the one or more segments identified by the user from the second training images, the second labels being added to the aggregated label data; characterize the second training images and extract a set of features including the first feature and the second feature; train an ML model using the aggregated label data, values ​​of the pixels associated with the aggregated label data, and feature values ​​of each feature of the set of features corresponding to the pixels associated with the aggregated label data; and perform processing and decisions based on the second training images. (v) at least one of the one or more segments corresponds to one or more structural elements presented in the first training image; (vi) the first features include texture features extracted by: processing the first training images with a set of filters to produce a set of feature responses each including a feature response value corresponding to a respective pixel of the first training images, where each pixel corresponds to a vector including its set of feature response values; clustering feature vectors corresponding to pixels in the first training images into a plurality of clusters, and assigning to each pixel a feature value according to which cluster its feature vector belongs to, producing a feature map including the feature values ​​corresponding to pixels of the first training images, where the feature map is separable into a plurality of channels corresponding to the respective feature values; and sampling each channel of the feature map using a sampling filter to produce a feature map having a plurality of sampled channels. (vii) the first features further include intensity features extracted by clustering pixel values ​​of the first training image into a plurality of clusters; assigning a feature value for each pixel according to the cluster to which the pixel value belongs, resulting in a feature map separable into a plurality of channels corresponding to the respective feature values; and sampling each channel of the feature map using a sampling filter, resulting in a feature map having a plurality of sampled channels. (viii) the set of filters includes one or more of a differential filter, a Gabor filter, and a Laplacian filter. (ix) Second features are extracted by computing a statistical measure for pixels in each segment, and for each pixel in the first training image, computing a corresponding feature value as the distance between the pixel's value and the statistical measure for each segment, resulting in one or more feature maps, each containing a feature value corresponding to the distance between the pixel value and the statistical measure for the respective segment. (x) The training images include multiple channels captured from different viewpoints, and a set of features is extracted from the multiple channels. (xi) The PMC is further configured to train a DNN using the DNN training data and validate the trained DNN using a validation set of images.

[0008] According to another aspect of the subject matter of the present disclosure, there is provided a method for generating training data for training a deep neural network (DNN training data) usable for inspection of a specimen, the method being executed by a processor and memory circuit (PMC) and comprising the steps of obtaining a first training image representing at least a portion of a semiconductor specimen and first labels respectively associated with selected pixels in each of one or more segments identified by a user from the first training image; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features including first features providing information of a contextual relationship between one or more segments in the first training image and second features providing information of a distribution of pixels in the first training image relative to a statistical measure of the pixels in each segment. training a machine learning (ML) model using the first label, values ​​of selected pixels in each segment associated with the first label, and feature values ​​of each feature of the set of features corresponding to pixels in each segment, where the ML model is trained for image segmentation; processing the first training images using the trained ML model to obtain a first segmentation map providing information of predicted labels associated with each pixel in the first training images, where each predicted label indicates a segment to which the respective pixel belongs; and determining to include the first training sample including the first training images and the first segmentation map in the DNN training data if a criterion is met, and to repeat the extraction, training, and processing of a second feature if the criterion is not met.

[0009] This aspect of the subject matter of the present disclosure may include one or more of the features (i)-(xi) recited above for the system, in any desired combination or permutation that is technically possible, mutatis mutandis.

[0010] According to another aspect of the subject matter of the present disclosure, a non-transitory computer readable medium is provided that includes instructions that, when executed by a computer, cause the computer to perform a method for generating training data for training a deep neural network (DNN training data) that can be used to inspect a specimen, the method including obtaining a first training image representing at least a portion of a semiconductor specimen and first labels associated with selected pixels in each of one or more segments identified by a user from the first training image, the first labels being associated with selected pixels in each of one or more segments identified by a user; extracting a set of features characterizing the first training image, each feature having a feature value corresponding to a pixel in the first training image, the set of features including first features providing information of a contextual relationship between one or more segments in the first training image, and a statistical measure of a distribution of pixels in the first training image relative to a statistical measure of the pixels in each segment. training a machine learning (ML) model using the first label, values ​​of selected pixels in each segment associated with the first label, and feature values ​​of each feature of the set of features corresponding to pixels in each segment, where the ML model is trained for image segmentation; processing the first training images using the trained ML model to obtain a first segmentation map informative of a predicted label associated with each pixel in the first training images, where each predicted label indicates a segment to which the respective pixel belongs; and determining to include the first training sample including the first training images and the first segmentation map in the DNN training data if a criterion is met, and to repeat the extraction, training, and processing of the second features if the criterion is not met.

[0011] This aspect of the subject matter of the present disclosure may include one or more of the features (i)-(xi) recited above for the system, in any desired combination or permutation that is technically possible, mutatis mutandis.

[0012] In order to understand the present disclosure, and how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a generalized block diagram of an inspection system in accordance with certain embodiments of the presently disclosed subject matter. [Figure 2A] 1 is a generalized flow diagram for generating training data for training a DNN that can be used for inspection of a semiconductor specimen, in accordance with certain embodiments of the disclosed subject matter. [Figure 2B] 1 is a generalized flow diagram of a continuing process of generating DNN training data upon receiving negative user feedback for a first segmentation map, in accordance with certain embodiments of the subject matter of this disclosure. [Figure 3A] 1 is a generalized flow diagram for extracting texture features from a first training image in accordance with certain embodiments of the subject matter of this disclosure. [Figure 3B] 1 is a generalized flow diagram for extracting statistical features from a first training image in accordance with certain embodiments of the subject matter of this disclosure. [Figure 4] 1 is a generalized flow diagram of using generated DNN training data to train a DNN, in accordance with certain embodiments of the subject matter of this disclosure. [Diagram 5] 1 is an example of a training image in accordance with certain embodiments of the subject matter of this disclosure. [Figure 6] 1 is an example of partial user annotations for a training image in accordance with certain embodiments of the subject matter of this disclosure. [Figure 7] 1 is a diagram of an example of a texture feature and an example of an intensity feature in accordance with certain embodiments of the subject matter of this disclosure. [Figure 8]FIG. 13 is an example of a segmentation map with negative user feedback in accordance with certain embodiments of the subject matter of this disclosure. [Figure 9A] FIG. 13 is a diagram of an example of an additional first label provided by a user in accordance with certain embodiments of the subject matter of the present disclosure. [Figure 9B] FIG. 13 is an example of an updated segmentation map with positive user feedback in accordance with certain embodiments of the subject matter of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, those skilled in the art will understand that the subject matter of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the subject matter of the present disclosure.

[0015] Unless otherwise indicated, and as will be apparent from the discussion below, discussions utilizing terms such as "generate," "train," "obtain," "extract," "process," "determine," "repeat," "include," "receive," "cluster," "assign," "sample," "calculate," "use," "verify," and the like throughout this specification refer to computer operations and / or processes that manipulate and / or transform data into other data, said data being represented as physical quantities such as electronic quantities, and / or said data representing physical objects. The term "computer" should be interpreted broadly to encompass any kind of hardware-based electronic device having data processing capabilities, including, as non-limiting examples, the inspection systems, training data generation systems, and respective portions thereof disclosed in this application.

[0016] The term "inspection" as used herein should be interpreted broadly to encompass any kind of metrology-related operation, as well as operations related to the detection and / or classification of defects in a specimen during production. Inspection is performed by using a non-destructive inspection tool during or after the production of the specimen to be inspected. As non-limiting examples, the inspection process can include run-time scanning (in single or multiple scans), sampling, reviewing, measuring, classifying, and / or other operations performed on the specimen or a portion thereof using the same or different inspection tools. Similarly, inspection can be performed prior to the production of the specimen to be inspected, and can include, for example, generating an inspection recipe and / or other set-up operations. It should be noted that unless otherwise stated, the term "inspection" as used herein or its derivatives is not limited with respect to the resolution or size of the inspection area. Various non-destructive inspection tools include, as non-limiting examples, scanning electron microscopes, atomic force microscopes, optical inspection tools, and the like.

[0017] As used herein, the terms "non-transitory memory" and "non-transitory storage medium" should be interpreted broadly to encompass any volatile or non-volatile computer memory suitable for the subject matter of this disclosure.

[0018] The term "sample" as used herein should be interpreted broadly to encompass all types of wafers, masks, and other structures, combinations and / or portions thereof, used in the manufacture of semiconductor integrated circuits, magnetic heads, flat panel displays, and other semiconductor manufacturing articles.

[0019] As used herein, the term "defect" should be interpreted broadly to encompass any type of abnormality or undesirable feature formed on or within a specimen.

[0020] The term "design data" as used herein should be broadly interpreted to encompass any data that represents a hierarchical physical design (layout) of a sample. The design data can be provided by a respective designer and / or derived from a physical design (e.g., via complex simulations, simple geometric operations, Boolean operations, etc.). The design data can be provided in different formats, such as, as non-limiting examples, GDSII format, OASIS format, etc. The design data can be displayed in vector format, grayscale intensity image format, etc.

[0021] It will be understood that, unless otherwise stated, certain features of the presently disclosed subject matter that are described in the context of separate embodiments can also be provided in combination in a single embodiment. Conversely, various features of the presently disclosed subject matter that are described in the context of a single embodiment can also be provided separately or in any suitable subcombination. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the methods and apparatus.

[0022] With this in mind, attention is directed to FIG. 1, which illustrates a functional block diagram of an inspection system in accordance with certain embodiments of the disclosed subject matter.

[0023] The inspection system 100 shown in FIG. 1 can be used for inspection of semiconductor specimens (e.g., wafers and / or portions thereof) as part of a specimen manufacturing process. The illustrated inspection system 100 comprises a computer-based system 101 that can automatically determine metrology-related and / or defect-related information using images acquired during specimen manufacturing (hereinafter referred to as manufacturing process (FP) images). According to certain embodiments of the subject matter of the present disclosure, the system 101 can be configured to generate training data for training a deep neural network (DNN) that can be used for inspection of semiconductor specimens. Accordingly, the system 101 is also referred to in this disclosure as a training system or a training data generation system. The system 101 can be operatively connected to one or more inspection tools 120. The inspection tools 120 are configured to capture FP images and / or to review the captured FP images and / or to enable or provide measurements related to the captured images.

[0024] By way of example, the FP images can be selected from images of the specimen (e.g., a wafer or portion thereof) captured during the manufacturing process, derivatives of captured images obtained by various pre-processing stages (e.g., images of a portion of a wafer or photomask captured by a scanning electron microscope (SEM) or optical inspection system, SEM images roughly centered on the defect classified by ADC, SEM images of a larger area where the defect is located by ADR, registered images of different inspection modalities corresponding to the same mask location, segmented images, height map images, etc.), and computer-generated images of a design database. It is noted that in some cases, the images can include image data (e.g., captured image, processed image, etc.) and associated numerical data (e.g., metadata, handcrafted attributes, etc.). It is further noted that the image data can include data related to the layer of interest and / or one or more other layers of the specimen.

[0025] The term "inspection tool" as used herein should be broadly construed to encompass any tool that can be used in an inspection-related process, including, as non-limiting examples, imaging, scanning (in single or multiple scans), sampling, reviewing, measuring, sorting, and / or other processes provided on a specimen or a portion thereof. One or more inspection tools 120 can include one or more inspection tools and / or one or more review tools. In some cases, at least one of the inspection tools 120 can be an inspection tool configured to scan a specimen (e.g., an entire wafer, an entire die, or a portion thereof) to capture inspection images (typically at a relatively high speed and / or low resolution) to detect potential defects. In some cases, at least one of the inspection tools 120 can be a review tool configured to capture review images of at least some of the defects detected by the inspection tool to verify whether the potential defects are indeed defects. Such review tools are typically configured to inspect pieces of a die one at a time (typically at a relatively low speed and / or high resolution). The inspection tool and review tool can be different tools located at the same or different locations, or a single tool operating in two different modes. In some cases, at least one inspection tool has metrology capability and can be configured to perform metrology measurements on the FP images.

[0026] It should also be noted, without limiting the scope of this disclosure in any way, that inspection tool 120 can be implemented as various types of inspection machines, such as optical imagers, electron beam inspection machines, etc. In some cases, the same inspection tool can provide low-resolution and high-resolution image data.

[0027] The system 101 includes a processor and memory circuit (PMC) 102 operatively connected to a hardware-based I / O interface 126. The PMC 102 is configured to provide the processing necessary to operate the system, as will be described in further detail with reference to Figures 2A and 2B, and includes a processor (not shown separately) and a memory (not shown separately). The processor of the PMC 102 can be configured to execute a number of functional modules according to computer-readable instructions embodied in a non-transitory computer-readable memory included in the PMC. Such functional modules are hereinafter referred to as being included in the PMC.

[0028] According to a particular embodiment, functional modules included in the PMC 102 may include a feature extractor 104, a training module 106, and a machine learning model 108. The PMC 102 may be configured to obtain, via the I / O interface 126, a first training image representing at least a portion of a semiconductor sample, and a first label associated with a selected group of pixels in each of one or more segments identified by a user from the first training image. The feature extractor 104 may be configured to extract a set of features characterizing the first training image. Each feature in the set has a feature value corresponding to a pixel in the first training image. The set of features includes a first feature that provides information of a contextual relationship between one or more segments in the first training image, and a second feature that provides information of a distribution of pixels in the first training image with respect to a statistical measure of the pixels in each segment. The training module 106 may be configured to train the machine learning model 108 using the first label, the value of the selected group of pixels in each segment associated with the first label, and the feature value of each feature of the set of features corresponding to the selected group of pixels in each segment. The trained machine learning model 108 can be used to process the first training image to obtain a first segmentation map that provides information of a predicted label associated with each pixel in the first training image, each predicted label indicating the segment to which the respective pixel belongs. The training module 106 can be further configured to include the first training sample including the first training image and the first segmentation map in training data (i.e., training data for training a deep neural network (DNN), also referred to herein as DNN training data) if the criterion is met, and to determine to repeat the extraction, training, and processing of the second feature if the criterion is not met. Details of the training data generation process are described below with reference to Figures 2A and 2B.

[0029] In certain embodiments, the DNN training data, such as that generated by the training system 101, can be used to train a DNN 110 that can be used to inspect a semiconductor specimen. In some cases, the DNN can be included in the PMC 102 of the system 101 or can be operatively connected to the system 101 as shown in FIG. 1. The DNN 110 can be configured to enable data processing using a deep neural network to output application-related data based on manufacturing input data. In one embodiment, the specific application can be image segmentation, and the application-related output can be a segmentation map corresponding to the input FP image. The operation of the system 101, the PMC 102, and the functional modules therein are further detailed with reference to FIGS. 2A and 2B.

[0030] The DNN 110 may include supervised or unsupervised DNN models including layers organized according to respective DNN architectures. As non-limiting examples, the layers of a DNN may be organized according to a convolutional neural network (CNN) architecture, a recurrent neural network architecture, a recursive neural network architecture, a generative adversarial network (GAN) architecture, etc. Optionally, at least some of the layers may be organized into multiple DNN sub-networks. Each layer of a DNN may include multiple fundamental computational elements (CEs), typically referred to in the art as dimensions, neurons, or nodes.

[0031] In general, a computational element in a given layer may be connected with a CE in a preceding layer and / or a subsequent layer. Each connection between a CE in a preceding layer and a CE in a subsequent layer is associated with a weighting value. A given CE may receive inputs from a CE in a preceding layer via a respective connection, and each given connection is associated with a weighting value that may be applied to the input of the given connection. The weighting value may determine the relative strength of the connection and thus the relative influence of each input on the output of the given CE. A given CE may be configured to calculate an activation value (e.g., a weighted sum of the inputs) and further derive an output by applying an activation function to the calculated activation. The activation function may be, for example, an identity function, a deterministic function (e.g., linear, sigmoid, threshold, etc.), a probability function, or other suitable function. The output from a given CE may be communicated to a CE in a subsequent layer via a respective connection. Similarly, as described above, each connection at the output of a CE may be associated with a weighting value that may be applied to the output of the CE before it is received as an input of a CE in a subsequent layer. In addition to the weighting values, there may be thresholds (including limiting functions) associated with the connections and CEs.

[0032] The weighting values ​​and / or thresholds of the deep neural network may be initially selected prior to training and may be further iteratively adjusted or modified during training to achieve an optimal set of weighting values ​​and / or thresholds in the trained DNN. After each iteration, the difference between the actual output generated by the DNN module and the target output associated with the respective training set of data may be determined. This difference may be referred to as an error value. Training may be determined to be complete when a loss / cost function indicating the error value is below a predetermined value or when limited change in performance between iterations has been achieved.

[0033] The set of DNN input data used to tune the weights / thresholds of the deep neural network is hereafter referred to as DNN training data. As mentioned above, the system 101 is configured to generate such DNN training data that can be used to train a DNN.

[0034] It should be noted that the teachings of the subject matter of this disclosure are not bound by the particular architecture of the DNN as described above.

[0035] In some embodiments, the DNN may be a segmentation DNN configured to perform image segmentation on the FP image. In some cases, in addition to the DNN 110, the inspection system 100 may include one or more inspection modules, such as, for example, a defect detection module, and / or an automatic defect review module (ADR), and / or an automatic defect classification module (ADC), and / or a metrology-related module, and / or other inspection modules usable for inspecting semiconductor specimens. The one or more inspection modules may be implemented as standalone computers or may have their functionality (or at least a portion thereof) integrated with the inspection tool 120. In some cases, the DNN 110 may be included in one or more inspection modules. Optionally, the DNN 110 may be shared between the inspection modules, or each of the one or more inspection modules may include its own DNN 110.

[0036] According to certain embodiments, system 101 may include a storage unit 122. Storage unit 122 may be configured to store any data required to operate system 101, such as data related to inputs and outputs of system 101, as well as intermediate processing results generated by system 101. By way of example, storage unit 122 may be configured to store images and / or derivatives thereof generated by inspection tool 120. Thus, one or more images may be retrieved from storage unit 122 and provided to PMC 102 for further processing.

[0037] In some embodiments, the system 101 may optionally include a computer-based graphical user interface (GUI) 124 configured to enable user-specified inputs associated with the system 101. For example, a user may be presented with a visual representation of the sample (e.g., by a display forming part of the GUI 124), including image data of the sample. The user may be provided with options to define certain operational parameters via the GUI. For example, the user may provide label data by manually annotating the image. The user may also view the results of the operation, such as, for example, a segmentation map, on the GUI.

[0038] As described in further detail with reference to FIGS. 2A and 2B, the system 101 is configured to receive FP input data via the I / O interface 126. The FP input data may include data generated by the inspection tool 120 (and / or derivatives thereof and / or metadata associated therewith) and / or data stored in one or more data repositories. It is noted that in some cases, the FP input data may include image data (e.g., captured images, images derived from the captured images, simulated images, synthetic images, etc.) and associated numerical data (e.g., metadata, handcrafted attributes, etc.). It is further noted that the image data may include data related to a layer of interest and / or one or more other layers of the specimen. In some embodiments of the present disclosure, for training purposes, the FP input data may include one or more training images for the purpose of generating training data usable for training the DNN.

[0039] The system 101 is further configured to process the received FP input data and send the results (or a portion thereof) via the I / O interface 126 to the storage unit 122, and / or the DNN 110, and / or the GUI 124 (for rendering the results).

[0040] Those skilled in the art will readily appreciate that the teachings of the subject matter of this disclosure are not bound by the system shown in FIG. 1 and that equivalent and / or modified functionality may be integrated or divided in other manners and may be implemented in any suitable combination of software, firmware and / or hardware.

[0041] It is noted that the inspection system shown in FIG. 1 may be implemented in a distributed computing environment, where the aforementioned functional modules shown in FIG. 1 may be distributed across several local and / or remote devices and linked via a communication network. It is further noted that in other embodiments, at least some of the inspection tools 120, storage unit 122, and / or GUI 124 may be external to the inspection system 100 and operate in data communication with the system 101 via the I / O interface 126. The system 101 may be implemented as a stand-alone computer used in conjunction with the inspection tools. Alternatively, each of the functions of the system 101 may be integrated, at least in part, with one or more inspection tools 120, thereby facilitating and enhancing the functionality of the inspection tools 120 in inspection-related processes.

[0042] For purposes of example only, the following description is provided for generating DNN training data for training a DNN that can be used to inspect a semiconductor specimen. Those skilled in the art will readily appreciate that the teachings of the presently disclosed subject matter are applicable to various types of machine learning models and various inspections, such as, for example, segmentation, defect detection, ADR, ADC, auto-navigation modules, metrology-related modules, and the like.

[0043] Referring to FIG. 2A, a generalized flow diagram for generating training data for training a DNN that can be used for inspection of a semiconductor specimen is shown in accordance with certain embodiments of the disclosed subject matter.

[0044] According to certain embodiments, the training data used to train the DNN (also referred to as DNN training data) may include a number of training samples, each including a respective training image and corresponding ground truth data associated therewith. The ground truth data may include label data indicative of application-specific information. As an example, for an image segmentation application, each training sample may include a training image of a semiconductor sample and label data indicative of one or more segments within the training image.

[0045] The training images may be "real-world" images of a semiconductor sample obtained during a manufacturing process of the semiconductor sample (e.g., FP images as described above). As a non-limiting example, the images may be inspection images obtained by inspecting the sample (typically at a relatively high speed and / or relatively low resolution) using one or more inspection tools to detect potential defects. Such inspection tools may be, for example, optical inspection systems, low-resolution SEMs, etc. Alternatively, the images may be review images obtained by inspecting the sample at a subset of potential defect locations (typically at a relatively low speed and / or relatively high resolution) using one or more review tools to confirm whether the potential defects detected by the inspection tools are in fact defects. Such review tools may be, for example, scanning electron microscopes (SEMs), etc.

[0046] Ground truth data can be obtained in a variety of ways. By way of example, ground truth data can be generated by human annotation, synthetically generated (e.g., CAD-based images), by machine learning annotation (e.g., labels based on feature extraction and analysis), or a combination of the above, etc.

[0047] According to certain embodiments, the DNN is trained for the specific application of image segmentation on semiconductor sample images (e.g., FP images). As used herein, the term "segmentation" can refer to any process that divides an image into meaningful parts / segments (e.g., background and foreground, noisy and noiseless regions, structural elements, defects and non-defects, etc.) while providing pixel-by-pixel or region-by-region values ​​indicative of such segments. DNN training data, such as that generated in accordance with the subject matter of the present disclosure, is used to train a DNN that can be used for segmenting FP images.

[0048] By way of example, the training images can be SEM images or derivatives thereof, and the label data can be a segmentation map (also called a segmentation mask) corresponding to the SEM image. The segmentation map provides information of predicted labels associated with corresponding pixels in the training images. Each predicted label indicates the segment in the image to which the respective pixel belongs.

[0049] Segmentation maps are traditionally generated by users who manually annotate the entire training image. The manual annotation process is very time-consuming given the size and resolution of the image, and the annotated results are sometimes inaccurate and prone to errors. This is undesirable for users because the quality of the label data directly affects the training process and the performance of the trained DNN. Therefore, the present disclosure proposes a more efficient way to automatically generate accurate label data that can be used to train a DNN, as described below with reference to Figures 2A and 2B.

[0050] According to certain embodiments, first training images representative of at least a portion of a semiconductor sample may be acquired (202) (e.g., by PMC 102 via I / O interface 126) along with first labels respectively associated with selected groups of pixels in each of one or more segments identified by a user from the first training image. In some cases, at least one of the one or more segments corresponds to one or more structural elements presented in the first training image.

[0051] Turning now to FIG. 5, an example of a training image is shown according to certain embodiments of the subject matter of this disclosure. Training image 502 is illustrated as an SEM image representing a portion of a die of a wafer, captured by an SEM review tool. As shown, a number of structural elements 504 (shown as polygons representing contact elements on the wafer) are presented in the image. A structural element, as used herein, may refer to any original object on the image data that has a geometric structure with a geometric shape or contour, possibly combined with other objects. The structural elements may be presented, for example, in the form of polygons.

[0052] In some embodiments, the training images may include multiple channels captured from different perspectives. For example, in the example of FIG. 5, 502 may represent one channel of an image taken from a vertical orientation by a top detector of the inspection tool, and 506 may represent another channel of an image taken from a side orientation by a side detector of the inspection tool. In some cases, there may be two or more side detectors from different angles, and accordingly, the training images may include multiple side channel images 506. In some cases, the multiple side channel images may be combined into one combined side channel image.

[0053] To save the user from the task of complete annotation of the entire image, partial label data associated with the training image may be provided by the user. As an example, the user may determine one or more segments that the user wishes to identify in the segmentation map, select pixels from each of the segments, and annotate the pixels to indicate the segment to which they belong. Figure 6 illustrates an example of partial user annotation for a training image according to certain embodiments of the subject matter of the present disclosure.

[0054] A training image 602 is illustrated in FIG. 6. The training image 602 captures the same portion of the wafer as the training images 502 and 506 in FIG. 5, and includes multiple structural elements that share the same design pattern. Such structural elements are also referred to as equivalent structural elements. As an example, a user may determine that the structural element may belong to one segment and the background may belong to another segment. Thus, the user may select a few pixels from either segment and mark them to indicate their respective segment. As an example, as shown in the circled area 604 in FIG. 6 (a close-up of 604 is shown on the right), pixels 606 inside the structural element are marked by the user with a particular gray level value indicating that they belong to one segment representing the structural element, and another set of pixels 608 in the background are marked by the user with another gray level value indicating that they belong to the other segment representing the background region. As described below with respect to block 206, the user annotations for the selected pixels in each segment can be used as partial label data for training the machine learning model.

[0055] It should be noted that the user can determine different types of segmentation in the training image that the user wants to include in the segmentation map. As an example, in some cases, the training image can include two or more families of equivalent structural elements. The user can determine that each family of equivalent structural elements belongs to a respective segment, or the user can determine that all structural elements belong to one segment, but the background belongs to another segment. As another example, the user may determine the segments according to the distribution of gray levels in the image. For example, there may be a gray level contrast within one structural element due to a particular structural profile, and the user may determine that there are two segments within each structural element corresponding to the separation of gray levels. The present disclosure is not limited by the particular manner of segmentation and the number of segments for the training image.

[0056] Continuing with FIG. 2A, a set of features characterizing the first training image may be extracted (204) (e.g., by feature extractor 104). Each feature has a feature value corresponding to a pixel in the first training image. The set of features may include a first feature indicative of contextual information or expression in the first training image. For example, the first feature may provide information of a contextual relationship / association between one or more segments in the first training image. The set of features may further include a second feature providing information of a distribution of pixels in the first training image relative to a statistical measure of pixels in each segment.

[0057] The first feature is also referred to herein as a context feature. The term context feature should be broadly interpreted to represent the context of a particular item / element in an image, such as a pixel, a structural element, or a segment (e.g., surrounding regions with possibly different extents), and / or the global contextual relationship between such elements or segments in an image. The contextual relationship can be reflected in terms of the relative texture, spatial layout, intensity, etc. between elements or segments. For example, pixels inside a structural element share a similar texture with their surrounding pixels (surrounding but still inside the structural element), whereas the texture of pixels outside the structural element is significantly different from pixels inside the structural element, indicating that pixels inside the structural element and pixels outside the structural element may belong to different segments. In some embodiments, the first feature can include a texture feature. In some cases, the first feature can further include an intensity feature.

[0058] Texture features can be used to represent the perceived texture of an image. For example, such features can represent the spatial arrangement of structures or patterns that characterize different elements or segments in an image. Turning now to FIG. 3A, a generalized flow diagram for extracting texture features from a first training image is shown, according to certain embodiments of the subject matter of this disclosure.

[0059] A first training image may be processed (302) with a set of filters (also referred to as a filter bank) to produce a set of feature responses corresponding to the set of filters. Each feature response includes a feature response value corresponding to a respective pixel of the first training image. Each pixel corresponds to a feature vector including a set of feature response values. The feature vectors corresponding to pixels in the image may be clustered (304) into a number of clusters. A feature value may be assigned (306) to each pixel according to the cluster to which the feature vector belongs, producing a feature map including the feature values ​​corresponding to the pixels of the first training image. The feature map may be separated into a number of channels corresponding to the respective feature values. Each channel of the feature map may be sampled (308) using a sampling filter to produce a feature map having a number of sampled channels.

[0060] In some embodiments, texture features can be implemented as texton features. The term texton generally refers to the underlying microstructure in an image. Textons can provide a compact representation for a range of different appearances of objects / elements and can be effective in classifying textures as well as generic object classes.

[0061] As an example, a first training image can be convolved with a set of filters, e.g., a multidimensional filter bank, to produce a multidimensional feature response for every pixel in the training image. The set of filters can be selected to extract features representative of the semiconductor image. For example, the set of filters can include various filter families, e.g., differential filters, Gabor filters, Laplacian filters, etc. Differential filters can provide a quantitative measure of the rate of change of the luminance information of pixels present in the image. By convolution with the image, the differential filters can generate derivatives of various orientations and scales depending on the choice of kernel. The information obtained regarding the rate of luminance change can be used to enhance contrast, detect edges and boundaries, and measure the orientation of features. Gabor filters are linear filters that analyze whether there is any particular frequency content in a particular orientation in the image in a local area around a point or region of interest. The frequency and orientation representation of Gabor filters is similar to that of the human visual system. A set of Gabor filters with different frequencies and orientations can be used to extract features representative of texture from the first training image. Laplacian filters are also called Laplacian of Gaussian (LoG) filters. It is a measure of the second spatial derivative of an image. The Laplacian of an image highlights areas where the intensity changes abruptly and can be used for edge detection. For example, in areas of an image where the intensity is constant (i.e., areas where the intensity gradient is zero), the LoG response is zero. However, in neighborhoods where the intensity changes, the LoG response is positive on the darker sides and negative on the lighter sides.

[0062] In some embodiments of the present disclosure, three families of filters can be used in combination, and the present disclosure proves that the combined filter set is more reliable in extracting features representing the texture and layout of structures in semiconductor images, which are typically noisy images with low signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR). However, this is in no way intended to limit the present disclosure. Other families / types of filters or filters with similar functionality can be used in addition to or instead of the above. In particular, it is noted that the specific filters in each family as described above are shown for illustrative purposes. Other suitable filters within the family can be used in addition to or instead of the above. The present disclosure is not limited by the specific types of filters and the number of filters used in each family.

[0063] As an example, assuming that the training images are in M*N dimensions and the set of filters includes F filters, i.e., an F-dimensional filter bank, a multidimensional feature response is generated, for example, in the form of a feature matrix having dimensions M*N*F. That is, for each pixel in the training images, there are F feature responses that form a feature vector corresponding to that pixel. The multidimensional feature response, i.e., a feature matrix of M*N*F, can be considered as including M*N feature vectors corresponding to M*N pixels.

[0064] Once the multidimensional feature responses for all pixels in the training image are obtained, clustering (e.g., unsupervised clustering) can be performed on the multidimensional feature responses. As an example, a Euclidean distance K-means clustering algorithm can be performed on the M*N feature vectors to produce multiple clusters (e.g., K clusters) of feature vectors. Each pixel in the training image is clustered into one of the K clusters and assigned a feature value corresponding to a cluster number, producing a feature map (e.g., texton map) that includes feature values ​​corresponding to pixels in the first training image. The texton map T includes the feature values, where for each pixel i, the corresponding feature value is Ti ∈ {1,...,K}. The feature map (e.g., texton map) can be separated into K channels corresponding to the respective feature values ​​(e.g., one for each texton), where for each channel, a separate channel map (e.g., integral map) can be derived, similar to a histogram with one bin for each texton. The separated channel map is in a dimension of M*N*K.

[0065] Sampling can be performed for each channel, for example, by using a sampling filter. For example, the K channel maps can each be convolved with a sampling filter to produce a sampled channel map. By way of example, the sampling filter can be implemented using averaging, Gaussian, or other types of sampling methods. When convolved with the channel map, the contextual situation of each pixel (e.g., the area surrounding each pixel) can be effectively reflected in the resulting convolution value in the sampled channel map, thereby adding sensitivity to the sampled map. The K sampled channel maps (e.g., in the form of a feature matrix with M*N*K dimensions) can be used as texture features corresponding to the first training image.

[0066] According to certain embodiments, the first features may further include intensity features in addition to texture features. The intensity features may represent a spatial arrangement of pixel intensities that characterize various elements or segments in the image. In some embodiments, the intensity features may also be implemented as texton features. As an example, clustering as described above may be performed on pixel values ​​of the first training image. Similarly, each pixel in the training image may be clustered into one of a number of clusters and assigned a feature value corresponding to the cluster to which the pixel value belongs, resulting in a feature map that may be separated into a number of channels. Sampling may be performed for each channel, for example, by using a sampling filter, and the sampled channel map may be used as an intensity feature corresponding to the first training image.

[0067] 7, an example of texture features and an example of intensity features are shown, in accordance with certain embodiments of the subject matter of this disclosure. As shown, feature map 702 is texture features extracted from a first training image 502, and feature map 704 is intensity features extracted from the same image.

[0068] In addition to the first feature, the set of features may further include a second feature that provides information of pixel distribution in the first training image with respect to a statistical measure of the group of pixels in each segment. The second feature is also referred to as a statistical feature. The statistical feature referred to herein may represent a quantitative measure according to the statistical significance of a pixel, the quantitative measure indicating the possibility that the pixel belongs to a particular segment. Referring to FIG. 3B, a generalized flow diagram of extracting statistical features from the first training image according to certain embodiments of the subject matter of the present disclosure is shown.

[0069] A statistical measure may be calculated for selected pixels in each segment (312). As an example, the statistical measure may be a mean value calculated based on the values ​​of the pixels in each segment. As another example, the statistical measure may be calculated based on the mean value and a standard deviation (STD), which is a measure of the amount of variation or variance of the values ​​of the pixels relative to the mean value. For each pixel in the first training image, a corresponding feature value may be calculated (314) as the distance between the value of the pixel and the statistical measure of each segment, resulting in one or more feature maps, each including a feature value corresponding to the distance between the pixel value and the statistical measure of the respective segment. For example, in the example of FIG. 6, a statistical measure may be calculated for pixels 606 in the segment representing the structural elements and for pixels 608 in the other segment representing the background region. Two feature maps may be generated for the first training image, a first feature map including distance values ​​between each pixel value in image 602 and the statistical measure calculated for 606, and a second feature map including distance values ​​between each pixel value in image 602 and the statistical measure calculated for 608. The feature map thus generated shows the distribution / variance of pixel values ​​in the image, relative to a statistical measure calculated for each segment (based on the pixels marked by the user), which indicates how likely it is that a pixel belongs to a particular segment.

[0070] 2A , once a first training image and a first label have been obtained as described with reference to block 202, and a set of features have been obtained as described with reference to block 204, a machine learning (ML) model may be trained (206) (e.g., by training module 106) using the first label, values ​​of selected pixels in each segment associated with the first label, and feature values ​​of each feature in the set of features corresponding to pixels in each segment. The ML model is trained for image segmentation.

[0071] In some embodiments, the ML model may be implemented as a classifier. The terms "classifier", "classifier model", or "classification model" referred to herein should be broadly construed to encompass any learning model that can identify which of a set of categories / classes a new instance belongs to based on a training set of data. By way of example, in this disclosure, a classifier may be trained to classify pixel candidates into a set of segment classes as defined by a user. For example, in the example of FIG. 6, the classifier is trained based on a training set that includes a first label indicating two segments (e.g., a user annotation for pixel group 606 representing one segment and a user annotation for pixel group 608 representing the other segment), pixel values ​​of pixel group 606 and pixel group 608 in a first training image, and feature values ​​(for each feature of the set of features) corresponding to pixel group 606 and pixel group 608. For example, assuming the set of features includes a texture feature such as illustrated at 702, an intensity feature such as illustrated at 704, and a statistical feature such as described above with reference to FIG. 3B, then the feature values ​​for each of the three features (which may be in the form of a feature map or matrix) corresponding to selected pixel groups 606 and 608 are used as part of the training set.

[0072] The trained classifier can be used for image segmentation, i.e., to provide each pixel in an image with a predicted label indicating which segment it belongs to. It should be noted that the classifier can be implemented as various types of machine learning models, such as, for example, a linear classifier, a support vector machine (SVM), a neural network, a decision tree, etc., and the present disclosure is not limited by the particular model implemented therein.

[0073] The first training image may be processed (208) using the trained ML model to obtain a first segmentation map that provides information of a predicted label associated with each pixel in the first training image. Each predicted label indicates the segment to which the respective pixel belongs. According to certain embodiments, the trained ML model (e.g., a classifier) ​​may be used to process at least the remaining pixels in the first training image (e.g., excluding the pixels selected in each segment and used to train the classifier). The output of the classifier is a segmentation map (also called a segmentation mask) of the same size as the training image and containing the predicted labels for the corresponding pixels in the image.

[0074] If the criteria are met, it is determined (e.g., by the training module 106) to include the first training sample including the first training image and the first segmentation map in the DNN training data, and if the criteria are not met, to repeat the extraction, training, and processing of the second feature (210). In other words, whether to include the first training sample in the DNN training data or to repeat the processing can be determined based on the criteria as described above. In some embodiments, the criteria can be related to, for example, feedback by a user on the first segmentation map. As an example, the criteria can be based on user feedback. For example, the criteria can be positive user feedback on the first segmentation map, e.g., the first segmentation map has a satisfactory quality. Thus, upon receiving positive user feedback on the first segmentation map, the first training sample can be included in the DNN training data.

[0075] On the other hand, if negative user feedback is received for the first segmentation map, e.g., if the quality of the first segmentation map is not satisfactory, this means that the ML model as currently trained has not learned to segment the first training images as well as the user expected, and therefore at least some of the operations described above with reference to Figure 2A need to be repeated. Figure 8 illustrates an example of a segmentation map with negative user feedback, according to certain embodiments of the subject matter of the present disclosure.

[0076] A segmentation map 800 has been generated corresponding to a training image as illustrated in FIG. 5 and FIG. 6. In this example, the segmentation map is illustrated as a binary map representing two segments, a first segment corresponding to a structural element in the image (shown as an open circle representing a contact element) and a second segment corresponding to a background region (shown as a black region). As shown, for region 802, which is a peripheral background region along the border of the image, it should have been marked in black, but is erroneously marked in white. Thus, region 802 is erroneously predicted as belonging to the first segment of the structural element. A similar error occurs for region 804, where the background region between two structural elements is erroneously predicted as belonging to the first segment (shown in the figure as two elements connecting with each other, and thus missing the background region between them). Such a segmentation map is not satisfactory due to the incorrectly segmented regions according to user reviews. Thus, the classifier needs to be retrained until it produces an output segmentation map with a satisfactory quality. Note that in some cases, the satisfactory quality of the segmentation map may be determined according to a quality condition, which may, for example, be a range or a threshold for the percentage of pixels having correct segmentation.

[0077] Turning now to FIG. 2B, there is shown a generalized flow diagram of a continuing process of generating DNN training data when negative user feedback is received for a first segmentation map, in accordance with certain embodiments of the subject matter of this disclosure.

[0078] Specifically, additional first labels associated with additional groups of pixels in at least one of the segments may be obtained from the user (212). In such a case, the additional first labels may be added as part of the label data along with the first labels as previously provided by the user. The first labels and the additional first labels constitute aggregated label data. Referring to FIG. 9A, an example of an additional first label provided by a user is shown, according to certain embodiments of the subject matter of the present disclosure.

[0079] As shown in the first training image 602 in FIG. 9A, in addition to the original first label (as shown in the circled region 604), additional pixels (as shown in the circled regions 902 and 904) have been marked by the user with labels indicating the expected segments to which these pixels belong. For example, the user can specifically mark the regions (e.g., in regions 802 and 804) where there was an incorrect segmentation in the first segmentation map. The user can choose whether to mark the additional pixels in one segment or in both segments. For example, in 902, the user marked pixels in both the structural element and the background, while in 904, the user marked pixels only in the background. Thus, after the additional marking, the aggregated label data includes the labels marked for the pixels in regions 604, 902, and 904.

[0080] Upon obtaining the aggregated label data, the extraction, training, and processing of the second features can be repeated based on the aggregated label data until a criterion is met, e.g., positive user feedback is received. Specifically, as the label data is updated, the second features, i.e., statistical features, calculated based on the pixels in each segment associated with the label should also be updated based on the aggregated label data (214). For example, the second features can be re-extracted using statistical measures calculated based on pixel values ​​of the pixels and additional pixels.

[0081] The classifier may be retrained using the aggregated labels, values ​​of pixels associated with the aggregated label data, and feature values ​​of each feature of the set of features corresponding to pixels associated with the aggregated label data (216). The first training images may be processed again using the retrained classifier (218) to obtain an updated first segmentation map. Then, based on user feedback on the updated first segmentation map, it may be determined whether to repeat the above process (operations referenced in blocks 212-218) or to include the updated first segmentation map and the first training images as first training samples in the DNN training data (220).

[0082] Referring to FIG. 9B, an example of an updated segmentation map with positive user feedback is shown, in accordance with certain embodiments of the subject matter of this disclosure.

[0083] The segmentation map 900 is generated according to the operations described with reference to FIG. 2B upon obtaining additional first labels as shown in FIG. 9A. This is an updated segmentation map of the first segmentation map 800 as shown in FIG. 8. As shown, the updated segmentation map 900 provides a satisfactory segmentation result of two segments in contrast to the segmentation map 800. The segmentation in the previous incorrectly segmented regions 802 and 804 has been corrected by the additional first labels provided by the user and retraining of the classifier. The updated segmentation map, together with the corresponding training images, is included in the DNN training data as a training sample.

[0084] According to certain embodiments, the process as shown in FIG. 2A and FIG. 2B can be repeated for one or more training images, thus generating one or more segmentation maps that can be included in the DNN training data along with the training images. As an example, a second training image can be obtained with a second label associated with a selected group of pixels in each of the one or more segments identified by the user from the second training image. The second label is added to the aggregated label data. A set of features characterizing the second training image, including a first feature and a second feature, can be extracted. A machine learning model can be trained using the aggregated label data, values ​​of pixels associated with the aggregated label data, and feature values ​​of each feature of the set of features corresponding to pixels associated with the aggregated label data. Processing and determination as described above with reference to blocks 208 and 210 can be performed based on the second training image.

[0085] Turning now to FIG. 4, a generalized flow diagram of using the generated DNN training data to train a DNN is shown, in accordance with certain embodiments of the subject matter of this disclosure.

[0086] According to a particular embodiment, the DNN is trained for the specific application of image segmentation on semiconductor sample images (e.g., FP images). As an example, such segmentation can be used by a metrology tool to perform measurements on the sample. As another example, such segmentation can also be used for ADC (e.g., to define whether a defect is on the main pattern, on the background, or both) in constructing attributes, ADR to apply segment-specific detection thresholds to each segment, etc.

[0087] The process includes a setup stage 410 of segmentation-specific training of a DNN, and a runtime stage 420 of using the trained DNN for image segmentation.

[0088] During the setup 410, DNN training data may be obtained (412), such as generated according to the description of FIG. 2A and FIG. 2B. The DNN training data includes one or more training samples, each sample includes a training image and a corresponding segmentation map. The DNN training data may be used to train a DNN (414), thereby obtaining a trained DNN characterized by segmentation-related training parameters. In some embodiments, the training process may be cyclical and may be repeated several times until the DNN is sufficiently trained. As an example, the DNN may be trained using a cost function related to the accuracy of the segmentation (e.g., correct segmentation label map vs. predicted label map). The trained DNN may be validated (416) using a validation set of images. The validation set of images may be a different image set from the training image set and may include FP images selected for validation purposes. A user may provide feedback on the results reached by the DNN during validation.

[0089] During runtime 420, the PMC uses the trained DNN to process one or more runtime images, including one or more captured FP images to be segmented (422), to provide a segmentation map for each image (424). The one or more FP images can be acquired by the same or different examination modalities. The resulting segmentation map can provide information of segmentation labels per pixel or per region that indicate different segments on the image. As an example, polygons in one layer can have one segmentation label, polygons in another layer can have a different segmentation label, while the background can have another segmentation label.

[0090] Among the advantages of certain embodiments of the training data generation process as described herein are the provision of an automated means of generating accurate ground truth data that can be used to train DNNs, compared to traditional manual annotation of entire training images, which is time-consuming and error-prone. The automation of such a process is made possible by at least extracting meaningful features suitable for segmentation and interactively training ML models based on user partial annotations and feedback.

[0091] Among the further advantages of certain embodiments of the training data generation process as described herein is that by extracting and using both contextual and statistical features that characterize the training images, the present disclosure successfully provides a reliable representation of semiconductor images in terms of texture, layout, context, statistical significance, etc., and thus the representative features can be used to train ML models and generate segmentation maps in an efficient manner with satisfactory results.

[0092] It should be noted that the illustrated examples are set forth herein for illustrative purposes and should not be construed as limiting the present disclosure in any way. Other suitable examples may be used in addition to or in place of the above.

[0093] It is to be understood that the disclosure is not limited in its application to the details set forth in the description contained herein or shown in the drawings.

[0094] It will also be appreciated that the system according to the present disclosure may be implemented, at least in part, on a suitably programmed computer. Similarly, the present disclosure contemplates a computer program readable by a computer for carrying out the methods of the present disclosure. The present disclosure further contemplates a non-transitory computer readable memory tangibly embodying a program of instructions executable by a computer for carrying out the methods of the present disclosure.

[0095] The present disclosure is capable of other embodiments and of being practiced and carried out in various ways. It is therefore to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the concepts underlying the present disclosure may be readily utilized as a basis for the designing of other structures, methods and systems for carrying out some of the purposes of the subject matter of the present disclosure.

[0096] Those skilled in the art will readily appreciate that various modifications and variations, as described above, can be applied to the embodiments of the present disclosure without departing from the scope of the present disclosure, which is defined in and by the appended claims.

Claims

1. 1. A computerized system for generating training data for training a deep neural network (DNN training data) usable for inspection of a semiconductor specimen, the system comprising: obtaining a first training image of the semiconductor sample and first labels respectively associated with selected groups of pixels in each of one or more segments identified by a user from the first training image and provided as partial label data of the first training image; extracting a set of features characterizing the first training images, each feature having a feature value corresponding to a pixel in the first training images, the set of features including a first feature indicative of contextual information in the first training images and a second feature providing information of a distribution of pixels in the first training images relative to a statistical measure of the pixels in each segment; training a machine learning (ML) model for image segmentation using the first labels, values ​​of the selected pixels in each segment, and the feature values ​​of each feature of the set of features corresponding to the pixels in each segment; processing the first training images using the trained ML model to obtain a first segmentation map providing information of a predicted label associated with each pixel in the first training images, each predicted label indicating to which segment a respective pixel belongs; if a criterion is met, including a first training sample including the first training image and the first segmentation map in the DNN training data; a processor and memory circuit (PMC) configured to: the first segmentation map is used as the complete label data for the first training image; Computerized systems.

2. the training data is usable to train a deep neural network (DNN) for at least one inspection process selected from the group including automatic segmentation, automatic metrology, automatic defect detection, automatic defect review, and automatic defect classification based on the runtime images; 2. The computerized system of claim 1.

3. the first features represent a context of one or more elements in the first training image and a contextual relationship between one or more elements; 2. The computerized system of claim 1.

4. the criteria being based on user feedback on the first segmentation map, and the PMC is configured, upon receiving negative user feedback on the first segmentation map, to: obtain additional first labels associated with additional groups of pixels in at least one of the segments, the first labels and the additional first labels constituting aggregated label data; and repeat the extraction, training, and processing of the second features based on the aggregated label data until receiving positive user feedback.

2. The computerized system of claim 1.

5. the PMC is configured to include the first training sample in the training data upon receiving positive user feedback on the first segmentation map.

5. The computerized system of claim 4.

6. the PMC is further configured to obtain second training images and second labels respectively associated with selected groups of pixels in each of one or more segments identified by a user from the second training images, the second labels being added to the aggregated label data; characterize the second training images and extract a set of features including the first feature and the second feature; train the ML model using the aggregated label data, values ​​of pixels associated with the aggregated label data, and the feature values ​​of each feature of the set of features corresponding to the pixels associated with the aggregated label data; and perform the processing and decision based on the second training images.

6. The computerized system of claim 5.

7. at least one of the one or more segments corresponds to one or more structural elements presented in the first training image; 2. The computerized system of claim 1.

8. The first feature is, processing the first training images with a set of filters to produce a set of feature responses each including feature response values ​​corresponding to a respective pixel of the first training images, each pixel corresponding to a feature vector including its set of feature response values; clustering the feature vectors corresponding to the pixels in the first training image into a plurality of clusters; assigning a feature value to each pixel according to the cluster to which the feature vector belongs, resulting in a feature map containing feature values ​​corresponding to pixels of the first training images, the feature map being separable into a plurality of channels corresponding to respective feature values; sampling each channel of the feature map using a sampling filter to result in a feature map having a plurality of sampled channels; The texture features extracted by 2. The computerized system of claim 1.

9. the first features further include intensity features extracted by: clustering pixel values ​​of the first training images into a plurality of clusters; assigning a feature value to each pixel according to the cluster to which the pixel value belongs, resulting in a feature map separable into a plurality of channels corresponding to the respective feature values; and sampling each channel of the feature map using a sampling filter, resulting in a feature map having a plurality of sampled channels.

9. The computerized system of claim 8.

10. the set of filters includes one or more of a derivative filter, a Gabor filter, and a Laplacian filter; 9. The computerized system of claim 8.

11. the second features are extracted by computing a statistical measure for the pixels in each segment, and for each pixel in the first training images computing a corresponding feature value as the distance between the value of the pixel and the statistical measure for each segment, resulting in one or more feature maps each containing a feature value corresponding to the distance between a pixel value and the statistical measure for a respective segment.

2. The computerized system of claim 1.

12. The PMC is further configured to train the deep neural network (DNN) using the DNN training data and validate the trained DNN using a validation set of images.

2. The computerized system of claim 1.

13. 1. A computerized method for generating training data for training a deep neural network (DNN training data) usable for inspection of a semiconductor specimen, the method being executed by a processor and memory circuit (PMC), comprising: obtaining a first training image of the semiconductor sample and first labels respectively associated with selected groups of pixels in each of one or more segments identified by a user from the first training image and provided as partial label data of the first training image; extracting a set of features characterizing the first training images, each feature having a feature value corresponding to a pixel in the first training images, the set of features including first features indicative of contextual information in the first training images and second features providing information of a distribution of pixels in the first training images relative to a statistical measure of the pixels in each segment; training a machine learning (ML) model for image segmentation using the first labels, values ​​of the selected pixels in each segment, and the feature values ​​of each feature of the set of features corresponding to the pixels in each segment; - processing the first training images using the trained ML model to obtain a first segmentation map giving information of a predicted label associated with each pixel in the first training images, each predicted label indicating to which segment the respective pixel belongs; If a criterion is met, determining to include a first training sample including the first training image and the first segmentation map in the DNN training data; Including, The first segmentation map is used as complete label data for the first training images.

14. the criteria being based on user feedback on the first segmentation map, and the method further comprising, upon receiving negative user feedback on the first segmentation map, obtaining additional first labels associated with additional groups of pixels in at least one of the segments, the first labels and the additional first labels constituting aggregated label data; and repeating the extraction, training, and processing of the second features based on the aggregated label data until positive user feedback is received.

14. The computerized method of claim 13.

15. and upon receiving positive user feedback on the first segmentation map, including the first training sample in the training data.

15. The computerized method of claim 14.

16. The first feature is, processing the first training images with a set of filters to produce a set of feature responses each including feature response values ​​corresponding to a respective pixel of the first training images, each pixel corresponding to a feature vector including its set of feature response values; clustering the feature vectors corresponding to the pixels in the first training image into a plurality of clusters; assigning a feature value to each pixel according to the cluster to which the feature vector belongs, resulting in a feature map containing feature values ​​corresponding to pixels of the first training images, the feature map being separable into a plurality of channels corresponding to respective feature values; sampling each channel of the feature map using a sampling filter to result in a feature map having a plurality of sampled channels; The texture features extracted by 14. The computerized method of claim 13.

17. the first features further include intensity features extracted by: clustering pixel values ​​of the first training images into a plurality of clusters; assigning a feature value to each pixel according to the cluster to which the pixel value belongs, resulting in a feature map separable into a plurality of channels corresponding to the respective feature values; and sampling each channel of the feature map using a sampling filter, resulting in a feature map having a plurality of sampled channels.

17. The computerized method of claim 16.

18. the second features are extracted by computing a statistical measure for the pixels in each segment, and for each pixel in the first training images computing a corresponding feature value as the distance between the value of the pixel and the statistical measure for each segment, resulting in one or more feature maps each containing a feature value corresponding to the distance between a pixel value and the statistical measure for a respective segment.

14. The computerized method of claim 13.

19. training the deep neural network (DNN) using the DNN training data; and validating the trained DNN using a validation set of images.

14. The computerized method of claim 13.

20. 1. A non-transitory computer readable storage medium tangibly embodying a program of instructions that, when executed by a computer, causes the computer to perform a method for generating training data for training a deep neural network (DNN training data) usable for inspection of a semiconductor specimen, the method comprising: obtaining a first training image of the semiconductor sample and first labels respectively associated with selected groups of pixels in each of one or more segments identified by a user from the first training image and provided as partial label data of the first training image; extracting a set of features characterizing the first training images, each feature having a feature value corresponding to a pixel in the first training images, the set of features including first features indicative of contextual information in the first training images and second features providing information of a distribution of pixels in the first training images relative to a statistical measure of the pixels in each segment; training a machine learning (ML) model for image segmentation using the first labels, values ​​of the selected pixels in each segment, and the feature values ​​of each feature of the set of features corresponding to the pixels in each segment; - processing the first training images using the trained ML model to obtain a first segmentation map giving information of a predicted label associated with each pixel in the first training images, each predicted label indicating to which segment the respective pixel belongs; If a criterion is met, determining to include a first training sample including the first training image and the first segmentation map in the DNN training data; and the first segmentation map is used as complete label data for the first training image. A non-transitory computer-readable storage medium.

Citation Information

Patent Citations

  • Information processing unit, control method, program

    JP2017117139A

  • Method of deep learining-based examination of a semiconductor specimen and system thereof

    US20170357895A1