Classifying defects in semiconductor samples

Abstract

A classifier and method for classifying defects in a semiconductor sample are provided. The method includes: receiving a first group of defects classified as a majority class, while some defects belong to (one or more) minority classes and some defects belong to the majority class; selecting a subset of attributes and defining a discriminator for the attributes so that a second classifier correctly classifies at least a portion of the defects into the minority class and the majority class using the subset of attributes and the discriminator; generating a training set, the training set including: defects of the majority class, defects of the minority class, and additional defects classified as the minority class by the second classifier; training an engine on the training set, the subset, and the discriminator, the engine obtaining a confidence that the defect belongs to the majority class; applying the engine to a second group of defects classified as the majority class to obtain a confidence that each defect is classified as the majority class; and outputting defects with low confidence as belonging to (one or more) minority classes.

Description

Technical Field

[0001] The presently disclosed subject matter relates generally to the field of sample inspection, and more particularly to methods and systems capable of automatically detecting defects to be classified as monitored defect categories. Background Art

[0002] Current demands for high density and performance with ultra-large scale integration of fabricated devices require sub-micron features, increased transistor and circuit speeds, and improved reliability. This demand requires that device features be formed with high precision and high uniformity, which in turn necessitates careful monitoring of the fabrication process, including frequent and detailed inspection of the devices while they are still in semiconductor sample form.

[0003] The term "specimen" used in this specification should be interpreted expansively to include any type of wafers, masks, and other structures, combinations and / or parts thereof used to manufacture semiconductor integrated circuits, magnetic heads, flat panel displays, and other semiconductor products.

[0004] The term "defect" as used herein should be interpreted expansively to encompass any type of abnormal or undesirable feature formed on or within a sample.

[0005] Complex sample manufacturing processes are not free of errors, and the errors may produce errors in the manufactured devices. The errors may include defects that may be detrimental to the operation of the device, as well as nuisances that may be defects but are not detrimental to the manufactured device or cause malfunctions. As a non-limiting example, defects may be produced during the manufacturing process due to errors in the raw materials, mechanical, electrical or optical errors, human errors, or other errors. In addition, defects may be caused by spatiotemporal factors, such as changes in wafer temperature that occur after one or more manufacturing stages in the inspection process, which may cause the wafer to deform. The inspection process may also introduce other so-called errors, such as optical, mechanical or electrical problems in the inspection equipment or process, making the capture less than perfect. Such errors may produce false positive results, which may appear to contain defects, but in fact there are no defects in this area.

[0006] In many applications, the type or category of defects is important. For example, defects can be classified into one of several categories, such as particles, scratches, process, etc.

[0007] The term "majority class" as used herein should be interpreted expansively to include any defect type or class that occurs frequently. For example, a class may be considered the majority class if a predetermined number of defects of a class occur when inspecting a sample, at least a predetermined portion of defects belonging to a class are detected when inspecting a sample, the number of defects belonging to a class encountered is sufficient to train a classifier to identify defects as belonging to the class, or other criteria.

[0008] The term "minority class" as used herein should be interpreted expansively to encompass any defect type or category that does not occur frequently, and therefore the statistics are insufficient to generate a reliable classification model by training a classifier to classify the defect into this category. For example, if less than a predetermined number of defects of a class occur when inspecting samples, less than a predetermined portion of defects belonging to a class are detected when inspecting samples, the number of defects belonging to a class encountered is insufficient to train a classifier to identify the defect as belonging to this class, etc., then this class can be considered a minority class.

[0009] Unless otherwise expressly stated, the term "inspection" used in this specification should be interpreted expansively to cover any kind of detection and / or classification of an object. Inspection is performed by using a non-destructive inspection tool during (or after) the manufacture of the object to be inspected. As a non-limiting example, the inspection process may include scanning (single or multiple scans), sampling, calibration, measurement, classification and / or other operations performed on an object or part of an object using one or more inspection tools. Similarly, inspection may be provided before the manufacture of the object to be inspected, and may, for example, include the generation of (one or more) inspection recipes. It should be noted that, unless otherwise expressly stated, the term "inspection" or its derivatives used in this specification are not limited to the size of the inspected area, the speed or resolution of the scan, or the type of inspection tool. As a non-limiting example, a wide variety of non-destructive inspection tools include optical tools, scanning electron microscopes, atomic force microscopes, and the like.

[0010] The inspection process may include multiple inspection steps. During the manufacturing process, the inspection steps may be performed multiple times, for example after manufacturing or after processing certain layers, etc. Additionally or alternatively, each inspection step may be repeated multiple times, for example for different sample positions, or for the same sample position with different inspection settings.

[0011] As a non-limiting example, a run-time inspection may employ a two-stage procedure, such as inspecting a sample and then verifying the sampled defects. During the inspection step, the surface of the sample or a portion of the sample (e.g., an area of ​​interest, a hot spot, etc.) is scanned, typically at a relatively high speed and / or low resolution. The captured inspection images are analyzed to detect defects and obtain location and other defect inspection attributes. In the verification step, images of at least some of the defects detected in the inspection stage are captured, typically at a relatively low speed and / or high resolution, so as to enable classification and (optional) other analysis of at least some of the defects. In some cases, these two stages may be implemented by the same inspection tool, and in some other cases, the two stages are implemented by different inspection tools. Summary of the invention

[0012] According to some aspects of the presently disclosed subject matter, a system for classifying defects in a sample into multiple categories is provided. The system includes a processor and a memory circuit system (PMC), which is configured to: receive multiple defects classified into majority categories by a first classifier, wherein one or more defects in the multiple defects belong to one or more minority categories, and at least a portion of the defects in the multiple defects belong to majority categories, and wherein each of the multiple defects is characterized by multiple attributes, each attribute having a respective value; select a subset of the multiple attributes, and define a differentiator for each attribute of the subset, so that a second classifier configured to classify defects according to the subset and the differentiator of each corresponding attribute can correctly classify at least a predetermined portion of the multiple defects into one or more minority categories and majority categories; generate a clinical diagnosis result; A temporary training set is provided, the temporary training set comprising: at least a portion of the defects in a plurality of defects belonging to the majority class, at least a portion of the defects in a plurality of defects belonging to the minority class, and additional defects, the additional defects having values ​​of the attribute subset so that the second classifier classifies the additional defects as the minority class; a training engine, the engine being configured to obtain a confidence that a defect characterized by a first value of the attribute subset belongs to the majority class, wherein training is performed on the temporary training set, the attribute subset, and a discriminator of each attribute of the subset; applying the engine to each given defect in a second plurality of defects classified as the majority class to obtain a confidence that the given defect is classified as the majority class; and outputting defects having a confidence below a predetermined threshold from the second plurality of defects, the defects being thereby assumed to belong to one or more minority classes.

[0013] According to further aspects of the presently disclosed subject matter, the PMC can be further configured to generate additional defects that can be classified by the second classifier into one or more minority classes, and include the additional defects in the temporary training set. At least one of the additional defects is a composite defect generated as a plurality of first values, each first value being associated with an attribute from the subset according to a distinguishing term for each attribute of the subset. As a non-limiting example, the first value associated with a given attribute is determined to be a value that is between a value of the given attribute of a first defect belonging to one or more minority classes and a value of the given attribute of a second defect adjacent to the first defect and belonging to the majority class, and is determined to be a value of the given attribute of a defect belonging to one or more minority classes, and so on.

[0014] According to further aspects of the presently disclosed subject matter, the PMC is further configured to: identify at least a portion of the output defects as belonging to one or more minority classes; update the first classifier by training the first classifier on a training set, wherein the training set includes at least some of the multiple defects belonging to one or more minority classes, at least some of the multiple defects belonging to the majority class, and output defects identified as belonging to one or more minority classes; and update multiple defects classified by the first classifier as majority classes, and output defects that occur during the update and for which the engine acquires a confidence level below a threshold.

[0015] According to other aspects of the presently disclosed subject matter, a method for classifying defects is provided, the method being performed by a computing platform, the computing platform including a processor and a memory circuit system (PMC). The method includes: receiving, by the PMC, a plurality of defects classified as majority classes by a first classifier, wherein one or more of the plurality of defects belong to one or more minority classes, and at least a portion of the plurality of defects belong to majority classes, and wherein each of the plurality of defects is characterized by a plurality of attributes, each attribute having a respective value; selecting, by the PMC, a subset of the plurality of attributes, and defining a distinguishing term for each attribute of the subset, so that a second classifier configured to classify defects according to the distinguishing term of the subset and each corresponding attribute can correctly classify at least a predetermined portion of the plurality of defects into one or more minority classes and majority classes; generating, by the PMC, a temporary training set, the temporary training set including: at least a portion of defects, at least a portion of a plurality of defects belonging to a minority class, and additional defects, wherein the additional defects have values ​​of a subset of attributes such that a second classifier classifies the additional defects as belonging to a minority class; a PMC training engine, the engine being configured to obtain a confidence that a defect characterized by a first value of the subset of attributes belongs to a majority class, wherein the training is performed on a temporary training set, the subset of attributes, and a distinguishing term for each attribute of the subset; the PMC applying the engine to each given defect in a second plurality of defects classified as the majority class to obtain a confidence that the given defect is classified as the majority class; and the PMC outputting defects having a confidence below a predetermined threshold from the second plurality of defects, the defects being thereby assumed to belong to one or more minority classes.

[0016] According to further aspects of the presently disclosed subject matter, the method can further include generating additional defects that can be classified into one or more minority classes by the second classifier and including the additional defects into the temporary training set. At least one of the additional defects is a composite defect generated as a plurality of first values, each first value being associated with an attribute from the subset according to a distinguisher for each attribute of the subset.

[0017] According to further aspects of the presently disclosed subject matter, the method may further include: identifying at least a portion of the output defects as belonging to one or more minority classes; updating the first classifier by training the first classifier on a training set, the training set including at least some of the multiple defects belonging to the one or more minority classes, at least some of the multiple defects belonging to the majority class, and the output defects identified as belonging to the one or more minority classes; and updating the multiple defects classified by the first classifier as the majority class, and outputting defects that occur when updating and for which the engine has a confidence level below a threshold for defect acquisition.

[0018] Among the advantages of certain embodiments of the presently disclosed subject matter, monitoring of user-set success criteria or measures related to automated defect classification is enhanced. Some non-limiting examples of success criteria include classification purity of one or more categories, classification accuracy of one or more categories, and extraction.

[0019] One of the further advantages is that the classification results are optimized according to user needs by correcting the effects of an unbalanced training set, by identifying other defects in one or more minority classes, and by correctly classifying defects that could not be classified before utilizing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to understand the invention and to see how it may be carried out in practice, embodiments will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which:

[0021] Figure 1 shows a general block diagram of an inspection system according to certain embodiments of the presently disclosed subject matter;

[0022] Figure 2 shows a general flow chart of a method according to certain embodiments of the presently disclosed subject matter that identifies defects that may belong to a minority class;

[0023] Figure 3 shows a confusion matrix for defect classification according to certain embodiments of the presently disclosed subject matter;

[0024] Figure 4 illustrates exemplary two-dimensional defects upon which additional defects may be created in accordance with certain embodiments of the presently disclosed subject matter; and

[0025] Figure 5 A general flow chart of a sample calibration method according to certain embodiments of the presently disclosed subject matter is shown. DETAILED DESCRIPTION

[0026] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the subject matter disclosed herein may be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits are not described in detail to avoid obscuring the subject matter disclosed herein.

[0027] Unless otherwise specifically noted, it will be apparent from the following discussion that it should be understood that throughout the specification discussion, terms such as "processing," "computing," "representing," "calculating," "generating," "distributing," "selecting," "determining," "receiving," "training," "applying," "outputting," and the like are used to refer to the actions and / or processing of a computer that manipulates and / or transforms data into other data, the data being represented by a physical form, such as electronics representing physical objects, quantities, and / or the data. The term "computer" should be expansively interpreted to cover any kind of hardware-based electronic device with data processing capabilities, including (as a non-limiting example) the presently disclosed classifier and the PMC therein.

[0028] The terms "non-transitory memory" and "non-transitory storage medium" as used herein should be interpreted expansively to cover any volatile or non-volatile computer memory suitable for use with the presently disclosed subject matter.

[0029] It should be understood that, unless otherwise stated, some features of the presently disclosed subject matter described in the context of multiple independent embodiments may be provided in combination in a single embodiment. Conversely, various features of the presently disclosed subject matter described in the context of a single embodiment may also be provided individually or in any suitable subcombination. In the following embodiments, several specific details are set forth in order to provide a thorough understanding of the methods and apparatus.

[0030] As used herein, the term "defect" in a sample should be interpreted expansively to encompass any type of abnormal or undesirable feature formed on or within the sample.

[0031] The term "known to belong to a class" as used in this specification should be interpreted expansively to cover any classification of defects to classes that is considered a reference standard, such as classification by a human user, classification by a trusted classifier, etc.

[0032] The term "classification" as used in this specification should be interpreted expansively as the assignment of any defect to one of a plurality of predetermined categories. Typically, a classifier is trained on defects known to belong to each of the desired categories. Once trained, the classifier can be used to classify a defect into any category, regardless of whether it is known to which category the defect belongs.

[0033] A given defect can belong to either the majority class or the minority class. When a classifier is applied to a set of defects, most defects will be automatically classified by the classifier as belonging to the majority class. However, due to the insufficient number of training defects known to belong to the minority class, many minority defects will also be misclassified as belonging to the majority class. Because only a small fraction of the defects detected during inspection are further examined during verification, minority defects classified as the majority class have a low probability of being verified.

[0034] Therefore, for multiple reasons, under-representation of defects from certain communities (e.g., minority classes) when training classifiers is a serious problem in manufacturing and inspection processes.

[0035] First, incorrect classification can lead to inaccurate estimates of the quality of the inspection process. Due to the low representativeness of a few defects, the impact of a few defects on the process estimate is very low, and thus the quality of the overall process may be inaccurately estimated.

[0036] Furthermore, the stability of the test treatment can be low: since smaller populations are more susceptible to adverse effects of changes when defects are misclassified, a small number of defects (or even a single defect) can significantly affect the standard deviation of the treatment estimate. It is therefore difficult to estimate the test treatment stably.

[0037] Third, misclassification alone can produce defects that are too severe to be handled. Moreover, great efforts are made to eliminate a few very severe defects, thereby reducing the remaining defects in these categories. Therefore, if these defects are underrepresented during training, other defects will not be classified, which may cause serious problems in the sample to go undetected; and

[0038] Fourth, some classes of defects, while referred to as belonging to the same class, can have different characteristics. For example, a defect of type “particle” can be significantly different in metal and silicone. By not identifying and training on specific minority classes, important information about specific defects may be lost.

[0039] With this in mind, now pay attention Figure 1 , showing a general block diagram of an inspection system according to certain embodiments of the presently disclosed subject matter. Figure 1The inspection system 100 shown in the figure can be used to inspect samples (e.g., wafers and / or portions thereof) as part of sample manufacturing. Inspection can be part of object manufacturing and can be performed during or after object manufacturing. The inspection system may include various inspection tools, such as one or more inspection inspection tools 101 and one or more calibration inspection tools 102, the inspection inspection tool 101 is configured to capture inspection images (usually at a relatively high speed and / or low resolution), and the calibration inspection tool 102 is configured to obtain calibration images of at least a portion of the defects detected by the inspection tool 101 (usually at a relatively low speed and / or high resolution). The inspection system 100 shown also includes a computer-based automatic defect classification tool (hereinafter also referred to as a classifier) ​​103, which is capable of automatically classifying defects into multiple categories according to defect types. As non-limiting examples, the classification can have different purposes, and the classification results can be used to identify specific defects of interest (DoI), filter out false defects from real defects, establish Pareto to identify deviations in statistical process control (SPC), and / or other ways (depending on the classification purpose).

[0040] The user can define the categories and the defects assigned to each category to train the classification engine 121 with the help of the GUI 108. In addition, the GUI 108 can be configured to enable the user (and / or the management system) to set and monitor success criteria or measurements related to the automated defect classification, such as purity, accuracy, extractability, or other parameters for each category, and optimize the classification results according to user requirements.

[0041] The classifier 103 may be operably connected to one or more inspection tools 101 and / or one or more verification tools 102. Optionally, the classifier 103 may be fully or partially integrated with one or more inspection tools 101 or with one or more verification tools 102. The classifier 103 may further be operably connected to a design server 110 and a data repository 109.

[0042] The sample may be inspected by an inspection tool 101, such as an optical inspection system, a low resolution SEM, etc. The resulting images and / or derived data may be processed, optionally along with other data (e.g., design data and / or defect classification data), to select defects for review.

[0043] The subset of potential defect locations selected for calibration may be inspected by a calibration tool 102, such as a scanning electron microscope (SEM) or an atomic force microscope (AFM), etc. Data related to the calibration image and / or its derivative data (hereinafter referred to as calibration data 122) may be transmitted to the classifier 103 directly or through one or more intermediate systems. Note that the calibration data may include data generated by one or more intermediate systems as derivative data of the calibration image.

[0044] The classifier 103 includes a processor and memory circuit system (PMC) 104, which is operably connected to a hardware-based input interface 105 and a hardware-based output interface 106. The PMC 104 is configured to provide the processes necessary to operate the classifier, as described in reference to Figure 2 As further described in detail, the PMC 104 includes a processor and a memory (not separately shown in the PMC 104). Figures 2 to 4 The operations of the classifier 103 and the PMC 104 are described in further detail.

[0045] The processing result of the classifier 103 output by the output interface 106 can be provided to the inspection and testing tool 101 or the verification and testing tool 102 as inspection instruction related data 123 or verification instruction related data 124 respectively, so as to change the operation of the inspection and testing tool 101 or the verification and testing tool 102, for example by changing the value of the operating parameter.

[0046] Those skilled in the art will readily appreciate that the teachings of the presently disclosed subject matter are not limited to Figure 1 The system shown; equivalent and / or modified functionality may be combined or divided in another manner and may be implemented in any suitable combination of software, firmware and hardware.

[0047] Notice that Figure 1 The inspection system shown can be implemented in a distributed computing environment, where Figure 1The above functional modules shown can be distributed on multiple local and / or remote devices and can be linked through one or more communication networks. It should also be noted that in other embodiments, at least a portion of the inspection tools 101 and / or 102, the data repository 109, the storage system 107, the design server 110 and / or the GUI 108 can be outside the inspection system 100 and operate to communicate data with the classifier 103 through the input interface 105 and the output interface 106. The classifier 103 can be implemented as one or more independent computers used in conjunction with one or more inspection tools. Depending on the circumstances, the classifier 103 can operate on pre-acquired calibration data stored in the data repository 109 and / or the storage system 107. Alternatively or additionally, the corresponding functions of the classifier 103 can at least partially integrate one or more inspection tools, process control tools, recipe generation tools, systems for automatic defect calibration and / or classification, and / or other systems related to inspection.

[0048] The PMC 104 may include a minority defect recognition engine 117, a training engine 120, and one or more classification engines 121. The minority defect recognition engine 117 may be used to recognize minority defects, and the training engine 120 may train one or more classification engines 121 using the minority defects.

[0049] The minority defect identification engine 117 may include an initial minority defect acquisition engine 118 for identifying minority defects for which a reference standard exists and which are misclassified as majority class by the classification engine 121. The minority defect identification engine 117 may further include an additional minority defect acquisition engine 119 configured to identify defects that may be minority defects but for which a reference standard does not exist. The user may then verify these defects and determine whether one or more defects are indeed minority defects. Once a sufficient number of minority class defects are available, a training engine 120 may be used to train a classification engine (also referred to as a classifier 121).

[0050] The PMC 104 may include a training engine 120 configured to receive a plurality of defects, each defect having a corresponding classification, and output a classifier adapted to receive the defects and provide a class into which the defects are classified.

[0051] PMC 104 may include one or more classification engines 120, also referred to as classifiers, which may be trained by training engine 118 or another training engine, or received from an external source. As described below, one such engine may be an initial engine trained by an insufficient number of defects, while another engine may be trained by more defects.

[0052] Reference Figure 2, shows a general flow chart of operating the few defect identification engine 117 according to certain embodiments of the presently disclosed subject matter. The PMC 104 is configured to perform the various operations detailed below according to computer-readable instructions implemented on a non-transitory computer-readable storage medium.

[0053] Reference now Figure 3 , which displays the majority and minority classes and their classifications in the form of a "confusion matrix". The term "confusion matrix" should be interpreted expansively to cover any tabular layout that allows for a visual display of the ground truth classification versus the automatic classification results.

[0054] therefore, Figure 3 Table 300 shows, for example, human and machine classification of a total of 250 defects detected during inspection of an exemplary sample, each defect belonging to one of the A, B, C, or D categories. Table 300 includes columns 304, 308, 312, and 316, each column associated with each category of defects classified by the user, and thus, this assignment to category is referred to as "ground truth." The number of defects classified by the user is shown in the "Total" row 332: 60 in category A, 44 in category B, 119 in category C, and 27 in category D.

[0055] As shown in rows 320, 324, and 328 of table 300, the defects are also classified using a preset classifier or any other initial classifier: for example, 52 class A defects are indeed classified as class A, 2 class B defects are classified as class C defects, and so on.

[0056] As shown in the first three entries of the main diagonal of Table 400, most of the defects in categories A, B, and C are correctly classified: 52 out of 60 defects in category A, 38 out of 44 defects in category B, and 110 out of 119 defects in category C. Therefore, categories A, B, and C are considered the majority category. However, no category defects are classified as category D. This is most likely because there were not enough category D defects when training the classifier, so the classifier cannot classify the defects into this category. Therefore, category D is called the minority category, and defects in this category are classified as the majority category. Therefore, more category D defects need to be found so that the classifier can be trained accordingly.

[0057] Now back to Figure 2 , the PMC 104 receives (200) a plurality of defects classified by a first classifier into a first majority class. The received defects have been pre-classified into predefined classes (e.g., by a human expert and / or another classifier and / or a version of a previous classifier, etc.), such that this pre-classification is considered a reference standard. Thus, the PMC 104 receives defects that belong to the same majority class or minority class.

[0058] For example, the PMC 104 may receive defects that are assigned to one category (e.g., category A shown on row 320) by the first classifier and are known to belong to category A or category D. Defects that are classified as category A but belong to most other categories (e.g., 4 defects belonging to category B and 3 defects belonging to category C) will be ignored. Therefore, there is a need to distinguish between the 52 category A defects that are correctly classified as category A and the 3 category D defects that are incorrectly classified as category A.

[0059] Each defect can be described as a collection of values ​​assigned to various attributes, such as a vector. Typically, each defect can be described as values ​​assigned to hundreds or more attributes, such as those related to its type, color, size, location, cause, etc.

[0060] The PMC 104 may then select 204 a subset of attributes and define a discriminator for each attribute of the subset in order to correctly distinguish between defects belonging to the majority class and defects belonging to the minority class. Attribute subset selection may be referred to as dimensionality reduction. This distinction may be mathematically possible when there are more dimensions (i.e., attributes) than defects. However, reducing the number of dimensions may help to increase generality, such as distinguishing stability, so that additional defects may also be classified as either class.

[0061] The differentiation may be described as a temporary classifier based only on a selected subset of attributes, and the defects are classified into a majority class and a minority class. It should be understood that correctly distinguishing defects assigned to the majority class from defects assigned to the minority class may involve classifying at least a predetermined portion of the defects of the majority class into the majority class, and classifying at least a (possibly different) predetermined portion of the defects of the minority class into the minority class. The predetermined portion may involve a predetermined proportion of the individual defects, at least a predetermined number of the individual defects, etc.

[0062] If the attribute can receive a range of values, such as a discrete range or a continuous range, the differentiator can be a cut-off point such that defects with values ​​above a threshold are associated with one category, while defects with values ​​below the threshold are associated with the other category. If the attribute can receive one of two possible values ​​(such as 0 or 1), the differentiator can be defined such that defects with a first value of the attribute are associated with one category, while defects with a second value of the attribute are associated with the other category.

[0063] Selecting a subset of attributes may use a random forest approach, which operates by building a plurality of decision trees, receiving classification results from each such decision tree, and making a decision that is consistent with the decisions made by the majority of trees. In the first stage, the first tree is built: the first attribute may be selected based on the Gini Impurity index, which may be considered as a measure of the frequency with which a randomly selected element from a set will be incorrectly labeled when randomly labeled according to the label distribution in the subset. For example, assume that nine defects are provided, each of which belongs to one of two categories: "X" and "0". The nine defects have the following values ​​for the attribute: [1, 1.5, 2, 3, 4, 5.5, 6, 8, 10]. When the defects are sorted in ascending order of the value of this attribute, the defect vector is: [XX 0X X 0 0X 0], which means that the first, second, fourth, fifth, and eighth defects belong to the "X" category, and the other defects belong to the "0" category. For this vector, the Gini impurity index of class "X" is defined as Pi*(1-Pi) (where Pi is the probability of correctly classifying "X"), which is It can be understood that the optimal value of the Gini impurity index is 0, which indicates that perfect discrimination is achieved by the attribute (where Pi is 0 or 1). If the discrimination value of the attribute used to discriminate the vector is selected to be between 1.5 and 2, the vector will be as follows: [XX|0X X 0 0 X 0], indicating that the defects on the left side of the "|" symbol are classified as "0" and the defects on the right side are classified as "X", then the Gini impurity index of category "X" is: Similarly, the Gini impurity index for class “0” is 0, and hence the weighted average is This means that the Gini index has decreased, i.e. purity has increased.

[0064] The attribute and the discriminant selection that provides the best weighted average can be chosen as the first level of the decision tree. At the next level, different attributes can be chosen, or the same attribute with different discriminant values.

[0065] In some embodiments, only some of the defects classified as the majority class may be selected to build the random forest, rather than all defects. However, the proportion of defects misclassified as the majority class that may be selected to build the random forest may be higher than the proportion of defects classified as the majority class. For example, if 100 defects have been classified as the majority class, two of which belong to the minority class and 98 belong to the majority class, then 20 defects may be selected for the random forest, including two defects belonging to the minority class, and 18 defects belonging to the majority class.

[0066] Multiple such trees can be constructed, each based on a different set of defects and a different choice of attributes and discriminators.

[0067] Once a predetermined number of decision trees have been built, a subset of attributes may be selected. The subset of attributes may be selected based on each attribute participating in a tree: attributes participating in more trees (or at higher levels in one or more trees) may be given a higher rank than attributes participating in fewer trees (or at lower levels in one or more trees).

[0068] The random forest method can eliminate the situation of overfitting the attribute set to the available training set, where overfitting involves a situation where a subset of attributes is designed to discriminate well for the training set, but is therefore less effective for other defect sets.

[0069] PMC 104 may then generate (212) a temporary training set including: a plurality of defects that have been correctly classified as the majority class; and a plurality of minority class defects that have been incorrectly classified as the majority class. In some embodiments, not all defects classified as the majority class are selected. In addition, the proportion of minority class defects that are incorrectly classified as the majority class may be higher than their respective proportions among all defects that are classified as majority defects. The temporary training set may also include additional defects.

[0070] The PMC 104 may generate additional defects. Additional defects are defects that are classified as minority classes by a classifier based on a subset of attributes (e.g., the random forest classifier above). Generating additional defects may also be referred to as oversampling. It will be appreciated that generating defects may be performed by assigning values ​​to attributes in a subset of attributes.

[0071] Reference now Figure 4 , which shows an exemplary embodiment of oversampling in an attribute subset comprising two dimensions.

[0072] Given multiple defects of the majority and minority classes, oversampling may involve generating one or more synthetic defects, ie, one or more sets of values, each of which is relatively close to an existing minority defect. Figure 4 Three minority defects are shown, 400, 404, and 412, represented by "O", and several majority defects, such as 408 or 416, represented by "X". In order to create a classifier that can distinguish between the two classes of defects well, more synthetic defects can be generated that are closer to minority defects that have majority class neighbors. Thus, synthetic defects can be generated near defects 404 and 412, but not near defect 400, which has no "X" class neighbors.

[0073] Generating synthetic defects may be performed using a synthetic minority oversampling technique (SMOTE): for each minority defect, the distance to the nearest majority defect may be calculated. The distance calculation may be performed using any desired metric, such as a (possibly weighted) sum of squares over a selected subset of attributes. A predetermined number of minority defects may be selected whose distances to their nearest neighbors of the majority class are minimized. In some embodiments, the same minority defect may be selected multiple times. Thus, minority defects 404 and 412 may be selected. For each such selected defect, a synthetic defect may be created having an attribute value in the range between the attribute value of the minority defect and the attribute value of the nearest majority defect. In Figure 4 In the example of FIG. 4 , a synthetic defect can be created along a line connecting minority defect 404 and majority defect 408. In some embodiments, a synthetic defect can be generated at a position closer to the minority defect, for example, the value of each attribute can be calculated as 0.8*attribute value of minority defect+0.2*attribute value of majority defect, or any other weighted value combination thereof. A second synthetic defect can be generated along a line connecting minority defect 412 and majority defect 416.

[0074] PMC 104 can then train 216 a confidence engine on the temporary training set, the attribute subset, and the distinguishers for each attribute of the subset. The confidence engine can be configured to receive a defect and a category into which the defect is classified, and output a confidence that the defect does belong to the category.

[0075] The credibility engine can also be implemented as a tree ensemble, where the trees of the tree ensemble can be generated using gradient boosting. The decision tree can be generated starting from a preset classification tree, which operates according to the selected attribute subset and classifies defects into majority and minority categories. The enhanced Gini impurity index can take into account the weight assigned to each category: WA*PA*(1-W A P A ), where W A is the weight assigned to category A, P A is the probability of correctly classifying a defect as class A. In order to train the confidence engine to better classify defects in the minority class, the weight of the minority class may be higher than the proportion of minority defects among all defects classified as the majority class. Further trees can then be generated in which higher weights are assigned to minority defects that were incorrectly classified by previously generated trees, for example:

[0076] The enhanced Gini impurity index after classification can be higher by choosing different discrimination values ​​to give higher weights to some defects than others.

[0077] The output generated by the credibility engine is therefore a collection of trees, each of which is associated with a class and a weight. The weight can be obtained based on the error rate associated with each tree. The error rate can be defined as the sum of the weights of defects for which the results output by the tree differ from the reference standard.

[0078] Once the credibility engine is available, the PMC 104 may apply (220) the credibility engine to further defects that are classified as the majority class, where there is no reference standard for these defects. Therefore, each such defect classified as the majority class is provided to the credibility engine and a credibility is obtained. The credibility represents the confidence that the defect does belong to the majority class and is not incorrectly classified as the majority class. The credibility can be obtained by classifying the defect through all trees in the tree ensemble generated in step 216 and summing the weights of all trees that classify the defect as the majority class, thereby obtaining a credibility between 0 and 1.

[0079] The PMC 104 may then output (224) the defects classified as the majority class, with the majority class having a low confidence, such as below a predetermined threshold. Alternatively, a predetermined number of defects with a minimum confidence or a predetermined proportion of all defects classified as the majority class may be output. A low confidence may indicate a high probability that the defect is not a majority defect but a minority defect.

[0080] Reference now Figure 5 , showing a general flow chart of a method of calibrating a sample according to certain embodiments of the presently disclosed subject matter.

[0081] The PMC 104 may be configured to perform various operations described in detail below according to computer-readable instructions implemented on a non-transitory computer-readable storage medium.

[0082] The classifier is trained (500) on available training data, including training on defects classified by users.

[0083] Classification is then performed (504) for further defects for which the classification is not yet known.

[0084] As above combined Figure 2 As described, the classification results can be monitored and rejected (508). Monitoring involves obtaining confidence in the majority of defects, such as Figure 2 , and rejection may involve outputting ( 224 ) defects with low classification confidence.

[0085] The output defects may be checked, for example, by a user, and a reference standard classification of these defects may be received (512).

[0086] Defects indicated by the user as minority defects may be collected (516) and stored in a storage device or data structure called a "container." When a sufficient number of minority defects for training have accumulated in the container (e.g., at least a predetermined number, such as 30, 40, 100, etc.), a new classifier may be trained where this class is also the majority class and thus may also correctly classify minority defects. Thus, this classifier may replace the first classifier.

[0087] The training (500) may then be repeated, which may also be referred to as retraining or updating the classifier with the available reference standard classification.

[0088] Figure 5 The iterations may continue for a predetermined number of times, for a predetermined time period, until the rejection rate drops below a first predetermined threshold, until the number of rejections drops below a second predetermined threshold, or until another stopping criterion is met.

[0089] Those skilled in the art will readily appreciate that the teachings of the presently disclosed subject matter are not limited to Figure 2 and Figure 5 Constraints of the approach shown. To obtain a small number of defects and train the classifier accordingly, other appropriate algorithms and combinations thereof can be implemented.

[0090] The present invention is not limited to the details set forth in the description contained herein or shown in the accompanying drawings. The present invention can have other embodiments and can be practiced and implemented in various ways. Therefore, it should be understood that the terms and expressions used herein are for descriptive purposes and should not be considered restrictive. Therefore, those skilled in the art will recognize that the concepts upon which the present disclosure is based can be easily used as the basis for designing other structures, methods and systems for achieving several purposes of the presently disclosed subject matter.

[0091] It should also be understood that the system according to the present invention can be implemented at least in part on a suitably programmed computer. Similarly, the present invention contemplates a computer readable computer program for performing the method of the present invention. The present invention further contemplates a non-transitory computer readable memory tangibly embodying a program of instructions executable by a computer to perform the method of the present invention.

[0092] It will be readily appreciated by those skilled in the art that various modifications and changes may be applied to the embodiments of the invention as described above without departing from the scope of the invention as defined in and by the appended claims.

Claims

1. A system for classifying defects in a semiconductor sample into a plurality of categories, the system comprising a processor and memory circuit system (PMC), the processor and memory circuit system (PMC) being configured to: receiving a plurality of defects classified into majority classes by a first classifier, wherein one or more defects of the plurality of defects belong to one or more minority classes indicating ground facts of the one or more defects, and at least a portion of the defects of the plurality of defects belong to the majority class indicating ground facts of the at least a portion of the defects, and wherein each defect of the plurality of defects is characterized by a plurality of attributes, each attribute having a respective value; selecting a subset of the plurality of attributes and defining a differentiator for each attribute of the subset such that a second classifier configured to classify defects based on the subset and the differentiator for each respective attribute correctly classifies at least a predetermined share of the plurality of defects into the one or more minority classes and the majority class; generating a temporary training set, the temporary training set comprising: at least a portion of the defects correctly classified as the majority class, at least a portion of the defects belonging to the minority class but incorrectly classified as the majority class, and additional defects having values ​​for the subset of attributes to cause the second classifier to classify the additional defects as the minority class; a training engine configured to obtain a confidence that a defect characterized by a first value of the subset of attributes belongs to the majority class, wherein training is performed on the temporary training set, the subset of attributes, and the differentiator for each attribute of the subset; applying the engine to each given defect in a second plurality of defects classified into the majority class to obtain a confidence level that the given defect is classified into the majority class; and Defects having a confidence level lower than a predetermined threshold are output from the second plurality of defects, and are thus determined to be incorrectly classified as the majority class but have a higher probability of belonging to the one or more minority classes.

2. The system of claim 1, wherein the PMC is further configured to generate additional defects that can be classified into the one or more minority classes by the second classifier and include the additional defects into the temporary training set.

3. The system of claim 2, wherein at least one of the additional defects is a composite defect generated as a plurality of first values, each first value being associated with an attribute from the subset according to a differentiator for each attribute of the subset.

4. The system of claim 3, wherein a first value associated with a given attribute is determined to be a value between a value of the given attribute for a first defect belonging to the one or more minority classes and a value of the given attribute for a second defect adjacent to the first defect and belonging to the majority class.

5. The system of claim 3, wherein a first value associated with a given attribute is determined to be a value of the given attribute for defects belonging to the one or more minority classes.

6. The system of claim 1, wherein the PMC is further configured to: identifying at least a portion of the output defects as belonging to the one or more minority classes; updating the first classifier by training the first classifier on a training set, the training set including at least some of the plurality of defects belonging to the one or more minority classes, at least some of the plurality of defects belonging to the majority class, and the output defects identified as belonging to the one or more minority classes; and The plurality of defects classified into the majority category by the first classifier are updated, and defects are output, the defects occurring when updating and for which the engine obtains a confidence level lower than the threshold.

7. The system of claim 1, wherein the engine comprises a tree ensemble.

8. The system of claim 1, wherein the differentiator is a threshold from a range of values.

9. The system of claim 1, wherein the differentiator comprises one or more values ​​from a set of discrete values.

10. A method for classifying defects in a semiconductor sample into a plurality of categories, the method being performed by a computing platform comprising a processor and a memory circuit system PMC, the method include: receiving, by the PMC, a plurality of defects classified into majority classes by a first classifier, wherein one or more defects of the plurality of defects belong to one or more minority classes indicating ground facts of the one or more defects, and at least a portion of the plurality of defects belong to the majority class indicating ground facts of the at least a portion of the defects, and wherein each defect of the plurality of defects is characterized by a plurality of attributes, each attribute having a respective value; selecting, by the PMC, a subset of the plurality of attributes and defining a differentiator for each attribute of the subset so that a second classifier configured to classify defects based on the subset and the differentiator for each corresponding attribute correctly classifies at least a predetermined share of the plurality of defects into the one or more minority classes and the majority class; generating, by the PMC, a temporary training set, the temporary training set comprising: at least a portion of the defects correctly classified as the majority class, at least a portion of the defects belonging to the minority class but incorrectly classified as the majority class, and additional defects having values ​​for the subset of attributes such that the second classifier classifies the additional defects as the minority class; training an engine by the PMC, the engine being configured to obtain a confidence that a defect characterized by a first value of the subset of attributes belongs to the majority class, wherein training is performed on the temporary training set, the subset of attributes, and the differentiator for each attribute of the subset; applying, by the PMC, the engine to each given defect in a second plurality of defects classified into the majority class to obtain a confidence level that the given defect is classified into the majority class; and Defects having a confidence level lower than a predetermined threshold are outputted by the PMC from the second plurality of defects, and are thus determined to be incorrectly classified as the majority class but have a higher probability of belonging to the one or more minority classes.

11. The method of claim 10, wherein the method further comprises include: Additional defects are generated that can be classified into the one or more minority classes by the second classifier, and the additional defects are included in the temporary training set.

12. The method of claim 11, wherein at least one of the additional defects is a composite defect generated as a plurality of first values, each first value being associated with an attribute from the subset according to a differentiator for each attribute of the subset.

13. The method of claim 12, wherein a first value associated with a given attribute is determined to be a value between a value of the given attribute of a first defect belonging to the one or more minority classes and a value of the given attribute of a second defect adjacent to the first defect and belonging to the majority class.

14. The method of claim 12, wherein a first value associated with a given attribute is determined as a value of the given attribute for defects belonging to the one or more minority classes.

15. The method of claim 10, further comprising: include: identifying at least a portion of the output defects as belonging to the one or more minority classes; updating the first classifier by training the first classifier on a training set, the training set including at least some of the plurality of defects belonging to the one or more minority classes, at least some of the plurality of defects belonging to the majority class, and the output defects identified as belonging to the one or more minority classes; as well as The plurality of defects classified into the majority category by the first classifier are updated, and defects are output, the defects occurring when updating and for which the engine obtains a confidence level lower than the threshold.

16. The method of claim 10, wherein the engine is included in a random forest classifier. The method of claim 10 , wherein the differentiator is a threshold from a range of values.

18. The method of claim 10, wherein the differentiator comprises one or more values ​​from a set of discrete values.

19. A non-transitory computer-readable medium comprising instructions, which when executed by a computer, cause the computer to perform a method for automatically classifying defects in a semiconductor sample into a plurality of categories, the method include: receiving a plurality of defects classified into majority classes by a first classifier, wherein one or more defects of the plurality of defects belong to one or more minority classes indicating ground facts of the one or more defects, and at least a portion of the defects of the plurality of defects belong to the majority class indicating ground facts of the at least a portion of the defects, and wherein each defect of the plurality of defects is characterized by a plurality of attributes, each attribute having a respective value; selecting a subset of the plurality of attributes and defining a differentiator for each attribute of the subset such that a second classifier configured to classify defects based on the subset and the differentiator for each respective attribute correctly classifies at least a predetermined share of the plurality of defects into the one or more minority classes and the majority class; generating a temporary training set, the temporary training set comprising: at least a portion of the defects correctly classified as the majority class, at least a portion of the defects belonging to the minority class but incorrectly classified as the majority class, and additional defects having values ​​for the subset of attributes to cause the second classifier to classify the additional defects as the minority class; a training engine configured to obtain a confidence that a defect characterized by a first value of the subset of attributes belongs to the majority class, wherein training is performed on the temporary training set, the subset of attributes, and the differentiator for each attribute of the subset; applying the engine to each given defect in a second plurality of defects classified into the majority class to obtain a confidence level that the given defect is classified into the majority class; and Defects having a confidence level lower than a predetermined threshold are output from the second plurality of defects, and are thus determined to be incorrectly classified as the majority class but have a higher probability of belonging to the one or more minority classes.

Images