Classification product unit

Abstract

One embodiment relates to a method of classifying a product unit subjected to a process performed by a device, the method comprising: receiving KPI data associated with a plurality of components of the device and comprising data associated with a plurality of KPIs; clustering the KPI data into clusters to identify a plurality of clusters; analyzing the plurality of clusters to identify a plurality of failure modes associated with the device, for each identified failure mode, assigning a threshold to each KPI associated with the failure mode; and for each product unit of the plurality of product units: determining a likelihood of each failure mode of the plurality of failure modes based on the KPI data of the product unit and the threshold assigned to each KPI associated with the failure mode of the plurality of failure modes; and performing classification based on the likelihood of each of the plurality of failure modes.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to European application No. 22190458.4 filed on August 16, 2022 and European application No. 22196685.6 filed on September 20, 2022, and the entire contents of these European applications are incorporated herein by reference. Technical Field

[0003] The present invention relates to a computer-implemented method of determining a classification model for classifying product units, such as semiconductor wafers, and for classifying product units subjected to a process performed by an apparatus, such as a lithographic apparatus. Background Art

[0004] A lithographic apparatus is a machine configured to apply a desired pattern onto a substrate. A lithographic apparatus may be used, for example, in the manufacture of integrated circuits (ICs). A lithographic apparatus may, for example, project a pattern (also often referred to as a "design layout" or "design") at a patterning device (e.g., a mask) onto a layer of radiation-sensitive material (resist) disposed on a substrate (e.g., a wafer).

[0005] To project a pattern onto a substrate, a lithographic apparatus may use electromagnetic radiation. The wavelength of this radiation determines the minimum size of a feature that can be formed on the substrate. Typical wavelengths currently in use are 365 nm (i-line), 248 nm, 193 nm, and 13.5 nm. Compared to a lithographic apparatus using radiation having a wavelength of, for example, 193 nm, a lithographic apparatus using extreme ultraviolet (EUV) radiation having a wavelength in the range of 4 nm to 20 nm (e.g., 6.7 nm or 13.5 nm) can be used to form smaller features on a substrate.

[0006] Low-k 1 Photolithography can be used to process features whose size is smaller than the classical resolution limit of the lithographic equipment. In such a process, the resolution formula can be expressed as CD = k 1 × λ / NA, where λ is the wavelength of the radiation employed, NA is the numerical aperture of the projection optics in the lithographic apparatus, CD is the “critical dimension” (usually the smallest feature size printed, but in this case it is the half pitch), and k 1 is the empirical resolution factor. Usually, k 1The smaller the k, the more difficult it becomes to reproduce on the substrate a pattern that resembles the shape and size planned by the circuit designer in order to achieve specific electrical functionality and performance. To overcome these difficulties, complex fine-tuning steps can be applied to the lithographic projection equipment and / or the design layout. These include, for example, but are not limited to, optimization of the NA, customized illumination schemes, use of phase-shifting patterning devices, various optimizations of the design layout such as optical proximity correction (OPC, sometimes also called "optical and process correction") in the design layout, or other methods generally defined as "resolution enhancement techniques" (RET). Alternatively, a tight control loop for controlling the stability of the lithographic equipment can be used to improve resolution at low k. 1 Reproduction of the following pattern.

[0007] Due to the high complexity of lithography machines, diagnostics are a considerable challenge. Especially when different modules and the complex interactions between them are involved, identifying the root cause of the problem becomes increasingly difficult.

[0008] Key performance indicators (KPIs) can be constructed to indicate the health of a particular subsystem within a lithographic apparatus. Typically, KPIs are constructed so that a large KPI value represents a poorly performing subsystem, and a small KPI value represents nominal performance. The various subsystems of the lithographic apparatus may interact with each other, which causes the KPIs to be interrelated.

[0009] Therefore, KPIs can be used to detect drift and random excursions in a subsystem. In many cases, setting a single threshold in some KPIs is sufficient to detect some out-of-spec wafers (e.g. Figure 4 depicted in ). Figure 4 Each data point in the graph of corresponds to a wafer. If one or more KPIs appear to be able to detect many wafers that are out of specification, this is a strong indication that this KPI (or combination of KPIs) is a failure mode (FM).

[0010] In most modeling approaches, the task of detecting FM and out-of-spec wafers is treated as a supervised learning problem. The KPI consists of features of the problem based on which a classifier (or regressor) is trained to predict overlap or other specific metrics. Given the unbalanced nature of the task, oversampling the total number of out-of-specs or adapting the optimization metric during hyperparameter tuning is often used. Finally, diagnostics are usually provided by using standard methods such as Shapley values, feature importance visualization or lime parameters and / or other methods. Summary of the invention

[0011] The inventors of the present invention have identified that most standard machine learning methods are unable to solve such problems due to the very low rate of wafers that are out of specification (typically 0.3% to 1.5%) and due to the way KPIs are designed (each KPI is specifically designed to solve a specific problem). In addition, it is often assumed that KPIs are sufficient for such detection, while in reality, there may be cases where no KPI can explain why a wafer is out of specification. In addition, many KPIs are correlated with each other. All of the above strongly indicate that the trained classifier / regressor is likely to overfit the data, no matter how much effort is spent in optimizing the data.

[0012] The inventors of the present invention have identified that another method of identifying KPIs that may potentially lead to the detection of out-of-spec wafers would be to perform an exhaustive and brute force search or exhaustive search of all available KPIs. During such a search, for each KPI, all individual thresholds and different thresholds for KPI combinations can be evaluated. This way of detecting out-of-spec wafers is suboptimal in providing any diagnostic solution. In addition, the thresholds are determined over the entire total number of wafers exposed over a longer period of time, and this limits the detection performance: the interactions between different subsystem failures are ignored in a single KPI approach, and this is necessary to capture unknown and complex failure modes. If the KPIs are fine-grained and not mature enough to capture subsystem failures, they cannot be used for wafer failure detection.

[0013] Aspects of the present invention are directed to automating the identification of failure modes (FMs) to enhance detection capabilities and provide diagnostics for out-of-spec wafers.

[0014] According to one aspect of the present disclosure, there is provided a computer-implemented method of determining a classification model including a KPI threshold, the classification model being used to classify a product unit undergoing a process performed by a device, the method comprising:

[0015] receiving KPI data obtained as a result of a plurality of product units being subjected to the process, the KPI data being associated with a plurality of components of the device and including data associated with a plurality of KPIs;

[0016] clustering the KPI data into clusters to identify at least one cluster;

[0017] analyzing the at least one cluster to identify a plurality of failure modes associated with the device, wherein the analyzing includes identifying, for one or more of the at least one cluster, a plurality of subgroups of KPI data related to failures of a product unit, each of the plurality of subgroups of KPI data being associated with a failure mode of the plurality of failure modes; and

[0018] The classification model is determined by assigning, for each identified failure mode, a threshold value to each KPI associated with the failure mode.

[0019] The method may include projecting the KPI data into a low dimensional space before performing the clustering. The method may include projecting the KPI data into a 2-dimensional space.

[0020] The identifying a plurality of KPI data subsets of clusters may include determining that a first distance between KPI data points associated with faults in the cluster exceeds a second distance associated with all KPI data points in a largest cluster of the at least one cluster.

[0021] The first distance may correspond to a first principal component identified by performing a principal component analysis on the KPI data points associated with faults in the clusters, and the second distance may be identified by performing a principal component analysis on the KPI data points in the largest cluster.

[0022] The second distance may be a predetermined percentage of a length of a first principal component identified by performing the principal component analysis on the KPI data points in the largest cluster.

[0023] The identifying of multiple KPI data subgroups of a cluster may include: performing independent component analysis on the KPI data points in the cluster to identify multiple independent components, each of the multiple independent components being associated with one or more KPIs; wherein each of the multiple KPI data subgroups corresponds to an independent component of the multiple independent components, whereby the KPI data of each of the one or more KPIs of the independent component exceeds a corresponding threshold associated with the KPI.

[0024] Each of the identified failure modes may be associated with one or more KPIs.

[0025] The method may also include supplementing the KPI data with manually generated KPI data associated with product units that do not meet specifications.

[0026] According to another aspect of the present disclosure, there is provided a computer-implemented method of classifying product units that are subject to a process performed by a device, the method comprising:

[0027] Receiving the classification model according to any of the preceding aspects to obtain a threshold value for each KPI associated with at least one failure mode;

[0028] For each of said product units that is subjected to a process performed by the device:

[0029] determining a likelihood of each of the at least one failure mode based on the KPI data for the product unit and the threshold value assigned to each KPI associated with the at least one failure mode; and

[0030] Classification of the product unit is performed based on the likelihood of the at least one failure mode.

[0031] The classification of the product unit may include a prediction of whether the product unit is in-spec or out-of-spec.

[0032] The classification of the product unit may include a confidence level of the prediction.

[0033] The performing the classification of the product unit may include, for each failure mode, comparing the likelihood to a corresponding predetermined failure mode threshold to determine whether the failure mode predicts that the product unit is not in compliance with specification.

[0034] If only a single failure mode has a likelihood exceeding its predetermined failure mode threshold, the classification may include (i) a prediction that the product unit is not meeting specifications, and (ii) the one or more KPIs associated with the single failure mode.

[0035] If a plurality of failure modes have probabilities exceeding their predetermined failure mode thresholds, the classification may include a weighted prediction that the product unit is out of specification for each of the plurality of failure modes.

[0036] The apparatus may be a lithographic apparatus and the product unit may be a semiconductor wafer.

[0037] According to one aspect of the present disclosure, there is provided a computer-implemented method of classifying product units that are subject to a process performed by a device, the method comprising:

[0038] receiving KPI data obtained as a result of a plurality of product units being subjected to the process, the KPI data being associated with a plurality of components of the device and including data associated with a plurality of KPIs;

[0039] clustering the KPI data into clusters to identify at least one cluster;

[0040] analyzing the at least one cluster to identify a plurality of failure modes associated with the device, wherein the analyzing includes identifying, for one or more of the at least one cluster, a plurality of subgroups of KPI data associated with failures of a product unit, each subgroup of the plurality of subgroups of KPI data being associated with a failure mode of the plurality of failure modes;

[0041] For each identified failure mode, assigning a threshold value to each KPI associated with the failure mode; and

[0042] For each product unit in the plurality of product units:

[0043] determining a likelihood of each of the plurality of failure modes based on the KPI data for the product unit and the threshold value assigned to each KPI associated with one of the plurality of failure modes; and

[0044] Classification of the product unit is performed based on the likelihood of each failure mode of the plurality of failure modes.

[0045] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium comprising instructions is provided, which, when executed by a processor of a device, causes the processor to perform any one of the methods described herein.

[0046] The instructions may be provided on one or more carriers. For example, there may be one or more non-transitory memories, such as EEPROM (e.g., flash memory), disk, CD or DVD-ROM, programming memory (such as read-only memory (e.g., for firmware)), one or more temporary memories (e.g., RAM), and / or one or more data carriers, such as optical signal carriers or electrical signal carriers. One or more memories may be integrated into the corresponding processing chip and / or separated from the chip. The program code (and / or data) for implementing the embodiments of the present disclosure may include source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code, code for setting or controlling an ASIC (application-specific integrated circuit) or FPGA (field programmable gate array), or code for a hardware description language.

[0047] According to one aspect of the present disclosure, there is provided an apparatus comprising a processor configured to perform any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying schematic drawings in which:

[0049] Figure 1 A schematic overview diagram depicting a lithographic apparatus;

[0050] Figure 2 A schematic overview diagram depicting a lithography unit;

[0051] Figure 3depicts a schematic representation of overall lithography showing the collaboration between three key technologies used to optimize semiconductor manufacturing;

[0052] Figure 4 Describe known techniques for detecting failed wafers by setting thresholds on KPIs;

[0053] Figure 5 depicts an example computer device upon which embodiments may be implemented;

[0054] Figure 6 is a flow chart of the steps of a method according to the present invention;

[0055] Figure 7a illustrates how data related to multiple KPIs can be projected into a two-dimensional space;

[0056] Figure 7b Figure 2 shows how the allocation of wafers in two dimensions captures the time element and chuck id without accessing this information;

[0057] Figure 8 The diagram shows the clustering of KPI data;

[0058] Figure 9a Figure 1. Detection of failed wafers by using a single KPI to set a global threshold.

[0059] Figure 9b to Figure 9d The figure shows the detection of failed wafers by clustering the wafers into clusters and setting a threshold for the KPI of each cluster;

[0060] Fig.10 Illustration of the detection of a failure mode that is difficult to identify; showing the distribution of data items including two features, where some data items originate from modules of a failed machine and other data items originate from corresponding modules of a reference machine;

[0061] Figures 11a to 11c illustrates the steps involved in reporting failure modes belonging to the same cluster; and

[0062] Fig.12 is a Bayesian network graph that combines the use of multiple FMs to predict whether a wafer is in / out of specification. DETAILED DESCRIPTION

[0063] In this document, the terms "radiation" and "beam" are used to cover all types of electromagnetic radiation, including ultraviolet radiation (e.g., having a wavelength of 365 nm, 248 nm, 193 nm, 157 nm or 126 nm) and EUV (extreme ultraviolet radiation, e.g., having a wavelength in the range of about 5 nm to 100 nm).

[0064] The terms "reticle", "mask" or "patterning device" as used herein may be broadly interpreted as referring to a general patterning device that can be used to impart a patterned cross-section to an incident radiation beam that corresponds to the pattern to be produced in a target portion of the substrate. In such contexts, the term "light valve" may also be used. In addition to classical masks (transmissive or reflective, binary, phase-shifting, hybrid, etc.), other examples of such patterning devices include programmable mirror arrays and programmable LCD arrays.

[0065] Figure 1 A lithographic apparatus LA is schematically depicted. The lithographic apparatus LA comprises an illumination system (also referred to as an illuminator) IL configured to condition a radiation beam B (e.g., UV radiation, DUV radiation, or EUV radiation), a mask support (e.g., a mask stage) MT configured to support a patterning device (e.g., a mask) MA and connected to a first positioner PM configured to accurately position the patterning device MA according to certain parameters, a substrate support (e.g., a wafer stage) WT configured to hold a substrate (e.g., a resist-coated wafer) W and connected to a second positioner PW configured to accurately position the substrate support according to certain parameters, and a projection system (e.g., a refractive projection lens system) PS configured to project a pattern imparted to the radiation beam B by the patterning device MA onto a target portion C (e.g., comprising one or more dies) of the substrate W.

[0066] In operation, the illumination system IL receives a radiation beam from a radiation source SO, for example via a beam delivery system BD. The illumination system IL may include various types of optical components, such as refractive, reflective, magnetic, electromagnetic, electrostatic and / or other types of optical components, or any combination thereof, for directing, shaping and / or controlling the radiation. The illuminator IL may be used to condition the radiation beam B to have a desired spatial and angular intensity distribution in its cross-section at the plane of the patterning device MA.

[0067] The term "projection system" PS as used herein should be broadly interpreted as covering various types of projection systems, including refractive, reflective, catadioptric, anamorphic, magnetic, electromagnetic and / or electrostatic optical systems, or any combination thereof, as appropriate to the exposure radiation used and / or to other factors such as the use of an immersion liquid or the use of a vacuum. Any use of the term "projection lens" herein may be considered synonymous with the more general term "projection system" PS.

[0068] The lithographic apparatus LA may be of a type in which at least a portion of the substrate may be covered by a liquid having a relatively high refractive index (e.g. water) so as to fill the space between the projection system PS and the substrate W - this is also known as immersion lithography. More information on immersion techniques is given in US6952253, which is incorporated herein by reference.

[0069] The lithographic apparatus LA may also be of a type having two or more substrate supports WT (also referred to as a "dual stage"). In such a "multi-stage" machine, the substrate supports WT may be used in parallel and / or a step of preparing a substrate W for subsequent exposure may be performed on a substrate W located on one of the substrate supports WT while another substrate W on another substrate support WT is being used to expose a pattern on the other substrate W.

[0070] In addition to the substrate support WT, the lithographic apparatus LA may also comprise a measurement platform. The measurement platform is arranged to hold sensors and / or cleaning means. The sensors may be arranged to measure properties of the projection system PS or properties of the radiation beam B. The measurement platform may hold a plurality of sensors. The cleaning means may be arranged to clean parts of the lithographic apparatus, e.g. a part of the projection system PS or a part of the system for providing immersion liquid. The measurement platform may be moved under the projection system PS when the substrate support WT is away from the projection system PS.

[0071] In operation, a radiation beam B is incident on a patterning device (e.g. a mask) MA held on a mask support MT and is patterned by a pattern (design layout) present on the patterning device MA. Having traversed the mask MA, the radiation beam B passes through a projection system PS which focuses the beam onto a target portion C of the substrate W. With the aid of a second positioner PW and a position measurement system IF, the substrate support WT can be accurately moved, for example in order to position different target portions C in the path of the radiation beam B at a focused and aligned position. Similarly, the first positioner PM and possibly another position sensor (which is not in the position sensor) are also provided. Figure 1 The patterning device MA (depicted explicitly in FIG. 1 ) can be used to accurately position the patterning device MA relative to the path of the radiation beam B. The mask alignment marks M1, M2 and the substrate alignment marks P1, P2 can be used to align the patterning device MA and the substrate W. Although the substrate alignment marks P1, P2 occupy dedicated target portions as illustrated, they can be located in the spaces between target portions. When the substrate alignment marks P1, P2 are located between target portions C, they are referred to as scribe line alignment marks.

[0072] like Figure 2As shown in FIG. 1 , the lithography apparatus LA may form part of a lithography cell LC (sometimes also referred to as a litho cell or (lithography cluster), which often also includes equipment for performing pre-exposure and post-exposure processes on a substrate W. Conventionally, these include a spin coater SC for depositing a resist layer, a developer DE for developing the exposed resist, a chill plate CH and a bake plate BK, for example, for regulating the temperature of the substrate W (e.g., for regulating the solvent in the resist layer). A substrate transport device or robot RO picks up a substrate W from input / output ports I / O1, I / O2, moves the substrate W between different process equipment, and transfers the substrate W to a feed table LB of the lithography apparatus LA. The devices in the lithography cell, which are often also collectively referred to as a track or coating and developing system, are typically under the control of a track or coating and developing system control unit TCU, which itself may be controlled by a management control system SCS, which may also control the lithography apparatus LA, for example via a lithography control unit LACU.

[0073] In order to correctly and consistently expose a substrate W exposed by the lithographic apparatus LA, it is desirable to inspect the substrate to measure properties of the patterned structure, such as overlay errors between subsequent layers, line thickness, critical dimensions (CD), etc. For this purpose, an inspection tool (not shown) may be included in the lithography cell LC. If an error is detected, adjustments may be made, for example, to the exposure of subsequent substrates or to other processing steps to be performed on the substrate W, particularly where the inspection is performed before other substrates W of the same batch or lot are yet to be exposed or processed.

[0074] The inspection apparatus, which may also be referred to as a metrology apparatus, is used to determine properties of a substrate W, and in particular to determine how properties of different substrates W vary or how properties associated with different layers of the same substrate W vary from layer to layer. The inspection apparatus is alternatively configured to identify defects on the substrate W, and may for example be part of a lithography cell LC, or may be integrated into a lithography apparatus LA, or may even be a separate device. The inspection apparatus may measure properties on a latent image (the image in the resist layer after exposure), or on a semi-latent image (the image in the resist layer after a post-exposure bake step PEB), or on a developed resist image (wherein exposed or unexposed parts of the resist have been removed), or even on an etched image (after a pattern transfer step such as etching).

[0075] Typically, the patterning process in the lithographic apparatus LA is one of the most important steps in the process, which requires a high accuracy in the dimensioning and placement of the structures on the substrate W. In order to ensure this high accuracy, the three systems can be combined in a so-called "holistic" control environment, such as Figure 3Schematically depicted in . One of these systems is the lithography apparatus LA, which is (in practice) connected to a metrology tool MT (a second system) and to a computer system CL (a third system). The key to this "holistic" environment is to optimize the cooperation between these three systems to enhance the overall process window and provide a tight control loop to ensure that the patterning performed by the lithography apparatus LA remains within the process window. The process window defines the range of process parameters (e.g., dose, focus, overlay) within which a particular manufacturing process produces a defined result (e.g., a functional semiconductor device) - typically the process parameters of the lithography process or patterning process are allowed to vary within the process parameter range.

[0076] The computer system CL can use (part of) the design layout to be patterned to predict which resolution enhancement techniques to use and perform computational lithography simulations and calculations to determine which mask layout and lithography equipment settings achieve the maximum overall process window for the patterning process (in Figure 3 Typically, the resolution enhancement technique is arranged to match the patterning possibilities of the lithographic apparatus LA. The computer system CL may also be used to detect where within the process window the lithographic apparatus LA is currently operating (e.g. using input from a metrology tool MT) to predict whether defects may be present (e.g. at the process window) due to suboptimal processing. Figure 3 0 in the figure).

[0077] The metrology tool MT may provide input to the computer system CL to enable accurate simulation and prediction, and may provide feedback to the lithographic apparatus LA to identify, for example, possible drift in the calibration state of the lithographic apparatus LA (in Figure 3 SC3).

[0078] Figure 5 A simplified diagram of a computing device 300 suitable for performing the methods described herein is shown. Figure 5As shown in , computing device 300 includes a central processing unit ("CPU") 302 to which memory 304 is connected. The functionality of CPU 302 described herein may be implemented in program code (software) stored on a memory (e.g., memory 304) including one or more storage media and arranged to be executed on a processor including one or more processing units. The storage medium may be integrated into CPU 302 and / or separate from CPU 302. The program code is configured to perform operations consistent with the embodiments discussed herein when retrieved from the memory and executed on the processor. Alternatively, it is not excluded that some or all of the functionality of CPU 302 is implemented in a dedicated hardware circuit system (e.g., an ASIC, a simple circuit, a gate, logic, and / or a configurable hardware circuit system similar to an FPGA).

[0079] The computing device 300 includes an input device 306 to allow a user to input data. The input device 306 may include a keyboard, a mouse, a touch screen, a microphone, etc. The computing device 300 also includes an output device 308 to output data to the user. The output device 308 may include a display and / or a speaker. The computing device 300 may include a communication interface 310 for communicating data to and from the computing device 300.

[0080] Figure 6 600 of classifying product units subjected to a process performed by a device according to the present invention. Figure 5 The method is performed by a computing device 300 (especially a CPU 302 of the computing device 300). The embodiments herein are described with reference to examples, whereby the product unit is a semiconductor wafer subjected to a lithography process performed by a lithography device. The method 600 is a two-stage process. The first stage is a training stage, which involves determining a classification model including a KPI threshold for later use in classifying product units subjected to a process performed by the device. This first stage includes steps S602 to S612 described in more detail below. The second stage is a classification stage, which involves performing classification of the product unit (applying the KPI data to the determined KPI threshold). This second stage includes step S614 described in more detail below.

[0081] The first and second stages of method 600 may be performed on a single KPI data set. That is, a single KPI data set may be used to determine the classification model and classify the wafers to which the KPI data is related.

[0082] Alternatively, the KPI thresholds per FM may be determined (trained) based on a KPI data set for a training wafer, and the actual use (classification) of these thresholds may be based on different KPI data (eg, of a production wafer). The two phases of method 600 are described in more detail below.

[0083] At step S602, KPI data is received. The CPU 302 may receive the KPI data from the memory 304. Alternatively or additionally, the CPU 302 may receive the KPI data from a remote device via the communication interface 310. The remote device may be a lithography apparatus or a storage device such as a network server or a database. The KPI data is obtained as a result of each of a plurality of wafers being subjected to a lithography process performed by the lithography apparatus. Multiple components of the lithography apparatus are involved in the lithography process, and the KPI data received by the CPU 302 at step S602 are related to these multiple components of the lithography apparatus. For example, when a wafer is subjected to the lithography process, a lens of the projection system PS may be deformed due to high temperature, and the KPI data may include data related to the degree of deformation of the lens when one or more wafers are subjected to the lithography process. In another example, the KPI data may include data related to the flatness of the substrate support WT when one or more wafers are subjected to the lithography process. In yet another example, the KPI data may include data related to the alignment of the mask when one or more wafers are subjected to the lithography process. Thus, it should be appreciated that for a particular wafer subjected to the lithography process, the received KPI data may include KPI data associated with one or more components of the lithography apparatus. The KPI data received at step S602 includes KPI data associated with one or more wafers that do not meet specifications.

[0084] The KPI data received at step S602 is in multiple dimensions (in at least three dimensions). The KPI data received at step S602 may be in more than 100 dimensions. As an optional step, at step S604, the CPU 302 projects the KPI data into a low-dimensional space. Step S604 may be performed by the CPU 302 executing a manifold learning algorithm (such as UMAP and t-SNE), which has the ability to project data into a low-dimensional space while maintaining internal structure.

[0085] Projecting the data into this space tries to summarize the totality of the data and show how much variation is expected with respect to KPIs and intra-KPI correlations. In other words, wafers that are "similar" with respect to their KPIs are expected to be close to the low-dimensional space created by the manifold learning algorithm. This is in Figure 7a, which illustrates the result of projecting approximately 70 KPIs into a two-dimensional space created by a manifold learning algorithm 700. It should be understood that the low-dimensional space in two dimensions is only an example.

[0086] Although it is mentioned above that the projection of the KPI data into a low-dimensional space is performed using a manifold learning algorithm, this is only an example.

[0087] There are alternative methods for performing this projection at step S604, including the use of principal component analysis (PCA) and autoencoder variants. However, manifold learning algorithms typically project the data by maintaining temporal structure, which is often desirable when performing detection of failure modes. Figure 7b , where the dark dots correspond to wafers at newer batch start times of the above-mentioned lithography process, and the light dots correspond to wafers at older batch start times of the lithography process. In addition, in embodiments whereby the lithography apparatus comprises more than one support WT (more than one chuck), it is observed that the KPI data of the two chucks "Chuck 1" and "Chuck 2" tend to be separated and form different giant clusters (depicted by the drawn ellipses). The manifold learning algorithm maintains the temporal structure without accessing the time when the wafers were exposed. This means that the manifold learning algorithm can capture drift effects and distribution shifts.

[0088] At step S606, the CPU 302 clusters the KPI data using a clustering algorithm (such as agglomeration, k-means, spectrum, etc.). If step S604 of projecting the KPI data into a low-dimensional space is followed (preferably by using a manifold learning algorithm), the clustering algorithm is applied to the dimensions created by the previous step. If the previous step is not followed, clustering is applied by using available KPIs.

[0089] Clustering is performed at step S606 for a number of reasons.

[0090] First, the clustering performed at step S606 is used as a way to group KPIs that lead to failure modes (FMs), which are important for diagnosis.

[0091] It is contemplated that multiple KPIs are used to explain why a wafer is "out of spec," and these KPIs may be different for these wafers. When reference is made herein to a wafer that is "out of spec" or a "failed wafer," reference is made to a wafer that has been subjected to the lithography process and therefore has parameters (e.g., related to overlay, focus, critical dimension, or other parameters) that are outside of a predetermined acceptable range of values. When reference is made herein to a wafer that is "in spec," reference is made to a wafer that has been subjected to the lithography process and therefore has parameters (e.g., related to overlay, focus, critical dimension, or other parameters) that are within a predetermined acceptable range of values. Clustering provides a way to group KPIs in order to identify the FMs required for diagnostic purposes. Figure 8 The reason for clustering is shown in the figure. By setting global thresholds on six different KPIs, a total of four wafers can be detected (note that not all six KPIs can detect the four wafers). Reporting all KPIs together would be outdated and would not have any diagnostic power. KPIs are grouped based on the cluster to which their corresponding wafers belong.

[0092] Figure 8 The diagram shows how clustering provides a way to group KPIs for detecting FM. In particular, Figure 8 Five KPI data clusters are shown. The left graph shows where "true positive" wafers (wafers that can be detected by a single KPI) are located. The middle list shows the KPIs that may have led to the detection of these wafers. The right list shows how these KPIs can be reported. Figure 8 As can be seen, no FMs were detected from Cluster 1 or Cluster 5. A single FM was detected from Cluster 2, where this FM was associated with a single KPI (KPI_1). A single FM was detected from Cluster 3, where this FM was associated with four KPIs (KPI_2, KPI_3, KPI_4, and KPI_5). A single FM was detected from Cluster 4, where this FM was associated with a single KPI (KPI_6). Clustering wafers into clusters means grouping together wafers that are similar with respect to their KPIs. When there is a FM associated with a particular problem, it is likely that it will affect "similar" wafers, thus affecting the rationale for studying the FMs for each cluster.

[0093] Second, the clustering performed at step S606 enhances the detection performance of existing KPIs.

[0094] For example, in Figure 9a In the example shown in , four wafers (identified by reference numerals 902, 904, 906, and 908) can be detected by using a global threshold on the KPI (i.e., KPI_10). That is, for the KPI data cluster ( Figure 9a Each cluster in the example of clusters 1 to 3) uses the same threshold KPI threshold.

[0095] on the contrary, Figure 9b to Figure 9d The diagram illustrates how a total of eight wafers can be detected by clustering the wafers into clusters and setting a KPI threshold for each cluster, thereby doubling the recall without sacrificing any precision. In particular, a first cluster-specific KPI threshold is used for use with the KPI data for cluster 1 to detect wafers 902, 904, and 906; a second cluster-specific KPI threshold is used for use with the KPI data for cluster 2 to detect wafers 908, 910, 912, and 914; and a third cluster-specific KPI threshold is used for use with the KPI data for cluster 3 to detect wafer 916. Thus, Figure 9b to Figure 9d shows how the clustering performed at step S606 is performed by detecting the clustering using Figure 9a The global thresholds shown in FIG. 9 are used to enhance the detection of failed wafers when there are no failed wafers 910, 912, 914, and 916 detected.

[0096] Third, the clustering performed at step S606 results in the detection of new FMs that are difficult to identify.

[0097] exist Fig.10 In the example shown in FIG. 1 , the KPI, namely KPI_20, is not able to detect any out-of-spec wafers in the entire total KPI data of the three clusters (clusters 1 to 3) (see FIG. 1 ). Fig.10 However, by setting the threshold for each cluster, (from the KPI data of cluster 2, such as Fig.10 ) can detect a wafer that does not meet specifications (identified by number 1002).

[0098] Referring back to method 600, at step S608, CPU 302 analyzes the plurality of clusters identified at step S606 to identify a plurality of failure modes (FMs) associated with the lithographic apparatus. It is possible that there are more than one FM in one of the plurality of clusters identified at step S606, and step S608 is performed to make method 600 more robust to such a scenario.

[0099] Out-of-spec wafers that belong to the same cluster but are far from each other on the manifold learning space are likely to have different FMs. At step S608, the CPU 302 analyzes each of the plurality of clusters identified at step S606 to determine that one or more of the plurality of clusters includes a plurality of KPI data subsets. That is, step S608 performs intra-cluster detection of FMs.

[0100] For example, in Fig.11aIn the example, there are two subgroups of wafers that are out of specification for cluster 1. These subgroups are likely to have different FMs. Thus, these FMs should be reported separately for cluster 1. In this example, two subgroups are analyzed, and the result of the analysis is that there are different KPIs that cause these failures. The analysis performed at step S608 can be performed in a variety of different ways.

[0101] The first internal cluster detection method that can be utilized at step S608 is based on principal component analysis, and refers to Fig.11b and Fig.11c Give a description.

[0102] In the first internal cluster detection method, the 1st principal component (PC) of each cluster identified at step S606 is calculated. In an embodiment performing optional step S604, the 1st principal component (PC) of each cluster is calculated on the space created by the manifold learning algorithm (or on the space created by any other technique for projecting KPI data into a low-dimensional space). The 1st PC is defined separately for each cluster by analyzing all wafers (both in-spec and out-of-spec wafers). This means that the 1st PC is different between cluster 1, cluster 2, and cluster 3, and so on.

[0103] Identify the largest cluster (in Fig.11a In the example shown in , cluster 1 is the largest), and a line "A" is identified whose length is a predetermined percentage of the 1st PC of cluster 1 (the largest of the clusters). This predetermined percentage may be, for example, 50%, but it should be understood that this is configurable and is only an example. The largest cluster is the cluster that extends the most in the latent space. Typically, the largest cluster includes the KPI data for the majority of wafers (compared to the other clusters), however, this may not always be the case.

[0104] The 1st PC for each cluster identified at step S606 is also calculated using the KPI data associated with the out-of-spec wafers in the cluster. Fig.11b The cluster-specific 1st PC based on out-of-spec wafers in cluster 1 (as indicated by line "B") is shown in FIG. By using only out-of-spec wafers (of a particular cluster), the corresponding line B is calculated separately on each cluster. This means that in the first intra-cluster detection method, line A is constant, but line B varies between clusters.

[0105] In the first internal cluster detection method, for each cluster, the CPU 302 determines whether the distance between the non-conforming wafers in the cluster (indicated by the length of line B) exceeds a predetermined percentage of the 1st PC of the maximum cluster length (indicated by the length of line A). If the length of line B is greater than line A, the CPU 302 performs internal cluster clustering on the non-conforming wafers of the cluster. It should be understood that when the optional step S604 is not performed, the principal components mentioned above can be calculated based on the KPI data. In other words, the process of separating the data based on PC described above is not exclusively applicable to the manifold learning space (or the space created by any other algorithm used at step S604), and the PC can be directly calculated based on the KPI data.

[0106] exist Fig.11a In the example of , only cluster 1 satisfies this condition, however, it should be understood that intra-cluster clustering may be performed with respect to one or more of the clusters identified at step S606.

[0107] As noted above, in the first inner cluster detection method, line A is constant, but line B varies between clusters. This is because some clusters may be very small and if A is calculated for each cluster separately, some clusters (because they are small) are likely to have false inner cluster FM.

[0108] Fig.11c The result of performing intra-cluster clustering on cluster 1 is shown, whereby three subgroups 1102 , 1104 , 1106 of out-of-spec wafers are identified for cluster 1 , each subgroup corresponding to a respective FM (subgroup 1104 consists of only one out-of-spec wafer).

[0109] An alternative way to detect failure modes per cluster is to rely on independent component analysis (ICA), rather than checking each KPI per cluster for whether it can detect out-of-spec wafers. ICA can provide a list (or multiple lists) of KPIs per cluster. As before, these KPI lists are evaluated by checking whether there is a threshold that separates in-spec and out-of-spec wafers. This second intra-cluster detection method is described below.

[0110] ICA is able to separate source signals from observed data by finding independent components (ICs) that maximize the statistical independence of the estimated components. Thus, based on the KPIs, ICA can be applied to identify different independent components, and some components may be potential failure modes.

[0111] In a second internal cluster detection method that can be utilized at step S608, for each cluster identified at step S606, ICA is applied to identify the IC for each cluster. Each identified IC consists of a list of KPIs that contribute to this component. For example, for cluster 1, the ICA performed by CPU 302 on the KPI data of the cluster can identify a list of three ICs (IC1, IC2, and IC3). IC1 can consist of a list of KPIs including KPIa, KPIb, and KPIc. IC2 can consist of a list of KPIs including KPIb and KPId. IC3 can consist of a list of KPIs including KPIa, KPIb, and KPIe. There are some functions to measure independence, such as functions for minimizing mutual information or maximizing the non-Gaussianity of the components.

[0112] After the ICs are detected, each of these ICs is evaluated to see if an out-of-spec die can be detected by simply drawing a threshold on its KPI. If this is the case, the IC is considered to be in a failure mode. In particular, if the KPIb data associated with Cluster 1 includes KPIb data for one or more die that exceeds a predetermined KPIb threshold, and if the KPId data associated with Cluster 1 includes KPId data for one or more die that exceeds a predetermined KPId threshold, then IC2 may be detected as FM.

[0113] Upon completion of step S608, the CPU 302 will have identified a plurality of FMs and, for each FM, one or more KPIs that contribute to the FM.

[0114] As an optional step, at step S610, the CPU 302 supplements the KPI data with artificially generated KPI data associated with non-compliant wafers. The total number of non-compliant wafers may be very limited compared to the total number of compliant wafers. In order to properly study the total number of non-compliant wafers and optimize the threshold settings on the KPI (mentioned in more detail below), step S610 may be performed by the CPU 302 executing a generative learning algorithm to generate artificial data for non-compliant wafers. Through the use of autoencoders and GANs, advances in generative deep learning allow this situation. Through the simple use of stacked autoencoders, the inventors have observed that it is possible to establish a latent space in which there are areas where most wafers do not meet specifications, and artificial examples can be created from the areas in order to create non-compliant wafers with similar characteristics.

[0115] Step S610 may be performed before or after the clustering performed at step S606. Performing step S610 after clustering may be preferred, as it is envisaged that the generative learning algorithm may be better trained by using data from the individual clusters identified at step S606.

[0116] At step S612, for each FM identified at step S608, the CPU 302 determines a classification model by assigning thresholds to each KPI associated with the FM. The values ​​of these thresholds are set depending on the metric (precision, recall, etc.) that needs to be optimized. The FMs and their corresponding KPI thresholds can be stored in memory, allowing them to be used directly for online prediction to detect wafers that do not meet specifications. For example, the FMs and their corresponding KPI thresholds can be stored in the memory 304 or in a remote storage device such as a network server or database. These thresholds are FM specific, and in a later classification stage, each wafer is evaluated for all known FMs.

[0117] At step S614, for each wafer in the plurality of wafers, the CPU 302 determines the likelihood of each of the plurality of FMs based on the KPI data of the wafer and the KPI threshold assigned to each KPI associated with one of the plurality of FMs, and performs classification of the wafer based on the likelihood of each of the plurality of FMs. In the context of the wafer, the classification includes whether the wafer meets the specification or does not meet the specification. The classification may also include a confidence level for such a prediction. As noted above, the KPI data of the plurality of wafers classified at step S614 may be the same or different from the KPI of the wafer used to determine the classification model.

[0118] Steps S612 and S614 may be performed by using a Bayesian network. Fig.12 is an illustration of an example Bayesian network that combines the use of multiple FMs to predict whether a wafer is in-spec / out-of-spec. The basic principle of using a Bayesian network is to refine the combination of the likelihoods of each of the multiple FMs to perform a classification of whether a wafer is in-spec or out-of-spec and give a final probability estimate of the prediction (i.e., the confidence of the prediction).

[0119] exist Fig.12In the example Bayesian network shown in , for each wafer, the contribution of KPI1 to KPI4 to FM1 and the contribution of KPI4 to KPI6 to FM2 can be learned and determined by the Bayesian network. The Bayesian network links the probability of a wafer not meeting specifications due to a KPI (estimated by the Bayesian network and generally independent of the KPI threshold) to the probability of FM occurring. This can be done by: (i) building a Bayesian network by incorporating domain knowledge and allowing a nn relationship between FM and KPI (which may have hidden states in between); (ii) from data; or (iii) a combination of Bayesian networks and data, by manually fixing the problem structure and learning the parameters (how the KPI propagates toward FM) from the data.

[0120] exist Fig.12 In the example Bayesian network shown in FIG. 6 , it is shown that the Bayesian network has determined that 30% of a particular wafer does not meet specification due to a first FM, FM1, and has determined that 65% of a particular wafer does not meet specification due to a second FM, FM2. The Bayesian network outputs the probabilities of all FMs for each wafer evaluated. The FM probabilities in the middle layers of the Bayesian network (the likelihood of each of the multiple FMs determined at step S614) will vary with each wafer evaluated.

[0121] To determine the likelihood of each of the plurality of FMs at step S614, the CPU 302 may perform the following steps:

[0122] Select all wafers in the data set that meet the specification;

[0123] Find the distribution (type and parameters) that minimizes the sum of squared errors for the population that meets the specifications;

[0124] Calculate the class probability of wafer being in specification / out of specification;

[0125] Identify the distribution (type and parameters) that minimizes the sum of squared errors for the total number of wafers (including in-spec / out-spec);

[0126] For each predicted wafer, the Naive Bayes formula is applied using all the previously calculated parameters to predict P(out of spec | FM 值 ) = 1-P (meets specification | FM 值 ).

[0127] In particular, it should be noted that P (compliant with specification | FM 值 ) can be found based on the following formula:

[0128] P(in-spec|FM 值 ) =

[0129] Thus P(FM value | meets specification) is given by the distribution (type and parameters) that minimizes the sum of squared errors of the total number of meets specification:

[0130] P (meets specification) is determined by a simple division among the number of wafers that meet specification among all wafers; and

[0131] P(FM value) is given by the distribution (type and parameters) that minimizes the sum of squared errors for the total number of wafers (including in-spec / out-spec).

[0132] At step S614, the classification is performed by the CPU 302 using the output (predicted label plus predicted probability) from each of the multiple FMs by combining all FM predictions into one global out-of-spec prediction.

[0133] Each FM in the plurality of FMs is associated with an FM threshold that defines when a wafer is classified as out of specification. These FM thresholds can be configured according to user requirements. Fig.12 In the first example of the Bayesian network shown in , the FM threshold associated with FM1 may be 30%, while the FM threshold associated with FM2 may be 80%. In this example, for this particular wafer, the only FM predicted to be out of spec is FM1, and the overall classification would be out of spec as described by FM1, e.g., "There is a 30% chance that this wafer will be out of spec due to FM1."

[0134] If two (or more) FMs predict that a particular wafer is out of spec, then CPU 302 weights their confidences to give an overall weighted prediction per FM (and thus, per symptom) since each FM is associated with a symptom. Fig.12 ), if there is a 50% chance that a particular wafer is out of specification due to FM1, and there is a 30% chance that a particular wafer is out of specification due to FM2 (and both of these probabilities exceed their respective FM thresholds). In marking the wafer as out of specification, the contribution to the marking due to FM1 may be provided as 50 / (50+30), and the contribution to the marking due to FM2 may be provided as 30 / (50+30).

[0135] The inventors have observed that by utilizing the method 600, the recall rate of detected wafers that are out of specification can be doubled, resulting in a significant improvement in detection performance. In addition, by utilizing the method 600, it is possible to detect FMs that are difficult to identify.

[0136] Further embodiments of the invention are disclosed in the following list of numbered aspects:

[0137] 1. A computer-implemented method of determining a classification model including a KPI threshold, the classification model being used to classify a product unit undergoing a process performed by a device, the method comprising:

[0138] receiving KPI data obtained as a result of a plurality of product units being subjected to the process, the KPI data being associated with a plurality of components of the device and including data associated with a plurality of KPIs;

[0139] clustering the KPI data into clusters to identify at least one cluster;

[0140] analyzing the at least one cluster to identify a plurality of failure modes associated with the device, wherein the analyzing includes identifying, for one or more of the at least one cluster, a plurality of subgroups of KPI data related to failures of a product unit, each of the plurality of subgroups of KPI data being associated with a failure mode of the plurality of failure modes; and

[0141] The classification model is determined by assigning, for each identified failure mode, a threshold value to each KPI associated with the failure mode.

[0142] 2. The computer-implemented method of aspect 1, wherein the method comprises projecting the KPI data into a low-dimensional space before performing the clustering.

[0143] 3. The computer-implemented method of aspect 2, wherein the method comprises projecting the KPI data into a 2-dimensional space.

[0144] 4. The computer-implemented method according to any preceding aspect, wherein the identifying a plurality of KPI data subsets of clusters comprises:

[0145] A first distance between KPI data points associated with the fault in the cluster is determined to exceed a second distance associated with all KPI data points in a largest cluster of the at least one cluster.

[0146] 5. A computer-implemented method according to aspect 4, wherein the first distance corresponds to a first principal component identified by performing a principal component analysis on the KPI data points associated with the fault in the cluster, and the second distance is identified by performing a principal component analysis on the KPI data points in the largest cluster.

[0147] 6. The computer-implemented method of clause 5, wherein the second distance is a predetermined percentage of a length of a first principal component identified by performing the principal component analysis on the KPI data points in the largest cluster.

[0148] 7. The computer-implemented method of any one of aspects 1 to 3, wherein the identifying a plurality of KPI data subsets of clusters comprises:

[0149] performing independent component analysis on the KPI data points in the cluster to identify a plurality of independent components, each independent component of the plurality of independent components being associated with one or more KPIs;

[0150] Each of the plurality of KPI data subsets corresponds to an independent component of the plurality of independent components, whereby KPI data of each of the one or more KPIs of the independent component exceeds a corresponding threshold associated with the KPI.

[0151] 8. The computer-implemented method of any preceding aspect, wherein each of the identified failure modes is associated with one or more KPIs.

[0152] 9. The computer-implemented method of any preceding aspect, further comprising supplementing the KPI data with manually generated KPI data associated with product units that do not meet specifications.

[0153] 10. A computer-implemented method of classifying product units subjected to a process performed by an apparatus, the method comprising:

[0154] Receiving the classification model according to any of the preceding aspects to obtain a threshold value for each KPI associated with at least one failure mode;

[0155] For each of said product units that is subjected to a process performed by the device:

[0156] determining a likelihood of each of the at least one failure mode based on the KPI data for the product unit and the threshold value assigned to each KPI associated with the at least one failure mode; and

[0157] Classification of the product unit is performed based on the likelihood of the at least one failure mode.

[0158] 11. The computer-implemented method of clause 10, wherein the classification of the product unit comprises a prediction of whether the product unit is in-spec or out-of-spec.

[0159] 12. The computer-implemented method of clause 11, wherein the classification of the product unit includes a confidence level of the prediction.

[0160] 13. The computer-implemented method of any of aspects 10 to 12, wherein performing the classification of the product unit comprises, for each failure mode, comparing the likelihood to a corresponding predetermined failure mode threshold to determine whether the failure mode predicts that the product unit does not meet specifications.

[0161] 14. The computer-implemented method of clause 13, wherein if only a single failure mode has a likelihood exceeding its predetermined failure mode threshold, the classification comprises (i) a prediction that the product unit is not meeting specifications, and (ii) the one or more KPIs associated with the single failure mode.

[0162] 15. The computer-implemented method of aspect 13, wherein the classification comprises a weighted prediction that the product unit will not meet specification for each of the plurality of failure modes if the plurality of failure modes have a likelihood exceeding a predetermined failure mode threshold thereof.

[0163] 16. The computer-implemented method of any preceding aspect, wherein the apparatus is a lithographic apparatus and the product unit is a semiconductor wafer.

[0164] 17. A computer-implemented method of classifying product units undergoing a process performed by an apparatus, the method comprising:

[0165] receiving KPI data obtained as a result of a plurality of product units being subjected to the process, the KPI data being associated with a plurality of components of the device and including data associated with a plurality of KPIs;

[0166] clustering the KPI data into clusters to identify at least one cluster;

[0167] analyzing the at least one cluster to identify a plurality of failure modes associated with the device, wherein the analyzing includes identifying, for one or more of the at least one cluster, a plurality of subgroups of KPI data associated with failures of a product unit, each subgroup of the plurality of subgroups of KPI data being associated with a failure mode of the plurality of failure modes;

[0168] For each identified failure mode, assigning a threshold value to each KPI associated with the failure mode; and

[0169] For each product unit in the plurality of product units:

[0170] determining a likelihood of each of the plurality of failure modes based on the KPI data for the product unit and the threshold value assigned to each KPI associated with one of the plurality of failure modes; and

[0171] Classification of the product unit is performed based on the likelihood of each failure mode of the plurality of failure modes.

[0172] 18. A non-transitory computer-readable storage medium comprising instructions which, when executed by a processor of an apparatus, cause the processor to perform the method according to any preceding aspect.

[0173] 19. An apparatus comprising a processor configured to perform the method according to any one of aspects 1 to 17.

[0174] Although specific reference may be made herein to the use of lithographic apparatus in IC manufacturing, it should be understood that the lithographic apparatus described herein may have other applications. Possible other applications include the manufacture of integrated optical systems, guidance and detection patterns for magnetic domain memories, flat panel displays, liquid crystal displays (LCDs), thin film magnetic heads, etc.

[0175] Although specific reference may be made herein to embodiments of the invention in the context of a lithographic apparatus, embodiments of the invention may be used for other apparatus. Embodiments of the invention may form part of a mask inspection apparatus, a metrology apparatus, or any apparatus that measures or processes an object such as a wafer (or other substrate) or a mask (or other pattern forming device). These apparatuses may generally be referred to as lithographic tools. Such lithographic tools may use vacuum conditions or ambient (non-vacuum) conditions. Embodiments of the invention are not limited to the case where the apparatus is a lithographic tool, and instead extend to classifying product units that undergo a process performed by any apparatus.

[0176] While the above may have made specific reference to the use of embodiments of the invention in the context of optical lithography, it will be appreciated that the invention is not limited to optical lithography and may be used in other applications (eg, imprint lithography) where the context permits.

[0177] Although specific embodiments of the present invention have been described above, it will be appreciated that the present invention may be practiced in other ways than those described. The above description is intended to be illustrative, not restrictive. Therefore, it will be appreciated by those skilled in the art that modifications may be made to the present invention as described without departing from the scope of the claims set forth below.

Claims

1. A computer-implemented method of determining a classification model including a KPI threshold, the classification model being used to classify a product unit undergoing a process performed by a device, the method comprising: receiving KPI data obtained as a result of a plurality of product units being subjected to the process, the KPI data being associated with a plurality of components of the device and including data associated with a plurality of KPIs; clustering the KPI data into clusters to identify at least one cluster; analyzing the at least one cluster to identify a plurality of failure modes associated with the device, wherein the analyzing includes identifying, for one or more of the at least one cluster, a plurality of subgroups of KPI data associated with failures of a product unit, each subgroup of the plurality of subgroups of KPI data being associated with a failure mode of the plurality of failure modes; as well as The classification model is determined by assigning, for each identified failure mode, a threshold value to each KPI associated with the failure mode.

2. The computer-implemented method of claim 1, wherein: The method includes projecting the KPI data into a low-dimensional space before performing the clustering.

3. The computer-implemented method of claim 2, wherein: The method includes projecting the KPI data into a 2-dimensional space.

4. The computer-implemented method of claim 1 , wherein: Multiple KPI data subgroups of the identified clusters include: A first distance between KPI data points associated with the fault in the cluster is determined to exceed a second distance associated with all KPI data points in a largest cluster of the at least one cluster.

5. The computer-implemented method of claim 4, wherein: The first distance corresponds to a first principal component identified by performing a principal component analysis on the KPI data points associated with faults in the clusters, and the second distance is identified by performing a principal component analysis on the KPI data points in the largest cluster.

6. The computer-implemented method of claim 5, wherein: The second distance is a predetermined percentage of the length of a first principal component identified by performing the principal component analysis on the KPI data points in the largest cluster.

7. The computer-implemented method of claim 1 , wherein: Multiple KPI data subgroups of the identified clusters include: performing independent component analysis on the KPI data points in the cluster to identify a plurality of independent components, each independent component of the plurality of independent components being associated with one or more KPIs; Each of the plurality of KPI data subsets corresponds to an independent component of the plurality of independent components, whereby KPI data of each of the one or more KPIs of the independent component exceeds a corresponding threshold associated with the KPI.

8. The computer-implemented method of claim 1, wherein: Each of the identified failure modes is associated with one or more KPIs.

9. The computer-implemented method of claim 1, further comprising supplementing the KPI data with manually generated KPI data associated with product units that do not meet specifications.

10. A computer-implemented method of classifying product units subjected to a process performed by an apparatus, the method comprising: Receiving a classification model according to any one of claims 1 to 6 to obtain a threshold value of each KPI associated with at least one failure mode; For each of said product units that is subjected to a process performed by the device: determining a likelihood of each of the at least one failure mode based on the KPI data for the product unit and the threshold value assigned to each KPI associated with the at least one failure mode; as well as Classification of the product unit is performed based on the likelihood of the at least one failure mode.

11. The computer-implemented method of claim 10, wherein: The classification of the product unit includes a prediction of whether the product unit is in-spec or out-of-spec.

12. The computer-implemented method of claim 11, wherein: The classification of the product unit includes a confidence level of the prediction.

13. The computer-implemented method of claim 10, wherein: Performing the classification of the product unit includes, for each failure mode, comparing the likelihood to a corresponding predetermined failure mode threshold to determine whether the failure mode predicts that the product unit is not in compliance with specification.

14. The computer-implemented method of claim 13, wherein: If only a single failure mode has a likelihood exceeding its predetermined failure mode threshold, the classification includes (i) a prediction that the product unit is not meeting specifications, and (ii) the one or more KPIs associated with the single failure mode.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to perform the method of claim 1.

Images