Computer-implemented method for detecting anomalies in imaging dataset of wafer and system using such method

By optimizing the hyperparameters of the machine learning model in semiconductor wafer anomaly detection, combining the Lp-norm and the objective function of the discriminant loss function, the problems of low accuracy and high interference rate in the prior art are solved, and high-precision anomaly detection is achieved .

CN118984995APending Publication Date: 2024-11-19CARL ZEISS SMT GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380030166.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-25
Filing Date
2023-03-27
Publication Date
2024-11-19

Smart Images

  • Figure CN118984995A_ABST
    Figure CN118984995A_ABST
Patent Text Reader

Abstract

The invention relates to a computer-implemented method (10, 10 ') for detecting anomalies (15), the computer-implemented method comprising: selecting an imaging data set (12) of a wafer (120) and a hyper-parameter value defining a machine learning model (16) for anomaly detection; training and evaluating the machine learning model (16) by calculating an objective function value; one of the trained machine learning models is selected and applied to detect anomalies (15). The invention also relates to a computer-implemented method (10 '') for detecting anomalies (15) in an imaging data set (12) of a wafer (120), the computer-implemented method comprising: providing a sample of an anomaly detection image value distribution for each defect category; the anomaly detection image (20, 26, 26 ') is calibrated for anomaly localization by training a machine learning model, and a critical value is applied to the calibrated anomaly detection image (98, 108) to detect an anomaly (15).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 324,258 filed on March 28, 2022 and German Patent Application No. 102022113352.4 filed on May 25, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to a computer-implemented method for detecting anomalies in an imaging data set of a wafer. The present invention also relates to a machine-readable hardware storage device, a system for controlling wafer production in a semiconductor manufacturing plant, and a system for controlling the quality of wafers produced in a semiconductor manufacturing plant.

[0004] The present invention is not limited to computer-implemented methods for wafers, but can be applied to any other manufacturing objects. Background Art

[0005] Quality control (QC) is used to iteratively improve the production process for the final product quality. Due to its central role, QC technology needs to be not only versatile, efficient and flexible, but also able to adapt to changing production conditions and use cases. Since there is no information about potential defects, which may change by application or over time, QC systems must be based on broad assumptions to cover all defect types. Therefore, QC systems should be able to address cold start conditions.

[0006] Cold starts involve the problem of a learning system being unable to make any inferences about items for which it has not yet collected enough information. This problem often occurs in the semiconductor industry as processes and chip types are constantly being adapted. Therefore, cold starts are a common problem in machine learning systems involving automatic data modeling, as the machine learning model must be retrained from scratch every time the system parameters are modified.

[0007] Unsupervised machine learning techniques, such as autoencoders, successfully address the QC problem. During training, such techniques learn a compressed internal representation of a large amount of available "flawless" or "defect-free" data. As a result, the model is able to perfectly reconstruct defect-free image samples. During testing, defects in the input images are not faithfully reconstructed. Spatial regions with high reconstruction errors represent outliers with respect to the training data, also known as anomalies. Anomalies are local deviations of an imaging dataset from a previously defined norm, in this case, deviations from a normative semiconductor structure.

[0008] However, not all anomalies are defects: for example, anomalies may also include imaging artifacts, image acquisition noise, different imaging conditions, semiconductor structure changes within the standard range, rare semiconductor structures or changes due to imperfect lithography, changing manufacturing conditions or changing wafer processing, etc. Such anomalies that are not defects but are detected by some anomaly detection method are called nuisances.

[0009] Machine learning models based on unsupervised learning acquire information only from data without any human input, such as annotations, which may be labor-intensive, noisy, or impractical. However, expert knowledge is still required to supervise model training, i.e., to define various model parameters called hyperparameters, such as the design of the model and its complexity (e.g., number of layers, number of filters per layer, size of the autoencoder bottleneck), its regularization (e.g., strategy and amplitude), data preprocessing techniques, dataset diversity, and learning strategies (e.g., learning rate and number of epochs). Expert selection of such hyperparameters is very important to obtain high-quality machine learning models.

[0010] In the prior art, a hyperparameter optimization method is proposed to optimize the hyperparameter values ​​of a machine learning model, and by finding a set of optimal hyperparameter values, the validation loss expectation of the machine learning model is minimized. Such a machine learning model suitable for anomaly detection is, for example, an autoencoder.

[0011] Hyperparameter optimization methods have been proposed that automatically search for optimal hyperparameter values ​​for a machine learning model. Among them, a neural architecture search (NAS) method has been proposed that automatically searches for optimal hyperparameter values ​​that define a neural network architecture.

[0012] In these hyperparameter optimization methods, various techniques can be used to predict good hyperparameter candidates, such as those disclosed in “Max-value Entropy Search for Multi-Objective Bayesian Optimization” by S. Belakaria, A. Deshwal, and J. Doppa at the Conference on Neural Information Processing Systems 2019.

[0013] A known hyperparameter optimization framework called Optuna is disclosed, for example, in “2019, Optunna: A next-generation hyperparameter optimization framework” proposed by Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. et al., in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631.

[0014] For example, another known hyperparameter optimization method is disclosed in patent case US2020 / 0342329 A1. In this disclosure, the algorithm hyperparameters of the autoencoder are optimized by evaluating the quality of the training progress (i.e., the history of the loss function value). The hyperparameter optimization method is unsupervised and does not use any labeled training data.

[0015] Another hyperparameter optimization method is disclosed in US2021 / 0256392 A1, in which the hyperparameters are optimized by exploring the search space of the hyperparameters and comparing the performance differences of the corresponding anomaly detection networks.

[0016] However, it is not simple to apply the disclosed hyperparameter optimization method to anomaly detection machine learning models. Anomaly detection models are required to reconstruct defect-free parts of the image, for example, through mean squared error loss (L2 loss), rather than defects. Known hyperparameter optimization methods, such as Optuna or US2020 / 0342329A1, evaluate machine learning models using the same metric used during model training, which in most cases is preset to the L2 metric, i.e., the mean squared error of the prediction. However, Optuna and similar methods of the prior art are not suitable for defect or anomaly detection because they converge to a perfect reconstruction of the input image including all possible defects, and therefore anomaly or defect detection will not be feasible.

[0017] Since not all anomalies are defects, machine learning models for anomaly detection often suffer from low accuracy due to noise and high noise rates. As a result, anomaly detection methods applied to wafer imaging datasets may face the problem of very high noise rates n, which is the inverse of the accuracy rate p, i.e. n = 1-p, because too many and mostly irrelevant deviations will be found on the wafer surface. Therefore, the anomaly detection methods of the prior art require a lot of post-processing before they can be used for defect detection on the wafer surface. In addition, many anomaly detection methods do not produce pixel-accurate anomaly detection results.

[0018] An improved anomaly detection method was revealed in "Attention Guided Anomaly Detection and Localization in Images" by S. Venkataramanan, KC. Peng, RVSingh, A. Mahalanobis at ECCV 2020. The proposed method is based on the idea of ​​using attention mechanism to distinguish between non-anomaly areas and abnormal areas, so that the anomaly detection model can be trained on non-anomaly data, which produces better results. However, this method is not suitable for semiconductor images. In addition, the generated attention map is not pixel-accurate and therefore cannot meet the high precision requirements of the semiconductor field.

[0019] Another improved anomaly detection method is disclosed in "End-to-end Deep Semi-supervised Anomaly Detection (ESAD)" proposed by C. Huang, F. Ye, Y. Zhang, Y Wang, Q. Tian in Arxiv2020. The paper proposes a semi-supervised training for anomaly detection based on a small annotated dataset and a large amount of unannotated data. However, this method only performs image-level anomaly detection, indicating whether there is an anomaly in the image without locating it. Again, this method cannot meet the accuracy requirements in the semiconductor field.

[0020] A method for semantic segmentation was disclosed in "What's the point: semantic segmentation with Point Supervision" presented by A. Bearman, O. Russakovsky, V. Ferrari, L. Fei-Fei at the European Conference on Computer Vision 2016. This method addresses the problem of reducing user workload during image annotation by indicating single points of objects rather than pixel-accurate segmentation.

[0021] Methods for automatically detecting defects (with the required accuracy in the semiconductor field) include anomaly detection algorithms that are based on the die-to-die or die-to-database principle. The die-to-die principle compares parts of a wafer to other parts of the same wafer to find deviations from a typical or average wafer design. The die-to-database principle compares parts of a wafer to ideal simulation data from a database (e.g., a CAD file of the wafer) to find deviations from the ideal data.

[0022] Therefore, for defect detection methods like die-to-die or die-to-database, the intermediate output is usually a difference image that shows the difference between the expected image (comparison dataset) and the actual image. Anomaly suggestions are usually obtained by setting a threshold on the difference image. In this way, pixel-accurate anomaly detection can be obtained. However, setting the threshold is a balancing act between maximizing the capture rate (real defects marked as anomalies) and minimizing the nuisance rate (imaging artifacts, noise, irrelevant defects, etc. marked as anomalies). This step can be laborious, especially when defects with different shapes, sizes, and appearances are to be detected.

[0023] Traditionally, setting thresholds is a manual process where experts filter by increasing the thresholds. At each step, newly flagged anomalies are analyzed and threshold windows for various defect classes are selected. Alternatively, experts provide some annotations for each defect class to extract the thresholds. However, both practices are intensive hunting operations and are ineffective for low-contrast defects due to the high noise rates generated.

[0024] Therefore, the problem of the present invention is to improve anomaly detection within a semiconductor chip. Another problem of the present invention is to improve the accuracy of a machine learning method for anomaly detection within a semiconductor chip. Another problem of the present invention is to reduce the user interaction required to select hyperparameter values ​​of a machine learning model for anomaly detection. Another problem of the present invention is to provide an anomaly detection method containing hyperparameter optimization. Another problem of the present invention is to make hyperparameter optimization applicable to a machine learning method for anomaly detection. Another problem of the present invention is to provide an evaluation metric for determining the performance of a machine learning model for anomaly detection. Another problem of the present invention is to provide an improved anomaly detection method, including an improved post-processing method, such as an improved critical value setting method. Summary of the invention

[0025] The present invention solves the problem of improving the accuracy of machine learning methods for anomaly detection, in particular for imaging datasets of wafers. Furthermore, the present invention aims to improve the accuracy of machine learning methods for anomaly detection without reducing the recall.

[0026] This object is achieved by the invention specified in the independent claim. Advantageous embodiments and further developments of the invention are specified in the dependent claims.

[0027] According to a first embodiment of the present invention, a method is provided to reduce the effort required to select hyperparameter values ​​of a machine learning model for detecting anomalies or defects in semiconductor wafers. The method provides a method for automatically optimizing at least one hyperparameter value of a machine learning model to obtain an optimized machine learning model for anomaly detection. Therefore, a computer-implemented method for detecting anomalies according to the first embodiment of the present invention comprises:

[0028] - Selecting an imaging dataset of a wafer;

[0029] - generating training data from the imaging dataset, and

[0030] -selecting an optimized machine learning model from one of at least two trained machine learning models based on the associated objective function value; and

[0031] - applying the optimized machine learning model to the imaging dataset of the wafer to detect anomalies,

[0032] The method step of selecting an optimized machine learning model comprises the following steps for each of at least two trained machine learning models:

[0033] - selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection;

[0034] - training a machine learning model controlled by the hyperparameter value on an input subset of the training data;

[0035] -Evaluate the trained machine learning model by calculating an associated objective function value of an objective function.

[0036] By selecting the optimized machine learning model from a plurality of different machine learning models, the accuracy of the anomaly detection method is improved, and at the same time, the workload required for the user to train the machine learning model is reduced.

[0037] A computer-implemented method for detecting anomalies according to a second embodiment of the present invention comprises:

[0038] - Selecting an imaging dataset of a wafer;

[0039] - generating training data from the imaging dataset;

[0040] -Iterate the following steps:

[0041] i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection;

[0042] ii. training the machine learning model defined by the hyperparameters based on the subset of the generated training data;

[0043] iii. Evaluate the trained machine learning model by calculating the associated objective function value of an objective function.

[0044] -Selecting one of the trained machine learning models based on the associated objective function value and applying it to the wafer imaging dataset to detect anomalies.

[0045] By selecting one of the trained machine learning models based on the associated objective function value, the accuracy of the detected anomaly is increased due to the optimized hyperparameters (e.g., architecture) of the machine learning model. At the same time, the workload required for the user to train the machine learning model is reduced. The trained machine learning model can be used to detect anomalies in an imaging dataset of a wafer. The imaging dataset can be acquired, for example, by a structured electron microscope.

[0046] In a machine learning model, a hyperparameter is a parameter whose value is used to control the learning process, but is not learned from data. In contrast, the values ​​of other parameters (usually node weights) are obtained by training the training data. According to an example of the first or second embodiment, the hyperparameter includes at least one of a design hyperparameter or an algorithm hyperparameter. Design hyperparameter refers to the design of the machine learning model. The design hyperparameter contains all hyperparameters related to the machine learning model architecture, such as the number of layers, the size of the layer, the type of layer, the size of the filter, the kernel size of the convolution layer, the type of convolution used, the upsampling scheme, the connection between the layers, the bottleneck size, the bottleneck filter size, etc. An example of a model or design hyperparameter is the topology and size of a neural network. In principle, the algorithm hyperparameter has no effect on the performance of the model, but it will affect the speed and quality of the learning process. Examples of algorithm hyperparameters include learning rate, mini-batch size, loss rate, the size and content of the training data set, and the type of loss function. Another example of an algorithm hyperparameter is the optimization algorithm used (stochastic gradient descent, Adam, RmsProp, etc.).

[0047] According to the first or second embodiment, the at least one hyperparameter defining the machine learning model for anomaly detection may include at least one of the following examples, but is not limited to these:

[0048] Design hyperparameters:

[0049] - Bottleneck size

[0050] - Bottleneck filter size (number of features in the bottleneck);

[0051] - Initial filter size (number of filters in the first layer of the network, other network features are scaled proportionally to the first layer);

[0052] - The type of convolution used;

[0053] - Sampling plan;

[0054] -Connections between layers;

[0055] - Number of layers within the model;

[0056] - Layer size within the model;

[0057] - Types of layers within the model;

[0058] - filter size;

[0059] - kernel size of convolutional layers;

[0060] Algorithm hyperparameters:

[0061] - Type and / or parameters of the loss function;

[0062] - initial learning rate;

[0063] -Learning rate decay factor;

[0064] - Use of momentum;

[0065] - the number of epochs;

[0066] - the regularization scale;

[0067] - size and content of the training set (number of images);

[0068] - a sample representing the dataset;

[0069] - loss rate;

[0070] -Use Nesterov accelerated gradient;

[0071] - Type of optimization algorithm.

[0072] In an example according to the first or second embodiment, the method according to the first or second embodiment further comprises the step of selecting a plurality of hyperparameters that together define a machine learning model for anomaly detection. For each selected hyperparameter, a set of associated hyperparameter values ​​is selected. Preferably, at least one hyperparameter to be optimized is related to the design of the machine learning model.

[0073] In an example of the method according to the first or second embodiment, the step of generating training data comprises expert annotation of anomalies of a subset of the imaging dataset.Therefore, an objective function suitable for selecting an optimized machine learning model or hyperparameter optimization may be defined.

[0074] According to an example of the first or second embodiment, the subset of the training data includes multiple samples of the training data. It may also include the entire generated training data. The size of the training data subset can increase with the increase of the number of iterations, thereby reducing the calculation time.

[0075] In an example of a method according to the first or second embodiment, the input data and training data for the machine learning model may consist of tiles of a specific size of an imaging data set. In an example according to the first or second embodiment, tiles (e.g., 2-D images or 3-D voxel arrays) are extracted from the imaging data set and input into the machine learning model. The tiles may include sufficient spatial range to detect anomalies. In an example according to the first or second embodiment, the tiles are at least as large as the expected anomalies and are combined with a spatial neighborhood range.

[0076] In an example according to the first or second embodiment, an autoencoder model is selected as the machine learning model, wherein the autoencoder machine learning model is trained to calculate a reconstructed subset of an imaging data set of a wafer without anomalies.

[0077] The autoencoder machine learning model is an unsupervised machine learning method that requires minimal user input. The autoencoder model is an artificial neural network used for unsupervised learning to learn effective encodings of unlabeled data. The autoencoder includes two main parts: an encoder that maps the input to the encoding, and a decoder that maps the encoding to the input reconstruction. The encoder neural network and the decoder neural network can be trained to minimize the difference between the reconstructed representation of the input data and the input data itself. The code can be a representation of the input data with a lower dimension and can therefore be considered a compressed version of the input data. In an example according to the first or second embodiment, the autoencoder is forced to approximately reconstruct the input, retaining only the most relevant aspects of the data in the reconstruction. Therefore, the autoencoder can be used for anomaly detection. Anomalies typically involve rare deviations from the norm in an imaging data set. Because they rarely occur, the autoencoder will not reconstruct such information, thereby suppressing anomalies in the imaging data set.

[0078] In an example according to the first or second embodiment, during the anomaly detection step according to the first or second embodiment, anomalies in the imaging dataset are detected based on a comparison between the imaging dataset and a comparison dataset. The comparison dataset contains information that enables anomaly detection when compared with the imaging dataset. In an example according to the first or second embodiment, the comparison dataset contains an ideal defect-free reconstructed imaging dataset. In an example according to the first or second embodiment, the reconstructed imaging dataset is reconstructed by an autoencoder or by principal component analysis. In an example according to the first or second embodiment, anomaly detection is performed using a distance indicator or a threshold operation applied to the comparison dataset.

[0079] Anomalies can then be detected by comparing the imperfect reconstruction of the imaging data set with the original imaging data set. Any difference between the input image and the reconstructed representation of the input image indicates the presence of an anomaly. A distance metric between the input image and the reconstructed representation of the input image can be used to quantify whether an anomaly exists. The greater the difference therebetween, the greater the likelihood that an anomaly is contained in the tile. Anomaly detection can include applying one or more critical values ​​to a difference image between the original image and the reconstructed image. In instances according to the first or second embodiments, local critical values ​​can be applied to a subset of the difference image. Further measurements can also be used for anomaly detection, such as the size, location or shape of the difference or its local distribution. Thereby, defects of interest are detected while suppressing deviations due to noise.

[0080] In an example according to the first or second embodiment, the method for detecting anomalies further comprises a step for defining an objective function suitable for selecting an optimized machine learning model or suitable for hyperparameter optimization. The objective function according to the example may comprise at least one model evaluation metric.

[0081] The model evaluation metric is a quantifiable expression of the characteristics of the trained machine learning model, and may include a measurement of at least one of the characteristics, which are related to the performance of the trained machine learning model, the quality of anomaly detection, the complexity of the trained machine learning model, the workload or cost of applying the trained machine learning model, etc. In an example according to the first or second embodiment, multiple model evaluation metrics may be defined and appropriately combined within the objective function. In one example, previously generated validation data is provided for evaluating the model evaluation metric.

[0082] In an example according to the first or second embodiment, the objective function includes at least two model evaluation indicators. The method using the objective function including at least two model evaluation indicators is particularly suitable for anomaly detection, because the combination of different model evaluation indicators can prevent the reconstruction of defects by the trained machine learning model (such as an autoencoder), ensuring that the comparison data set deviates sufficiently from the imaging data set when the defect occurs. In this way, manpower is also reduced, the performance of anomaly detection is maximized, and the quality, repeatability and stability of the results are improved. In addition, a cold start can be performed.

[0083] The objective function is a measure of the quality of the trained machine learning model and therefore also a measure of the quality of the hyperparameter values ​​selected for the model. Applying an objective function containing at least two different model evaluation metrics ensures that the optimization of hyperparameter values ​​according to the first or second embodiment does not produce a perfect reconstruction of the training data containing anomalies, because all model evaluation metrics are calculated simultaneously and therefore contribute to the objective function value of the objective function.

[0084] The objective function for hyperparameter optimization may include or be a loss function used as a model evaluation metric during machine learning model training. The model evaluation metric may also include or be an Lp-norm loss function, where p ≥ 1, such as an L2 loss function. The Lp-norm loss function may measure the deviation of a training data sample from a corresponding target data sample, such as the deviation of a training data sample from its anomaly-free reconstruction, such as the deviation of a training data sample from its reconstruction through an autoencoder. This ensures the correct reconstruction of anomaly-free image regions, thereby producing improved anomaly detection results.

[0085] In an example according to the first or second embodiment, at least one of the model evaluation metrics comprises a discriminative loss function that evaluates the difference between an expert annotation of anomalies and detected anomalies. The expert annotation comprises a label assigned to a pixel or region in the imaging dataset, the label indicating, for example, anomaly or no anomaly. Thus, the objective function may also or alternatively comprise a defect / non-defect discriminative loss L using some pixelwise annotation provided by an expert user. CE. In an example according to the first or second embodiment, the difference between the original sample and the encoded sample (the original encoding) is retained as a continuous value, but constrained to the range (0,1). This value is directly compared to the binary label (abnormal / no abnormality) via a cross entropy loss. Note that the examples provided do not need to include all defects in the dataset. Preferably, less than 10%, more preferably less than 1% of the training data is expert annotated. Furthermore, the expert annotations only assign labels to a subset of anomalies present in the imaging dataset and / or a subset of anomaly types present in the imaging dataset. In this way, the required expert user input is minimized and cold starts are possible, since no specialized expert user annotations are required if a new defect appears in the imaging dataset. It should also be noted that these amounts of training data are insufficient to train a standalone model.

[0086] According to an example of the first or second embodiment, expert annotations may be provided in multiple formats (e.g., bounding boxes, click points, auxiliary processing information, etc.), and the discrimination loss (L CE ). For example, bounding box annotations are associated with an overlap metric, namely the intersection-over-union loss L IOU .

[0087] In one example of the first or second embodiment of the present invention, expert annotations are given, for example as pixel annotations or bounding boxes. These expert annotations are facilitated using expert knowledge, for example based on a critical distance, critical dimension or pitch size, which is related to the minimum size of the structure on the wafer. In this way, it is sufficient for the expert to click on the center of the defect in the imaging data set, and a larger area corresponding to the critical distance, critical dimension or pitch size is automatically assigned as a defect around the click point.

[0088] To avoid overfitting, one of the multiple model evaluation metrics may further include a machine learning model complexity metric, such as Occam-razor penalty for machine learning model complexity. The model evaluation metric considers the number of floating point operations (L FLOP ), for example, the logarithm of this number. Alternatively, the number of layers and / or size of the trained neural network and / or the number of connections between neurons and / or other suitable hyperparameter values ​​can be used by the model evaluation metric to measure the complexity of the machine learning model.

[0089] In an example of the first or second embodiment of the present invention, the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and the objective function comprises a weighted sum of Lp-norm loss functions, where p ≥ 1, measuring the deviation of a training data sample from a corresponding target data sample (e.g., the deviation of a training data sample from its autoencoder reconstruction); and a discriminant loss function for evaluating the difference between the model prediction and the anomaly expert annotations. This particular combination of model evaluation metrics is considered to ensure very high accuracy anomaly detection results because it simultaneously 1) evaluates the accuracy of predictions for non-anomaly data (Lp-norm loss), 2) evaluates the accuracy of predictions for anomaly data (discriminant loss), and 3) prevents class imbalance due to the specially selected anomaly expert annotations.

[0090] In one example of the first or second embodiment of the invention, the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and the objective function comprises a weighted sum of Lp-norm loss functions, where p ≥ 1, measuring the deviation of a training data sample from a corresponding target data sample (e.g., the deviation of a training data sample from its autoencoder reconstruction); a discriminant loss function for evaluating the difference between anomaly expert annotations and model predictions; and a machine learning model complexity measure. Considering this particular combination of model evaluation metrics ensures particularly high-precision anomaly detection results because it simultaneously 1) evaluates the accuracy of predictions for non-anomaly data (Lp-norm loss); 2) evaluates the accuracy of predictions for anomaly data (discriminant loss); 3) prevents class imbalance due to specially selected anomaly expert annotations; and 4) prevents overfitting (complexity measure). In one example, the objective function comprises a weighted sum of three model evaluation metrics, for example,

[0091] f=w1L2+w2L CE +w3L FLOP

[0092] The weights w1, w2, and w3 can be set by the user. Usually, the first two weights are similar in size, while the third weight is at least ten times lower, for example, w1 = 10, w2 = 1, and w3 = 0.01. One or more weights can also be set to 0, for example, w3 can be 0, so that the objective function does not penalize the complexity of the model. Any other L p -norm is used to replace L2-norm. The discriminant loss function L CE can be replaced by any loss function that penalizes the difference image deviation and some ground truth data of anomalies in the imaging dataset, such as the intersection-over-union loss L IOU Other model performance metrics can be added. In addition, other options of the Optuna library can be selected.

[0093] In an example according to the first or second embodiment, the objective function also includes or replaces any of the above items to include a quality value as a model evaluation indicator to evaluate the quality of the trained machine learning model, wherein the user interface is configured to present information about the trained machine learning model to the user and let the user indicate the quality value. The information presented to the user may include accuracy, recovery rate, sample anomaly detection, hyperparameter values ​​related to the trained machine learning model, such as the design of the model, the filter of the model for one or more layers of learning, etc. If the objective function is minimized, if the quality of the trained machine learning model is very high, the user can select a lower value as the quality value. If the quality of the trained machine learning model is low, the user can select a higher value as the quality value.

[0094] In an example according to the first or second embodiment, the hyperparameter value is selected according to a sampling strategy that determines which hyperparameter value should be selected next based on previously selected hyperparameter values ​​and associated objective function values. In an example according to the first or second embodiment, the sampling strategy includes using a sampling algorithm.

[0095] An example of a sampling strategy includes a tree-structured Parzen estimator (TPE), which processes classification hyperparameters in a tree-structured manner. For example, the number of layers of a neural network and the number of neurons in each layer define a tree structure. For example, if the second layer does not exist, there cannot be a third layer, and setting the number of neurons only makes sense if that layer exists in the graph. Another example is optimizer selection for a machine learning model, as each optimizer can have its own set of parameters.

[0096] Preferably, the sampling strategy for selecting hyperparameter values ​​includes considering the hyperparameter values ​​and corresponding values ​​of the objective function from one or more previous iterations by optimizing a criterion selected from a group including expected improvement, maximum probability of improvement, and upper confidence limit. In this way, the performance of the method for selecting and optimizing the machine learning model is improved, and the anomaly detection results are improved.

[0097] According to an example of the first or second embodiment, the method further comprises improving the evaluation of the acquisition function. The acquisition function is a proxy model of the validation loss as a function of the hyperparameter value, which can fit previously obtained objective function values, also known as previous observations, and is configured to predict where the local optimum of the objective function may be. These methods are further referred to as sequential model-based optimization (SMBO). The proxy model can be at least part of the objective function. The use of the proxy model simplifies the objective function evaluation and reduces the computation time because the expensive step of evaluating the objective function is less performed.

[0098] For example, the improvement probability (PI) can be used as an improvement acquisition function, which evaluates the objective function f at the point that is most likely to improve the value. The objective function f will be minimized. Let f' denote the minimum value of f observed so far, and D denote the previous observation, i.e., the previously obtained objective function value. This then corresponds to the following utility function associated with evaluating f at a given point x (corresponding to a set of hyperparameters):

[0099]

[0100] The probability of improving the acquisition function is then the expected utility, a function of x:

[0101]

[0102] in denotes the density of the normal distribution, Φ is the cumulative distribution function of the normal distribution, and K(x,x) is its variance. The point x with the highest probability of improvement (maximum expected utility) is then selected as the next hyperparameter value to evaluate.

[0103] An alternative improvement acquisition function that takes into account the size of the improvement is the expected improvement (EI). Expected improvement evaluates f at the point that is expected to improve f' the most. This corresponds to the following utility function:

[0104] u(x)=max(0,f′-f(x)).

[0105] Then, the expected improvement acquisition function is the expected utility as a function of x:

[0106]

[0107] The point with the highest expected improvement (largest expected utility) is chosen. The expected improvement has two components, the first of which can be increased by reducing the mean function μ(x), and the second by increasing the variance K(x,x). These two terms can be interpreted as explicitly encoding the trade-off between exploitation (evaluation at points with low mean) and exploration (evaluation at points with high uncertainty). The exploitation-exploration trade-off is a classic consideration in such problems, and the expected improvement criterion automatically captures both as a result of the Bayesian decision theory treatment.

[0108] An alternative improved acquisition function is often called an upper confidence limit (UCB). It is usually described by maximizing f rather than minimizing f; however, in the case of minimization, the improved acquisition function takes the form

[0109] a UCB (x; β) = μ(x) - βσ(x)

[0110] where β>0 is a trade-off parameter, and is the marginal standard deviation of f(x). In the minimization range, it is best described as a lower confidence bound. Similarly, the UCB acquisition function contains explicit exploitation μ(x) and exploration σ(x) terms. Iterative applications of this UCB acquisition function will converge to the true global minimum of f.

[0111] For each sampling strategy described, at least one set of hyperparameter values ​​may be associated with a probability distribution that indicates the likelihood of each hyperparameter value being selected by the sampling strategy. The probability distribution may be predefined and therefore independent of the objective function value. The probability distribution that indicates the likelihood of each hyperparameter value being selected by the sampling strategy may be modeled based on prior knowledge based on the application, preferably based on imaging hardware settings or design knowledge, such as critical distance. In this way, prior knowledge in parameter space may be incorporated into the method step of selecting an optimized machine learning model in the first embodiment or the iterative step of the second embodiment. Therefore, the anomaly detection results are improved due to improved sampling strategies and reduced computation time.

[0112] In one instance, the sampling strategy comprises at least two different sampling strategies for at least two trained machine learning models in the first embodiment, or for machine learning models trained during the iteration step in the second embodiment, resulting in improved results due to a more thorough exploration of the hyperparameter value space.

[0113] Optionally, the step for selecting an optimized machine learning model according to the first embodiment or the step of iteration in the second embodiment includes a pruning strategy, such as a pruning algorithm, which determines whether the training of a given machine learning model should be continued or interrupted. In one instance, the pruning strategy includes an early stopping criterion. In a further instance, the pruning strategy includes an asynchronous continuous halving strategy. In a further instance, the machine learning model generated by sampling a set of hyperparameter values ​​can be first tested based on the objective function of a small subset of training data samples. In the case of low performance, the sampled hyperparameter values ​​can be discarded in advance, otherwise the size of the training data subset can be increased. This pruning strategy reduces computing time. By applying the pruning strategy, if the objective function does not show good results, the hyperparameter values ​​will be tested and discarded in advance. Therefore, computing time is saved, and the method is suitable for cold start scenarios where retraining is often required.

[0114] According to a third embodiment of the present invention, there is provided a computer-implemented method for detecting anomalies in an imaging data set of a wafer, wherein the imaging data set contains defects belonging to a plurality of defect classes. The method comprises the following steps:

[0115] - Producing anomaly detection images by applying anomaly detection methods to imaging datasets;

[0116] - Perform one or more iterations, at least one of which includes the following steps:

[0117] i. providing one or more samples of anomaly detection image value distribution for each defect category of the defect category subset;

[0118] ii. calibrating the anomaly detection image by at least one calibration method, the method comprising the following steps:

[0119] a. Based on one or more samples of the anomaly detection image value distribution, train a machine learning model for anomaly localization, preferably anomaly segmentation

[0120] (anomaly segmentation);

[0121] b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image;

[0122] - Applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing nuisance and highlighting defects in the anomaly detection image.

[0123] In a fourth embodiment of the present invention, a computer-implemented method for detecting anomalies in an imaging dataset of a wafer is provided, wherein the imaging dataset contains defects belonging to a plurality of defect classes.

[0124] - Producing anomaly detection images by applying anomaly detection methods to imaging datasets;

[0125] - providing one or more samples of anomaly detection image value distribution for each defect category of the subset of defect categories;

[0126] - detecting anomalies within the anomaly detection image using at least one calibration method, the method comprising the steps of:

[0127] o training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of the anomaly detection image values;

[0128] o applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image;

[0129] o Applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing noise and prominent defects in the anomaly detection image.

[0130] According to the method of the third or fourth embodiment of the present invention, the machine learning method for anomaly detection is more suitable for defect detection because it requires only a minimum amount of user input, thereby reducing noise in the anomaly detection image and highlighting anomalies or defects. Based on this enhanced or calibrated anomaly detection image, anomalies can be robustly detected by applying only a single threshold.

[0131] According to an aspect of the third or fourth embodiment, the method further comprises a step of automatically setting a critical value by using available auxiliary information (e.g., information related to wafer structure dimensions, such as critical distance, critical dimension, pitch dimension, etc.) or by providing annotated defects of at least a subset of defect categories. To this end, the method according to the third or fourth embodiment is about the workflow of a solution that is robust to defect size and / or contrast and calibrates anomaly detection images (e.g., difference images) to highlight defects of interest while suppressing defects due to noise.

[0132] The concept of "anomaly localization" refers to any method of calculating the anomaly location, such as anomaly segmentation method, semantic anomaly segmentation method, anomaly detection method, classification method, regression method, distribution sample finding method, etc.

[0133] The number of defect classes may relate to a single defect class, multiple defect classes, or all defect classes present in the imaging data set.

[0134] The concept of "anomaly detection image" refers to an image output of an anomaly detection method that indicates anomalies, such as through pixel-by-pixel marking or bounding boxes. In an example of the third or fourth embodiment, the anomaly detection image is a difference image between an imaging data set and a comparison data set, such as a reconstruction of the imaging data set. In an example, the comparison data is based on a die-to-die principle or a die-to-database principle.

[0135] In an example of the third or fourth embodiment, the comparison dataset may include a reconstructed representation of the imaging dataset generated by training a machine learning autoencoder on the imaging dataset or a subset thereof, and applying the trained autoencoder to the imaging dataset to obtain a reconstructed imaging dataset. Such autoencoders and generation of reconstructed imaging datasets are discussed in the description of the first or second embodiment of the present invention.

[0136] Thus, the quality of the anomaly detection image is improved. Principal component analysis can be used to generate a reconstructed dataset instead of an autoencoder.

[0137] In the example of the third or fourth embodiment, the abnormality detection method is the method according to the first or second embodiment of the present invention.

[0138] In an example of the third or fourth embodiment, the at least one calibration method is selected from a group of calibration methods. In an example, for each calibration method in the group of calibration methods, the steps of determining a calibrated anomaly detection image and applying a single critical value to the calibrated anomaly detection image are iteratively repeated.

[0139] In an example of the third or fourth embodiment, the method further comprises the step of selecting one or more values ​​of the distribution domain of the abnormality detection image values ​​as a threshold value, for example, by user input. The threshold value may include a minimum value and a maximum value of the distribution domain of the abnormality detection image values, such as a minimum value and a maximum value of the intensity value of the abnormality or defect. Through this method, multiple adapted threshold values ​​may be applied in combination with the filter.

[0140] For example, calibrating the anomaly detection image by applying the at least one calibration method comprises the following steps:

[0141] - for each defect category of the subset of defect categories, calculating an intermediate calibrated anomaly detection image by adapting the anomaly detection image values ​​of the current defect category based on the selected one or more threshold values ​​of the anomaly detection image values;

[0142] - Apply one or more filters for the current defect category, such as a size filter;

[0143] And finally

[0144] - Producing a calibrated anomaly detection image by applying an operator to all intermediate calibrated anomaly detection images.

[0145] In this specification, the term "subset" of a set refers to a single, some, or all of the elements in the set.

[0146] In an example of the third or fourth embodiment, the operator is selected from the group consisting of pixel-by-pixel summation, pixel-by-pixel average, pixel-by-pixel minimum, pixel-by-pixel maximum, and pixel-by-pixel scaling. In this way, the final calibrated anomaly detection image contains defects from different defect categories, each of which is extracted from the original anomaly detection image based on a set of different thresholds and filters. Using the maximum operator can retain as many anomalies as possible (provided that the label value of the anomaly is higher than the background), while the minimum operator will further reduce noise and interference.

[0147] According to a second instance of the third or fourth embodiment, the method includes the step of providing annotated defects. The annotated defects are used to automatically set the required critical value to be applied to a calibrated anomaly detection image. According to the second instance, only a small number of annotated defects of a few defect categories are required, while other defect categories may not be annotated. The method according to the instance relies on the following assumption: the annotated defects cover the appearance spectrum of all defect categories. A small number of defect categories means that a subset of defect categories contains less than 50% of all defect categories, preferably less than 30% of all defect categories, more preferably less than 20% of all defect categories, and most preferably less than 10% of all defect categories. A small number of annotated defects in each category means that the number of annotated defects in each defect category is more than 5 but less than 20, preferably more than 5 but less than 10. Using the method according to the second instance, the problem of defect labeling is solved, and the critical value for the calibrated anomaly detection image can be automatically set.

[0148] In the calibration method according to the second example, the task of automatically finding the critical value can be formulated as a pixel-by-pixel localization problem, such as a pixel-by-pixel segmentation problem. Based on various assumptions, many different designs of pixel-by-pixel localization in the second embodiment can be thought of, for example, based on the type of annotation (bounding box, click point, pixel-level or image-level annotation, multi-user annotation, annotation from secondary sources, etc.), loss function (handling model complexity through regularization, including prior knowledge, handling category imbalance, etc.) or problem formulation (such as semantic segmentation, object detection, classification, regression, handling distribution samples, etc.).

[0149] In an example of the third or fourth embodiment, a machine learning model for anomaly localization is trained to optimize a loss function based on abnormal and non-abnormal samples. This allows for high-precision pixel-by-pixel detection of anomalies, which is important in the semiconductor field. Samples of the distribution of anomaly detection image values ​​for each defect category from a subset of defect categories can be used as abnormal samples. In a further example, the loss function is a semi-supervised loss function. Based on the semi-supervised loss function, the quality of anomaly detection can be improved because expert annotations can be considered during the training process of the calibration method. However, the required user workload remains at a low level because only a few annotations are required and most of the samples are selected automatically. In this specification, the term "foreground sample" is used as a synonym for "abnormal sample", and the term "background sample" is used as a synonym for "non-abnormal sample".

[0150] Specifically, samples of the abnormal detection image value distribution of each defect category from the defect category subset can be used as abnormal samples, and non-abnormal samples can be (automatically) selected from the remaining pixels of the abnormal detection image. If the abnormal detection image value of the abnormal detection image is below a critical value, the remaining pixels of the abnormal detection image can be selected as non-abnormal samples. Additionally or alternatively, each non-abnormal sample can be identified by its abnormal detection image value a i weighted by the weighting function w, in particular by the negative exponential weighting function w(a i )=exp(-a i ). This allows a large number of non-abnormal pixels to be automatically selected with minimal user effort. This also ensures that non-abnormal pixels are selected with high accuracy, because higher weights can be assigned to those pixels that have the lowest probability of being abnormal, for example, pixels with low abnormal detection image values, such as pixels with very low autoencoder reconstruction errors. Therefore, if only a small number of samples are selected from the distribution of abnormal detection image values ​​in each defect class of a subset of defect classes, it is possible to prevent samples that are not selected from these distributions from being mistakenly considered as non-abnormal samples. In addition, the user's workload can be reduced, because a small number of selected samples is sufficient to obtain high-accuracy abnormality detection.

[0151] The user only needs to provide a small number of pixel-level annotations for a subset of defect classes. The subset may include all defect classes. Preferably, the subset does not include all defect classes. More preferably, the subset includes only a small number of defect classes, such as less than 10% of the defect classes. The annotation process is simplified by utilizing available meta-information (e.g., critical dimension, critical distance, or pitch size). The user can provide a click point, which can be processed into a pixel-level annotation, such as by expanding the click point to a circle with a size corresponding to the critical dimension, critical distance, or pitch size representing the lower or upper limit of the smallest structure size on the wafer. Alternatively, the user can use a brush to mark the defective pixels. Based on these annotations, a machine learning anomaly localization model, such as an anomaly segmentation or anomaly detection model, can be trained. When applied to an anomaly detection image, this localization model marks each pixel as an anomaly or noise of interest. In this way, anomalies can be highlighted while noise can be reduced. Complementary post-processing techniques can be used to further reduce noise, such as histogram equalization, location- or size-based filtering, etc. The calibrated anomaly detection image can finally be thresholded using a single threshold, such as 0.5, to capture defects in the imaging dataset.

[0152] According to an example of the third or fourth embodiment of the present invention, during training, the machine learning model for anomaly localization is a machine learning model for anomaly segmentation. It takes a partially annotated anomaly detection image (e.g., a difference image) as input. At output, the model marks each pixel as a targeted anomaly or interference. Since not all input pixels are labeled by the user, the model learning is based on the following semi-supervised loss function:

[0153]

[0154] Where a is the abnormal detection image, y true Provide the user with pixel-level annotations from the set {unannotated = 0, anomaly = 1}, i.e., partially labeled anomaly detection images, y pred is the label predicted by the model. The loss function L is the sum of all pixel losses, which is a weighted cross entropy loss function defined as:

[0155]

[0156] The pixel-level weight w i This enables semi-supervised learning. In this paper, the anomaly The pixels of which are learned to be segmented as anomalies. Due to the negative index, the unannotated pixels is considered non-anomalous (at low a i value) or is ignored (in the case of high a i ). Therefore, the segmentation model suppresses noise and highlights anomalies, thereby improving the accuracy of anomaly detection. The advantage of semi-supervised learning methods is that only a small amount of user annotation is required, and most of the training data is automatically selected based on the anomaly detection image values. In this way, the user workload is limited and cold starts are possible.

[0157] Alternatively, various other loss functions can be used to train the anomaly localization machine learning model, such as Kullback-Leibler divergence loss function, L1 or L2 loss function, etc.

[0158] An example of the third or fourth embodiment provides a method for calibrating a difference image, allowing high recovery and manageable accuracy to be obtained by applying a single threshold to a calibrated anomaly detection image. Therefore, the machine learning model of the third or fourth embodiment includes method steps of setting, training, and applying a nonlinear filter configured to amplify defects while suppressing interference in a difference image for semiconductor defect detection.

[0159] In an example according to the third or fourth embodiment of the present invention, the user interface is configured to let the user indicate the location of a small number of defects of each category of a subset of defect categories in the anomaly detection image. To this end, the user interface is configured to allow the user to indicate the location of the defect by selecting a single pixel (preferably a pixel in the center area) of the defect, and to automatically generate an annotation of the defect by selecting an area, preferably a circle, around the selected single pixel, and sampling the anomaly detection image value from the area around the selected pixel. Preferably, the size of the area is selected based on auxiliary information such as critical size and / or critical distance and / or pitch size. The critical distance and critical size are related to the minimum size of the structure on the wafer, while the pitch size is related to the minimum distance between the structures, and can therefore be understood as an upper limit on the minimum size of the wafer structure. It can therefore be assumed that the anomaly is at least as large as the smallest structure on the wafer. Therefore, the anomaly area generated around the selected pixel can be automatically selected as the minimum size of the wafer structure relative to the critical size, critical distance or pitch size (converted to pixels). In this way, the accuracy of anomaly detection is improved by automatically increasing the number of marked samples. In addition, limiting the size of the anomaly area around the selected pixel according to the size of the wafer structure prevents incorrect marking, especially for very small structures on the wafer. Furthermore, since one click is sufficient to annotate defects, the annotation workload of users is greatly reduced. Therefore, cold start becomes feasible because despite the limited availability of training data at the beginning of training, a few selected samples are sufficient to train the machine learning model for anomaly localization.

[0160] The anomaly detection method according to the first or second embodiment, and the anomaly detection method according to the third or fourth embodiment can be trained in many variations. The method can be trained jointly as a single module, which is easier to maintain and evaluate. On the other hand, independent modules have the advantages of versatility and separation of expert opinions.

[0161] It should be noted that the computer-implemented method for detecting anomalies in an imaging data set may include hyperparameter optimization according to any instance of the first or second embodiment, and anomaly detection image calibration according to any instance of the third or fourth embodiment simultaneously. All features according to the embodiments or instances may also be applied to the combined method.

[0162] Anomaly detection can be the first step in a defect detection (and possibly classification) workflow. In particular, in order to obtain a method suitable for cold starts, anomaly detection is a valuable first step in scanning large amounts of data, so that only samples that are likely to contain defects are presented to the user for annotation.

[0163] Additionally, one or more characteristics of the detected anomalies may be measured, such as their size, location or shape parameters, or the density of anomalies in a specific area or in the entire imaging data set. Based on such measurements, at least one wafer manufacturing parameter may be controlled based on the one or more measured characteristics, or the quality of the wafer may be assessed based on the one or more measured characteristics and at least one quality assessment rule.

[0164] Therefore, the detected anomaly can be used to control the quality of wafers produced in a semiconductor manufacturing plant, or to control the production process of wafers in a semiconductor manufacturing plant.

[0165] Furthermore, one or more machine-readable hardware storage devices may include instructions executable by one or more processing devices to perform operations comprising any of the methods disclosed herein.

[0166] An inspection system for controlling the quality of wafers produced in a semiconductor manufacturing plant comprises the following features: an imaging device adapted to provide an imaging data set of the wafer; a selective graphical user interface configured to present data to a user and obtain input data from the user; one or more processing devices; and one or more machine-readable hardware storage devices containing instructions executable by the one or more processing devices to perform operations comprising any of the methods disclosed herein, including assessing the quality of the wafer based on one or more measurements and at least one quality assessment rule.

[0167] A system for controlling the production of wafers in a semiconductor manufacturing plant includes the following features: a production wafer assembly for producing wafers controlled by at least one manufacturing process parameter; an imaging device adapted to provide an imaging data set of the wafer; a selective graphical user interface configured to present data to a user and obtain input data from the user; one or more processing devices; and one or more machine-readable hardware storage devices containing instructions executable by the one or more processing devices to perform operations including a method including controlling at least one wafer manufacturing process parameter based on one or more measurements.

[0168] According to the embodiments described herein, various imaging modalities may be used to acquire imaging data sets for defect detection and classification. Together with the various imaging modalities, different imaging data sets may be obtained. The imaging data set may include one or more multi-sensory images. The imaging data set may be a multi-beam SEM image or a focused ion beam image, such as produced by a helium ion microscope (HIM). The imaging data set may include a two-dimensional image, a three-dimensional image, a sliced ​​three-dimensional image, or a multi-sensory fusion image. For example, the imaging data set may include a two-dimensional image. In this article, a multi-beam SEM may be used. The multi-beam SEM uses multiple beams to simultaneously acquire images in multiple fields of view. For example, no less than 50 beams, or even no less than 90 beams, may be used. Each beam covers a separate portion of the wafer surface. Therefore, a large number of imaging data sets may be acquired in a short time. Typically, 4.5 billion pixels are acquired per second. For example, a one square centimeter wafer may be imaged with a pixel size of 2nm, thereby generating 25 trillion pixels (terapixel) of data. Other examples of imaging data sets containing 2D images will involve imaging modalities such as optical imaging, phase contrast imaging, X-ray imaging, and the like. The imaging data set may also be a volumetric 3D data set, which can be processed layer by layer or as a three-dimensional volume. In this article, a cross-beam imaging device including a focused ion beam (FIB) source, an atomic force microscope (AFM) or a scanning electron microscope (SEM) can be used. A multi-mode imaging data set can be used, for example, a combination of x-ray imaging and SEM. The imaging data set 22 may additionally or alternatively include an aerial image acquired by an aerial imaging system. The aerial image is the radiation intensity distribution at the substrate level. It can be used to simulate the radiation intensity distribution generated by the lithography mask 14 during the lithography process. The aerial image measurement system can be equipped with a staring array sensor, a linear scanning sensor, or a time-delay integration (TDI) sensor, for example.

[0169] Although examples and embodiments of the present invention are described with respect to semiconductor wafers, it should be understood that the present invention is not limited to semiconductor wafers, but may also be applied, for example, to masks used in semiconductor manufacturing or in various other fields, for example, for manufacturing components or anomaly detection in biological samples.

[0170] The present invention described by way of examples and embodiments is not limited to these embodiments and examples, and those skilled in the art can implement the present invention by various combinations or modifications thereof. In the following, advantageous exemplary embodiments of the present invention are described and schematically shown in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0171] Figure 1 A schematic defective cell structure is shown containing multiple anomalies due to various defects;

[0172] Figure 2Flowcharts and results showing anomaly detection methods (such as autoencoders) applied to wafer imaging datasets;

[0173] Figure 3 A flowchart showing an abnormality detection method according to the first or second embodiment of the present invention;

[0174] Figure 4 Display results of anomaly detection methods obtained through common hyperparameter optimization methods;

[0175] Figure 5 A flowchart showing an abnormality detection method according to an example of the first or second embodiment of the present invention;

[0176] Figure 6 Displaying the results of an anomaly detection method according to an example of the first or second embodiment of the present invention based on an objective function containing at least two different model evaluation indicators;

[0177] Figure 7 Illustrate the sampling strategy of the tree-structured Parzen estimator;

[0178] Figure 8 Illustrate the pruning strategy of the asynchronous continuous halving algorithm;

[0179] Fig. 9 Displays the evolution of the objective function values ​​of different machine learning models defined by the hyperparameters chosen according to the sampling strategy of the tree-structured Parzen estimator;

[0180] Fig.10 A flowchart showing a computer-implemented method for detecting anomalies in an imaging data set of a wafer according to the third or fourth embodiment of the present invention;

[0181] Fig.11 A flowchart showing a computer-implemented method for detecting anomalies in an imaging data set of a wafer according to an example of the third or fourth embodiment of the present invention;

[0182] Fig.12 showing the steps of a computer-implemented method for detecting anomalies in an imaging data set of a wafer according to the third or fourth embodiment of the present invention;

[0183] Fig.13 showing a comparison of an anomaly detection image obtained by the computer-implemented method for detecting anomalies in an imaging data set of a wafer according to the third or fourth embodiment of the present invention with a calibrated anomaly detection image;

[0184] Fig.14 A confusion matrix showing an anomaly detection image obtained by the computer-implemented method for detecting anomalies in an imaging data set of a wafer according to the third or fourth embodiment of the present invention and a calibrated anomaly detection image;

[0185] Fig.15 A system for controlling wafer quality in a semiconductor manufacturing plant is schematically illustrated;

[0186] Fig.16 A system for controlling wafer production in a semiconductor fabrication plant is schematically illustrated. DETAILED DESCRIPTION

[0187] Figure 1 An illustrative defective cell structure 11 is shown containing a plurality of anomalies 15. Anomalies 15 are local deviations of the imaging data set 12 from a previously defined norm, here a deviation from a normative semiconductor structure.

[0188] Figure 2 A flow chart and results of an anomaly detection method (such as an autoencoder) applied to an imaging dataset 12 of a wafer 120 are shown. The input 14 of the method comprises an imaging dataset 12 of a wafer 120, which comprises one or more images of the wafer 120. Based on the imaging dataset 12, a machine learning model 16 is trained. The machine learning model 16 may be based on a die-to-database principle or a die-to-die principle. Based on the imaging dataset 12, the autoencoder model is trained based on a die-to-die principle. The autoencoder learns a compressed internal representation of a large amount of available "flawless" or "defect-free" data. Therefore, the machine learning model 16 is able to perfectly reconstruct defect-free image samples. During testing, the contaminated input images were not reconstructed faithfully. Spatial regions with high reconstruction errors represent outliers with respect to the training data, also referred to as anomalies 15. Based on the autoencoder model, a comparison dataset in the form of a reconstruction 18 of the input 14 is calculated. The defect 23 in the imaging dataset 12 is not reconstructed by the autoencoder, so the difference between the reconstruction 18 (comparison dataset) and the input 14 contains the defect 23 in the imaging dataset 12. Since any imaging artifacts, such as noise, varying semiconductor structures or imperfect lithography, etc., also cause differences in the difference image, not all of these differences are from defects 23. Therefore, defects 23 only constitute a subset of anomalies 15. Anomalies 15 that are not defects 23 but are detected by some anomaly detection method are called interferences. The difference image shows the anomalies 15 in the imaging data set 12. Therefore, the difference image is the anomaly detection image 20.

[0189] Based on the input image 22 without the defect, a reconstructed image 24 is obtained, which differs only due to noise, as shown in the difference image 26. The input image 22' containing the defect 23 is reconstructed in the reconstructed image 24', except that the area containing the defect 23 is only partially reconstructed. Therefore, the difference image 26' contains a deviation from 0 at the defect location. The defect can be located by applying a threshold to the difference image 26'.

[0190] An autoencoder is a machine learning model based on unsupervised learning, and therefore acquires information only from the input data without any human input. However, expert knowledge is still required to supervise the model training by defining hyperparameter values ​​for the hyperparameters that define the machine learning model, such as the bottleneck size. In order to reduce the required user interaction and improve the reconstruction results of the autoencoder, such as Figure 3 As shown, the computer-implemented method 10 for detecting anomalies 15 according to the first or second embodiment of the present invention includes the following steps: in a data selection step 19, an imaging data set 12 of a chip 120 is selected; in a training data generation step 21, training data is generated from the imaging data set 12, and the following steps are iterated: in a hyperparameter value selection step 25, based on a sampling strategy, a hyperparameter value is selected from an associated set of hyperparameter values, or multiple hyperparameter values ​​are selected from multiple associated sets of hyperparameter values, and the hyperparameter value corresponds to at least one hyperparameter of a machine learning model 16 defined for detecting anomalies 15; in a training step 27, a machine learning model 16 defined by hyperparameters 48 is trained based on a subset of the generated training data; in a model evaluation step 29, the trained machine learning model 16 is evaluated by calculating the associated objective function value of the objective function; and in a model selection step 31, one of the trained machine learning models is selected based on the associated objective function value and applied to the imaging data set 12 of the chip 120 to detect anomalies 15.

[0191] The objective function ensures that the hyperparameter or multiple hyperparameters are selected so that the machine learning model defined by the hyperparameter or multiple hyperparameters correctly reconstructs the imaging dataset 12, except for the area containing the defect 23. To this end, the objective function may include a model evaluation metric in the form of an Lp-norm loss function, where p ≥ 1, which ensures the correct reconstruction of the imaging dataset 12 without anomalies by penalizing deviations. For example, the L2-norm loss function, i.e., the mean squared error loss function, is selected by default as the objective function of most hyperparameter optimization methods.

[0192] However, if the objective function for hyperparameter optimization contains only a model evaluation metric based on Lp-norm, the autoencoder will also learn to reconstruct any defects 23 contained in the imaging dataset 12. Therefore, the defects 23 contained in the input image are at least partially reconstructed, and the difference image shows only slight deviations from 0 at the defect location or no deviation at all, as shown in FIG. Figure 4 shown.

[0193] exist Figure 4, the results of the anomaly detection method obtained by a common hyperparameter optimization method are shown. The input image 30 is reconstructed by the autoencoder machine learning method 16, which is obtained by a standard hyperparameter optimization method based on an objective function containing only the Lp-norm indicator. This produces a comparison dataset in the form of a reconstructed image 32 that includes all defects 23, so the anomaly detection image 28 in the form of a difference image does not show any defect 23 present in the imaging dataset 12 as an anomaly 15.

[0194] Figure 5 A flowchart of an anomaly detection method 10' according to an example of the first or second embodiment of the present invention is shown. The method comprises: a hyperparameter optimization unit 43, which performs a training data generation step 21 to generate training data from an imaging data set 12, a hyperparameter value selection step 25 to select a hyperparameter value, a training step 27 to train a machine learning model 16 based on the selected hyperparameter value, a model evaluation step 29 to evaluate the trained machine learning model through an objective function 46, and a model selection step 31 to select one of the trained machine learning models 16 based on the associated objective function value.

[0195] The machine learning model for anomaly detection trained according to the first or second embodiment of the present invention is based on the die-to-die or die-to-database principle. In both cases, a large amount of preferably defect-free data 34 is required to train the machine learning model 16, such as an autoencoder or principal component analysis.

[0196] The hyperparameter optimization unit 43 requires an objective function 46, which is used to evaluate each trained machine learning model 16. The objective function 46 includes one or more model evaluation indicators. In an example of the first or second embodiment of the present invention, the objective function includes at least two different model evaluation indicators 46 to prevent the trained machine learning model from also reconstructing defects 23 and anomalies 15.

[0197] In addition to L p -norm loss function, a better model evaluation metric is the discriminant loss function L CE , which also requires some expert annotations 36. The expert annotations 36 are given, for example, as pixelated annotations or bounding boxes. These expert annotations 36 can be facilitated using expert knowledge 38, for example based on critical distances, critical dimensions or pitch sizes, which are related to the minimum size of the structures on the wafer. In this way, it is sufficient for the expert to click on the center of the defect 23 in the anomaly detection image or imaging dataset 12, and a larger area corresponding to the critical distance, critical dimension or pitch size is automatically assigned as the defect 23 around the click point.

[0198] The hyperparameter optimization unit 43 includes a sampler 42 and a pruner 44. The sampler 42 is used to execute the sampling strategy by selecting a hyperparameter value for a hyperparameter 48 from an associated set of hyperparameter values. In one example, expert knowledge 38 is applied by introducing a hyperparameter range 40 for the hyperparameter. From these ranges, the hyperparameter value is selected by the sampler 42. Additionally or alternatively, the expert knowledge 38 can be applied by introducing a probability distribution indicating the likelihood that each hyperparameter value is selected by the sampling strategy, for example, based on imaging hardware settings or design knowledge, such as critical distance, critical size and / or spacing size. Each hyperparameter value corresponds to at least one hyperparameter 48 that defines a machine learning model for detecting anomalies 15. For example, the hyperparameter 48 may refer to a neural network architecture (e.g., the number and size of layers or the size of an autoencoder bottleneck) or a learning rate under the machine learning model. The selective pruner 44 determines whether the training of the machine learning model 16 should be continued by selecting another hyperparameter value according to the sampling strategy, or whether the training should be interrupted, for example, by an early stopping criterion. In case of an interruption, new hyperparameter values ​​may be randomly selected, possibly with respect to a given hyperparameter range and / or probability distribution of hyperparameter values, and then training may be started again according to the sampling strategy.

[0199] The machine learning model 16 is trained on a sub-sampled dataset 50, which is sampled from the full imaging dataset 52. The size of the sub-sampled dataset 50 increases with the number of iterations performed by the hyperparameter optimization unit 43. In this way, it is feasible for the user to cold start on a small sub-sampled dataset 50 with little effort and in a short time. In addition, online learning and optimization become possible because new training data can be easily incorporated into the training process, such as training data containing new types of defects. Therefore, the training process becomes fast and easy to adapt. After selecting the training machine learning model that achieves the best objective function value, the optimized machine learning model 54 is finally trained on the full imaging dataset 52 to obtain the final machine learning model 56 for anomaly detection.

[0200] Figure 6 The results of the anomaly detection method according to the first or second embodiment of the present invention are shown, where at least two different model evaluation indicators are combined in the objective function 46 to obtain improved results. In addition to the Lp-norm, such as the L2-norm that penalizes the deviation of the reconstructed image 32' from the input image 30', the objective function 46 contains a defect / non-defect discrimination loss function L based on several pixel-by-pixel annotations provided by experts. CE. The expert points out the locations of defects 23, which the autoencoder should not reconstruct. Therefore, by using the cross entropy loss as an additional model evaluation metric in the objective function 46, the anomaly detection image 28' (difference image) can be compared with the binary labels pointed out by the user. The expert annotations 36 do not need to cover all types of defects 23 present in the imaging dataset 12, and the number of annotations is very small, such as less than 10%, preferably less than 1% of the training data samples. Since the annotations can be provided in a variety of formats (e.g., bounding boxes, click points, auxiliary process information, etc.), the discriminant loss (L needs to be adjusted accordingly. CE ). For example, bounding box annotations are associated with an overlap metric, namely the Intersection-over-union loss L IOU .

[0201] In an example of the first or second embodiment of the present invention, the objective function 46 further includes a model evaluation index, which includes an Occam-razor penalty for complexity to avoid overfitting. The model evaluation index can be based on the total number of floating-point operations (L FLOP ). Alternatively, the number of layers and / or size of the trained neural network and / or the number of connections between neurons and / or other suitable hyperparameter values ​​can be used as model evaluation indicators to measure the complexity of the machine learning model. Such complexity-based model evaluation indicators are optional.

[0202] In one example of the first or second embodiment of the present invention, the objective function thus comprises a weighted sum of three model evaluation indicators, e.g.

[0203] f=w1L2+w2L CE +w3L FLOP

[0204] The weights w1, w2, and w3 are selected by experts. Usually, the first two weights are similar in size, while the third weight is at least ten times lower, for example, w1 = 10, w2 = 1, and w3 = 0.01. One or more weights, such as weight w3, can be set to 0. Any other L p -norm is used to replace L2-norm. The discriminant loss function L CE It can be replaced by any loss function that penalizes the difference image deviation and some ground truth data of anomalies in the imaging dataset, such as the intersection-over-union loss L IOUThe intersection. In one example, the objective function also or alternatively includes a quality value as a model evaluation metric to evaluate the quality of the trained machine learning model 16, where the user interface 128 is configured to present information about the trained machine learning model 16 to the user and allow the user to indicate the quality value. Additional model evaluation metrics can be added or used to replace one or more of the above model evaluation metrics. In addition, all options of the backbone Optuna library can also be added.

[0205] Figure 6 Show as Figure 4 The input image 30' containing the defect 23 as shown. In the reconstructed image 32', the defect 23 is not reconstructed because the objective function includes at least two model evaluation metrics. While correctly reconstructing the background image, the reconstruction does not copy any defect 23 (white circles) as required. Therefore, the difference image as the anomaly detection image 28' contains the defect 23.

[0206] Figure 7 Illustrate the sampling strategy of the Tree-structured Parzen Estimator (TPE), which can be used to sample hyperparameter values by the sampler 42.

[0207] The Parzen density estimator is a non-parametric kernel density estimator used to estimate the probability density function of a random variable (hyperparameter value) given a set of observations x = x1,..., x n :

[0208]

[0209] where K h is a kernel with bandwidth h.

[0210] The idea of the Parzen estimator is close to Bayesian optimization but stands on the opposite theoretical basis. While Bayesian optimization attempts to solve p(y|x) (y is the value of the objective function, x is the hyperparameter value), the Parzen estimator models the trees p(x|y) and p(y).

[0211] Regarding Bayesian optimization, the first step of TPE is to start sampling the objective function 46 through random search to initialize the algorithm.

[0212] Then the observations are divided into two groups: the group with the best performance according to the objective function 46 (the good group 58, e.g., the upper quartile) and the remaining group (the bad group 60), and y* is defined as the splitting value of the two groups.

[0213] The probability in each of these groups has been modeled (Gaussian processing models the posterior probability), such as p(x|y) = l(x) if y < y* and p(x|y) = g(x) if y ≥ y*.

[0214] The two densities 1 and g are modeled using a Parzen density estimator. Here, g represents the density of the good group 62 and l represents the density of the bad group 64.

[0215] p(y) is modeled using the fact that p(y < y*) = δ, which defines the percentile split between the two classes (i.e., if g models the upper quartile, then δ = 0.75).

[0216] Using Bayes' rule (i.e., p(x,y) = p(y)p(x|y)), it can be shown that the definition of the expected improvement (EI) 66 is equivalent to l(x) / g(x).

[0217] Then the next point is selected by this sampling strategy as the maximum of l(x) / g(x), as Figure 7 shown.

[0218] Figure 8 Illustrating the pruning strategy of the asynchronous successive halving algorithm, in an example of the first or second embodiment of the present invention, the pruner 44 uses this strategy to decide whether to continue or stop the training of the model. The vertical axis shows the training loss function and the horizontal axis shows the training time. The lower the objective function value, the better the performance of the model defined by the associated one or more hyperparameter values. Only the hyperparameter value or values that achieve the lowest objective function value are continuously improved, while the remaining set is discarded in the early iterations.

[0219] Asynchronous successive halving algorithm (ASHA)

[0220] As Figure 8 shown, the ASHA algorithm is a pruning strategy and a method that combines random search with principled early stopping in an asynchronous manner.

[0221] The successive halving algorithm (SHA) is a well-known multi-armed bandit algorithm for performing principled early stopping. The successive halving algorithm starts with all candidate configurations in the base rung and proceeds as follows:

[0222] 1. Uniformly allocate the budget (the value on the horizontal axis) to a set of candidate hyperparameter configurations in a known rung;

[0223] 2. Evaluate the performance of all candidate configurations;

[0224] 3. Promote the top 1 / η of the candidate configurations to the next rung, where η is the elimination rate chosen by the user;

[0225] 4. Double the budget for each configuration in the next rung and repeat until only one configuration remains.

[0226] A higher η indicates a more aggressive elimination rate, where all but the first 1 / η configurations are eliminated.

[0227] The SHA algorithm is difficult to perform concurrently because the algorithm takes a set of configurations as input and waits for all configurations in a rung to complete before promoting a configuration to the next rung.

[0228] To eliminate the bottleneck caused by synchronous promotion, the Asynchronous Continuous Halving Algorithm (ASHA) grows from the bottom up and promotes configurations as much as possible, rather than starting with a broad set of configurations and narrowing them down.

[0229] ASHA first assigns workers to add configurations to the bottom rung. When a worker completes a job and requests a new job, the rungs from top to bottom are checked to see if there are configurations in the first 1 / η of each rung that can be promoted to the next rung. If not, workers are assigned to add configurations to the lowest rung to increase the width of the level so that more configurations can be promoted.

[0230] The output of the ASHA algorithm is as follows Fig. 9 As shown, the results of all trials are summarized. Here, 17 hyperparameter optimization trials are performed. In the initial trials, the worst trial 68 that achieved the worst objective function value is discarded in the first iteration. In each subsequent iteration, the best model in the previous iteration is kept as a seed and trained for a longer time. Finally, the best trial 70 is retrained on the original full training data set.

[0231] Fig.10 A flowchart of a computer-implemented method 10" for detecting anomalies in an imaging data set 12 of a wafer 120 according to a third or fourth embodiment of the present invention is shown, the imaging data set 12 comprising defects 23 belonging to a plurality of defect categories, the method comprising the following steps: in an anomaly detection image generation step 61, generating an anomaly detection image 72 by applying an anomaly detection method to the imaging data set 12; performing one or more iterations 73, at least one of which comprises the following steps: in a sampling step 63, providing one or more samples of anomaly detection image value distribution for each defect category of the defect category subset; in a calibration step 65, calibrating the anomaly detection image by at least one calibration method, the calibration method comprising the following steps: in a training step 67, training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the anomaly detection image value distribution; in an application step 69, applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; in a critical step 71, applying a critical value to the calibrated anomaly detection image to detect anomalies 15, thereby reducing interference and protruding defects 23 in the anomaly detection image 72.

[0232] Fig.11A computer-implemented method 10'' for anomaly detection in an imaging dataset 12 of a wafer 120 according to an example of a third or fourth embodiment of the present invention is shown. The anomaly detection image 72 is calibrated in a calibration step 74 in each iteration 88. In the calibration step 74, a calibration method is selected from a group of calibration methods.

[0233] The set of calibration methods includes a first calibration method for training a machine learning model based on one or more samples of the distribution of anomaly detection image values ​​of all defect categories in the defect category subset for anomaly localization. The trained machine learning model is then applied to the anomaly detection image 72. Finally, a critical value is applied to the calibrated anomaly detection image to obtain anomalies 15 or defects 23.

[0234] The group of calibration methods also includes a second calibration method based on global or local thresholding and / or filter operations. To this end, one or more lower thresholds and / or upper thresholds are selected for the anomaly detection image 72, and the anomaly detection image 72 is adapted based on these thresholds. The filter operation may include, for example, morphological cleaning, cluster size filtering, size filtering, etc. The size filter, for example, indicates the minimum and maximum size of the corresponding defect type, so that only anomalies or defects within a specified size range can be detected. The size of the anomaly or defect can be measured by the number of connected pixels, the length of the anomaly or defect in a specific direction, or its diameter, etc. The thresholding and filtering operations can be combined in the calibration method. A user interface can be set to allow the user to select a threshold and / or filter. Finally, the threshold is applied to the calibrated anomaly detection image to obtain the anomaly 15 or defect 23.

[0235] In an evaluation step 76, it is checked whether all (or enough) defects 23 were detected. In case of a positive answer 77, the iteration 88 is terminated in a termination step 78. Otherwise in case of a negative answer 79, in a calibration method adaptation step 80, the calibration method is adapted.

[0236] To adapt the first calibration method, annotations are added or modified by expert annotations in an annotation adjustment step 84, for example by clicking on a defect and applying expert knowledge, such as critical dimensions and / or critical distances and / or spacing dimensions related to a minimum structure size. To adjust the second calibration method, thresholds and / or filters are added or adjusted in a threshold or filter adjustment step 82, for example via user input. The calibration method may be adjusted based on additional information, for example, by retraining a machine learning model for anomaly localization in a training step 86. In each iteration 88, a different calibration method may be selected and applied to the calibrated anomaly detection images from the previous iteration.

[0237] In an example of the third or fourth embodiment of the present invention, the second calibration method includes the following steps: for each defect category of the defect category subset, an intermediate calibrated anomaly detection image is calculated by: adjusting the anomaly detection image value based on one or more selected critical values ​​of the anomaly detection image value of the current defect category; applying one or more filters of the current defect category, such as a size filter, so that the calibration method only retains anomalies 15 within a specific size range; and finally generating a calibrated anomaly detection image by applying an operator to all intermediate calibrated anomaly detection images, the operator being selected from the group including pixel-by-pixel summation, pixel-by-pixel average, pixel-by-pixel minimum, pixel-by-pixel maximum, and pixel-by-pixel scaling. In this way, the final calibrated anomaly detection image includes defects 23 from different defect categories, each of which is extracted from the anomaly detection image 72 based on a different set of critical values ​​and filters. Using the maximum operator can retain as many anomalies 15 as possible, while the minimum operator further reduces noise and interference.

[0238] Adaptation of the anomaly detection image value a according to the first calibration method may involve normalizing the value based on an upper threshold u and a lower threshold 1.

[0239]

[0240] Alternatively, adaptation of the anomaly detection image value may involve clipping the value to the range [l,u]

[0241] a new =min(max(a,l),u).

[0242] In an example according to the third or fourth embodiment of the present invention, the calibration method is expressed as follows Fig.12The pixel-by-pixel segmentation problem is shown. The input image 90 of the anomaly detection method contains two types of defects 23, namely hollow diamonds and triangles. The output of the anomaly detection method is an anomaly detection image 92, which is a difference image between the input image 90 and a comparison data set, here the output of an autoencoder that reconstructs the input image 90. The speckled background is due to noise and high reconstruction errors around the defects 23. A simple critical value that ensures a high defect recovery rate will result in a high noise rate. Therefore, the anomaly detection image 92 is calibrated using a machine learning model for anomaly localization. To this end, the user interface is configured to let the user provide expert annotations 96 on several samples of a subset of defect categories. To simplify the annotation process, the user provides click points 94, which are automatically processed into pixel-level annotations 96 by utilizing available meta-information (e.g., critical distance and / or critical size and / or pitch size). Based on this meta-information, the click points are expanded to cover the area covering the smallest structure size of the wafer. In the partially annotated anomaly detection image 95, the annotated pixels have been set to one (1) and the unannotated pixels have been set to zero (0). It is sufficient if the user only provides a small number (about 5-10, less than 20) of click points for a subset of defect classes. Even if the remaining defects 23 are not part of the training data, they can still be segmented by the machine learning model for anomaly segmentation, because it is assumed that the annotation categories cover the appearance spectrum of all defect classes. Based on the annotations, the machine learning model for anomaly segmentation is trained. In this way, the user's workload is minimized and cold starts are possible, because new defects 23 can also be detected based on limited training data and because the annotation process requires minimal user workload.

[0243] According to an example of the third or fourth embodiment of the present invention, during training, a machine learning model for anomaly segmentation takes a partially annotated anomaly detection image such as a difference image as input. As output, the model labels each pixel as an anomaly or interference of interest. Since not all input pixels are annotated by the user, the model learning is based on the following semi-supervised loss function:

[0244]

[0245] Where a is the abnormal detection image 92, y true Provide the user with pixel-level annotations from the set {unannotated = 0, anomaly = 1}, i.e., partially labeled anomaly detection images 95, y pred is the label predicted by the model. The loss function L is the sum of all pixel losses, which is a weighted cross entropy loss, defined as follows:

[0246]

[0247] The pixel-level weight w i Enable semi-supervised learning. In this article, the anomaly is marked The pixels of which are learned to be segmented as anomalies. Due to the negative index, the unannotated pixels is considered non-anomalous (at low a i value) or is ignored (in the case of high a i 15). Therefore, the segmentation model suppresses the noise and highlights the anomaly 15, as shown in the segmentation result, which corresponds to the calibrated anomaly detection image 98. Despite the limited annotations and different appearances, both defect classes are segmented due to the semi-supervised loss function. The calibrated anomaly detection image 98 is superimposed on the input image 90, i.e., the imaging dataset 12 of the wafer 120, resulting in an overlap 100.

[0248] Fig.13 A qualitative analysis of the method according to the third or fourth embodiment of the present invention is shown. In the top row, the reconstructed image 102 differs from the input image 104 in the area corresponding to the defect 23. In the bottom row, an uncalibrated anomaly detection image 106 is shown before the calibration method is applied. This uncalibrated image 106 exhibits a high recovery rate (all defects 23 are highlighted) but a low precision (multiple specular highlights corresponding to noise), i.e., there are many false positives. Furthermore, the highlight intensity of each defect class is different. In the right column, a calibrated anomaly detection image 108 is shown after the calibration method is applied. The calibrated anomaly detection image 108 shows a high recovery rate and precision, and all defect classes show a uniform intensity of 1.0. These can be easily and automatically detected based on a single uniform critical value of 0.5. Therefore, the application of the calibration method maintains a high recovery rate, significantly improves the precision and ensures a uniform intensity value of the defect 23, so applying a single critical value to the calibrated anomaly detection image 108 is sufficient for anomaly detection.

[0249] The advantages of applying the calibration method according to the third or fourth embodiment of the present invention are: Fig.14 , which shows the confusion matrix 110 before calibration and the confusion matrix 112 after calibration, i.e. before and after segmenting the anomaly detection image 108 and applying a threshold of 0.5. In this way, the recall is kept high (0.93 compared to 1.0) and the false positives are significantly reduced (from 0.61 to 0.25), thereby significantly improving the accuracy.

[0250] Fig.15A system 114 for controlling the production quality of wafers 120 in a semiconductor manufacturing plant is schematically illustrated. The system 114 includes an imaging device 116 and a processing device 118. The imaging device 116 is coupled to the processing device 118. The imaging device 116 is configured to acquire an imaging data set 12 of the wafer 120. The wafer 120 may include semiconductor structures, for example, transistors such as field effect transistors, memory cells, etc. An example implementation of the imaging device 116 may be a SEM or a multi-beam mSEM, a helium ion microscope (HIM), or a cross-beam device containing a FIB and a SEM, or any charged particle imaging device.

[0251] The imaging device 116 may provide the imaging data set 12 to the processing device 118. The processing device 118 includes a processor, for example, implemented as a CPU 122 or a GPU. The processor may receive the imaging data set 12 via an interface 124. The processor may load program code from a memory 126. The processor may execute the program code. When executing the program code, the processor performs techniques such as those described herein, such as hyperparameter optimization, training anomaly detection methods, executing anomaly detection methods to detect one or more anomalies 15 in the imaging data set 12 of the wafer 120, calibrating anomaly detection images based on sample distributions of anomaly detection image values, etc. For example, the processor may execute, when loading the program code from the memory 126, Figure 3 , Figure 6 or Fig.10 The computer-implemented method shown. The processing device may optionally include a user interface 128 for entering user input, such as click points, bounding boxes, or distribution features of anomaly detection image values.

[0252] Fig.16 A system 114' for controlling the production of wafers 120 in a semiconductor manufacturing plant is schematically illustrated. The system 114' includes Fig.14 1 , and the above also applies to the corresponding components in this document. In addition, the system 114' has a device 130 for producing a wafer 120 controlled by at least one wafer manufacturing process parameter. To this end, the imaging data set 12 is provided to the processing device 118 by the imaging device 116. The processor of the processing device 118 is configured to perform one of the disclosed methods, including controlling at least one wafer manufacturing process parameter based on one or more measured characteristics of the detected anomalies 15 in the imaging data set 12 of the wafer 120. For example, since the detected anomaly 15 of the bridge defect indicates insufficient etching, the etching amount is increased, since the detected anomaly 15 of the broken line indicates excessive etching, the etching amount is reduced, the continuous occurrence of the anomaly 15 indicates that the mask is defective, so the mask must be checked, and since the anomaly 15 of the missing structure indicates undesirable material deposition, the material deposition is modified.

[0253] The following clauses contain preferred embodiments of the present invention:

[0254] 1a. A computer-implemented method for anomaly detection, comprising:

[0255] - Selecting an imaging dataset of a wafer;

[0256] - generating training data from the imaging dataset;

[0257] -Iterate the following steps:

[0258] i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection;

[0259] ii. training the machine learning model defined by the hyperparameters based on a subset of the generated training data;

[0260] iii. Evaluating the trained machine learning model by calculating the associated objective function value of an objective function;

[0261] -Selecting one of the trained machine learning models based on the associated objective function value and applying it to the wafer imaging dataset to detect anomalies.

[0262] 1b. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset comprising defects belonging to a plurality of defect classes, the method comprising:

[0263] - generating training data from the imaging dataset;

[0264] -Iterate the following steps:

[0265] i. Based on a sampling strategy, select some hyperparameter values ​​from a set of associated hyperparameter values, each hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection;

[0266] ii. training the machine learning model for detecting anomalies based on the subset of the generated training data;

[0267] iii. Evaluate the trained machine learning model using an objective function that contains at least two different model evaluation metrics;

[0268] - selecting one of the trained machine learning models based on the associated objective function value and applying it to the wafer imaging dataset to detect anomalies, thereby providing an anomaly detection image;

[0269] - performing one or more iterations, at least one of which comprises the following steps:

[0270] i. providing one or more samples of anomaly detection image value distribution for each defect category of a subset of defect categories;

[0271] ii. calibrating the anomaly detection image according to the sample of the distribution using a calibration method selected from a group of calibration methods, thereby reducing interference and prominent defects in the anomaly detection image, wherein at least one of the selected calibration methods comprises the following steps:

[0272] c. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of anomaly detection image values ​​for all defect categories of a subset of defect categories;

[0273] d. Applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image;

[0274] - Applying a single threshold to the calibrated anomaly detection image to detect anomalies.

[0275] 1c. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset comprising defects belonging to a plurality of defect classes, the method comprising:

[0276] - providing an anomaly detection image by applying an anomaly detection model to the imaging dataset;

[0277] - Perform one or more iterations, at least one of which includes the following steps:

[0278] i. providing one or more samples of anomaly detection image value distribution for each defect category of a subset of defect categories;

[0279] ii. calibrating the anomaly detection image according to the sample of the distribution using a calibration method selected from a group of calibration methods, thereby reducing interference and prominent defects in the anomaly detection image, wherein at least one of the selected calibration methods comprises the following steps:

[0280] a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of anomaly detection image values ​​for all defect categories of the defect category subset;

[0281] b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image;

[0282] - Applying a single threshold to the calibrated anomaly detection image to detect anomalies.

[0283] 1. A computer-implemented method for detecting anomalies in an imaging data set of a wafer, comprising:

[0284] - generating training data from the imaging dataset;

[0285] -Iterate the following steps:

[0286] i. Based on a sampling strategy, select some hyperparameter values ​​from a set of associated hyperparameter values, each hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection;

[0287] ii. training the machine learning model for detecting anomalies based on a subset of the generated training data;

[0288] iii. Evaluate the trained machine learning model using an objective function that contains at least two different model evaluation metrics;

[0289] -Selecting one of the trained machine learning models based on the associated objective function value and applying it to the imaging dataset of the wafer to detect anomalies.

[0290] 2. A method as described in any of the preceding clauses, wherein the machine learning model is trained when presented with a subset of the imaging dataset as input to compute a reconstruction of the subset without anomalies, and to detect anomalies within the subset based on a comparison between the subset and the reconstructed subset.

[0291] 3. A method as described in clause 2, wherein the machine learning model comprises an autoencoder.

[0292] 4. In a method as described in any of the preceding clauses, at least one hyperparameter is related to the design of the machine learning model.

[0293] 5. A method as described in any of the preceding clauses, wherein one of the model evaluation metrics comprises a loss function used during training of the machine learning model.

[0294] 6. A method as described in any of the preceding clauses, wherein one of the model evaluation metrics comprises an Lp-norm loss function, where p≥1.

[0295] 7. A method as described in any of the preceding clauses, wherein the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and one of the model evaluation metrics comprises a discriminant loss function for evaluating the difference between expert annotated anomalies and detected anomalies.

[0296] 8. A method as described in clause 7, wherein the expert annotations account for less than 10% of the training data, preferably less than 1%.

[0297] 9. A method as described in clause 7 or 8, wherein the expert annotation contains only a subset of anomalies present in the imaging dataset.

[0298] 10. A method as described in any of the preceding clauses, wherein one of the model evaluation metrics comprises a complexity measure of the machine learning model.

[0299] 11. A method as described in clause 10, wherein the measure of the complexity of the machine learning model includes the total number of floating-point operations in a forward pass of the machine learning model.

[0300] 12. A method as described in any of the preceding clauses, wherein the objective function comprises a weighted sum of the at least two model evaluation metrics.

[0301] 13. A method as described in clause 12, wherein the training data comprises expert annotations of anomalies on a subset of the imaging dataset, and the objective function comprises an Lp-norm loss function, a discriminant loss function for evaluating the difference between expert annotated anomalies and detected anomalies, and a weighted sum of a complexity measure of the machine learning model, where p ≥ 1.

[0302] 14. A method as described in any of the preceding clauses, wherein the sampling strategy for selecting the number of hyperparameter values ​​includes considering the hyperparameter value and the corresponding value of the objective function from one or more previous iterations.

[0303] 15. The method of clause 14, wherein the sampling strategy selects hyperparameter values ​​by optimizing a criterion selected from a group consisting of expected improvement, maximum probability of improvement, and an upper confidence limit.

[0304] 16. The method of any of the preceding clauses, wherein the sampling strategy comprises an early stopping criterion.

[0305] 17. A method as described in any of the preceding clauses, wherein the sampling strategy comprises randomly selecting hyperparameter values ​​from the associated set of hyperparameter values.

[0306] 18. A method as described in any of the preceding clauses, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value being selected by the sampling strategy.

[0307] 18a. A method as described in the preceding clause, wherein the probability distribution is modeled based on prior application knowledge, preferably based on imaging hardware settings or design knowledge, such as critical distance.

[0308] 19. A method as described in any of the preceding clauses, wherein the sampling strategies of at least two iterations are different.

[0309] 20. A method as described in any of the preceding clauses, wherein the size of the subset of generated training data varies depending on the sampling strategy.

[0310] 21. A method as described in clause 20, wherein the size of the subset of generated training data increases with the number of iterations.

[0311] 22. A method as described in any of the preceding clauses, wherein prior knowledge based on a specific application is used to select initial hyperparameter values.

[0312] 23. The method of any of the preceding clauses, wherein the sampling strategy comprises:

[0313] - in a first iteration: selecting an initial hyperparameter value from an associated set of hyperparameter values;

[0315] - In the next iteration: a hyperparameter value is selected from the associated set of hyperparameter values ​​based on the hyperparameter value and the objective function from one or more previous iterations

[0316] should value, the standard that maximizes the expected improvement;

[0317] - If the early stopping criteria is met in an iteration: select initial hyperparameter values ​​from the associated set of hyperparameter values.

[0318] 24. A method as described in any of the preceding clauses, wherein iterations are performed in parallel by multiple threads.

[0319] 25. A method as described in any of the preceding clauses, wherein the selected machine learning model is trained on the entire imaging dataset of the chip and then applied to the imaging dataset of the chip to detect anomalies.

[0320] 26. A method as described in any of the preceding clauses, wherein the trained machine learning model that achieves the lowest objective function value is selected.

[0321] 27. A method as described in any of the preceding clauses, wherein the trained machine learning model that achieves the highest objective function value is selected.

[0322] 28. The method of any of the preceding clauses, wherein the sampling strategy criteria implements at least one member selected from the group consisting of an exploratory approach and an exploitative approach.

[0323] 28a. A method as described in any of the preceding clauses, wherein the objective function includes a quality value to evaluate the quality of the trained machine learning model, and wherein a user interface is configured to present information about the trained machine learning model to a user and allow the user to indicate the quality value.

[0324] 29. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset comprising defects belonging to a plurality of defect classes, the method comprising:

[0325] - providing an anomaly detection image generated by applying the anomaly detection model to the imaging data set;

[0326] -Perform one or more iterations of the following steps:

[0327] i. providing one or more features of anomaly detection image value distribution for each defect category of a subset of defect categories;

[0328] ii. calibrating the anomaly detection image based on the characteristics of the distribution using a calibration method selected from a group of calibration methods, thereby reducing interference and prominent defects in the anomaly detection image.

[0329] 30. The method of clause 29, 1a, 1b or 1c, wherein the anomaly detection image is a difference image between the imaging dataset and a comparison dataset.

[0330] 31. The method of clause 30, wherein the comparison data is based on a die-to-die principle or a die-to-database principle.

[0331] 32. A method as described in clause 30 or 31, wherein the comparison data set is generated by a machine learning model.

[0332] 33. The method of clause 32, wherein the comparing dataset comprises generating a reconstructed representation of the imaging dataset by training an autoencoder on the imaging dataset or a subset thereof, and applying the trained autoencoder to the imaging dataset to obtain a reconstructed representation.

[0333] 34. A method as described in clause 29, 1a, 1b or 1c, wherein the anomaly detection image is generated by a machine learning model.

[0334] 35. A method as described in clause 34, wherein the anomaly detection and the calibration of the anomaly detection image are jointly learned by a machine learning model, which is applied to the imaging dataset to directly obtain a corrected anomaly detection image.

[0335] 36. The method of any of clauses 29 to 35, 1a, 1b or 1c, wherein the one or more features comprise a sample from the distribution.

[0336] 37. A method as described in any of clauses 29 to 36, 1a, 1b or 1c, wherein the one or more features include an upper quantile and / or a lower quantile of the distribution.

[0337] 38. A method as described in any of clauses 29 to 37, 1a, 1b or 1c, wherein the one or more characteristics include moments of the distribution.

[0338] 39. A method as described in any of clauses 29 to 36, 1a, 1b or 1c, wherein the one or more characteristics include a minimum and / or maximum value of the domain of the distribution.

[0339] 40. The method of any one of clauses 29 to 39, further providing a filter for each defect category of the subset of defect categories.

[0340] 41. The method of clause 40, wherein the filter comprises a size filter.

[0341] 42. The method of clause 40 or 41, wherein the set of calibration methods comprises a calibration method for calibrating the anomaly detection image by applying the following steps:

[0342] For each defect category of the defect category subset, an intermediate calibration image is calculated by the following steps:

[0343] i. adapting the anomaly detection image value based on one or more features of the anomaly detection image value distribution of the current defect category;

[0344] ii. Apply one or more filters for the current defect category;

[0345] - generating a calibrated anomaly detection image by applying an operator to all intermediate calibrated images, the operator being selected from the group consisting of pixel-wise sum, pixel-wise average, pixel-wise minimum, pixel-wise maximum, pixel-wise scaling.

[0346] 43. The method of clause 42, wherein the anomaly detection image values ​​are adapted by normalization.

[0347] 44. The method of clause 42, wherein the anomaly detection image values ​​are adapted by cropping.

[0348] 45. A method as described in any one of clauses 29 to 44, 1a, 1b or 1c, wherein the set of calibration methods includes a calibration method for calibrating the anomaly detection image by applying the following steps:

[0349] - training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more features of the distribution of anomaly detection image values ​​for all defect categories of the subset of defect categories;

[0350] -Applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image.

[0351] 46. ​​A method as described in clause 45, wherein the machine learning model is trained to optimize a loss function for anomaly localization based on foreground and background samples.

[0352] 47. A method as described in clause 46, wherein the loss function is a weighted cross entropy loss function.

[0353] 48. The method of clause 46, wherein the loss function is a Kullback-Leibler divergence loss function or an L1 loss function or an L2 loss function.

[0354] 49. A method as described in any of clauses 46 to 48, wherein the loss function is a semi-supervised loss function.

[0355] 50. A method as described in any of clauses 45 to 49, wherein the machine learning model for anomaly localization is trained on partially labeled anomaly detection images derived from features of the distribution.

[0356] 51. A method as described in any of clauses 45 to 50, wherein the one or more features include samples from the distribution, the samples serving as foreground or background samples.

[0357] 52. The method of clause 51, wherein the sample is used as a foreground sample and background samples are selected from the remaining pixels of the anomaly detection image.

[0358] 53. The method of clause 52, wherein if the value of the anomaly detection image is below a critical value, the remaining pixels of the anomaly detection image may be selected as background samples.

[0359] 54. A method as described in clause 52 or 53, wherein each background sample is weighted by a weighting function w of its anomaly detection image value a, in particular by a negative exponential weighting function w(a)=exp(-a).

[0360] 55. A method as described in any of clauses 45 to 54, wherein the machine learning model is pre-trained on training data from similar applications.

[0361] 56. A method as described in any of clauses 45 to 55, wherein the machine learning model addresses class imbalance by using a focal loss function and / or prior defect rate.

[0362] 57. A method as described in any one of clauses 29 to 56, 1a, 1b or 1c, further comprising presenting the anomaly detection image to a user via a user interface, the user interface being configured to allow the user to input information about each defect category in the defect category subset, from which one or more features of the anomaly detection image value distribution for each defect category are derived.

[0363] 58. A method as described in clause 57, wherein the user interface is configured to allow a user to indicate the locations of a small number of defects for each category of the subset of defect categories in the anomaly detection image, and one or more features of the distribution of anomaly detection image values ​​for each of these categories, provided by sampling anomaly detection image values ​​from the defects indicated by the user for that category.

[0364] 59. The method of clause 58, wherein the user indicates the location of the defect by selecting a single pixel, preferably a pixel within a central area of ​​the defect.

[0365] 60. A method as described in clause 59, wherein the annotation of the defect is automatically generated by selecting a region (preferably a circle) surrounding a selected single pixel.

[0366] 61. The method of clause 60, wherein the anomaly detection image values ​​are sampled from an area surrounding a selected pixel.

[0367] 62. A method as described in clause 60 or 61, wherein the region is selected based on application specific knowledge, in particular knowledge from the field of wafer manufacturing.

[0368] 63. The method of clause 62, wherein the size of the region is selected based on a critical dimension.

[0369] 64. A method as described in clause 62 or 63, wherein the size of the area is selected based on the spacing size.

[0370] 65. A method as described in any of clauses 29 to 64, 1a, 1b or 1c, wherein the calibrated anomaly detection image is post-processed to further reduce interference.

[0371] 66. A method as described in any of clauses 29 to 65, 1a, 1b or 1c, wherein one or more features of the distribution of anomaly detection image values ​​for each defect category of the subset of defect categories are provided based on similar applications.

[0372] 67. The method of any one of clauses 29 to 66, 1a, 1b or 1c, further comprising generating an uncertainty estimate for each anomaly in the calibrated anomaly detection image.

[0373] 68. The method of any one of clauses 29 to 67, 1a, 1b or 1c, further comprising a final step after performing said one or more iterations:

[0374] -Thresholding the calibrated difference image by a single threshold to obtain a binary anomaly detection image.

[0375] 69. A method as described in any of clauses 29 to 68, 1a, 1b or 1c, wherein the defect category subset contains less than 50% of the defect category, preferably less than 30% of the defect category, more preferably less than 20% of the defect category, and most preferably less than 10% of the defect category.

[0376] 70. The method of any of clauses 29 to 69, 1a, 1b or 1c, wherein a plurality of iterations are performed and the calibration method is different for at least two of the plurality of iterations.

[0377] 70a. A method as described in any of the preceding clauses, wherein the imaging dataset comprises one or more multi-sensory images.

[0378] 70b. ​​A method as described in any of the preceding clauses, wherein the imaging data set comprises a two-dimensional image, a three-dimensional image, a sliced ​​three-dimensional image, or a multi-sensory fusion image.

[0379] 71. The method of any of the preceding clauses, wherein the imaging dataset is a multi-beam SEM image.

[0380] 72. A method as described in any of the preceding clauses, wherein the imaging data set is a focused ion beam SEM image.

[0381] 73. A method as described in any of the preceding clauses, further comprising measuring one or more characteristics of the detected anomaly.

[0382] 74. The method of clause 73, further comprising controlling at least one wafer fabrication process parameter based on the one or more measured characteristics.

[0383] 75. The method of clause 73, further comprising assessing the quality of the wafer based on the one or more measured characteristics and at least one quality assessment rule.

[0384] 76. One or more machine-readable hardware storage devices comprising instructions executable by one or more processing devices to perform operations comprising the method of any one of clauses 1 to 75.

[0385] 77. A system for controlling the quality of wafers produced in a semiconductor manufacturing plant, the system comprising:

[0386] - an imaging device adapted to provide an imaging data set of the wafer;

[0387] - one or more processing devices;

[0388] - One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations comprising the method of clause 75.

[0389] 78. A system for controlling wafer production in a semiconductor manufacturing plant, the system comprising:

[0390] - a production wafer assembly for producing wafers controlled by at least one manufacturing process parameter;

[0391] - an imaging device adapted to provide an imaging data set of the wafer;

[0392] - one or more processing devices;

[0393] - One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations comprising the method of clause 74.

[0394] 79. A computer-implemented method for anomaly detection, comprising:

[0395] -selecting an imaging data set of a wafer;

[0396] - generating training data from the imaging dataset;

[0397] -Iterate the following steps:

[0398] i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection;

[0399] ii. training the machine learning model defined by the hyperparameters based on a subset of the generated training data;

[0400] i. Evaluate the trained machine learning model by calculating the associated objective function value of an objective function;

[0401] -Selecting one of the trained machine learning models based on the associated objective function value and applying it to the wafer imaging dataset to detect anomalies.

[0402] 80. A method as described in clause 79, wherein the machine learning model is trained when presented with a subset of the imaging dataset as input to compute a reconstruction of the subset without anomalies, and to detect anomalies within the subset based on a comparison between the subset and the reconstructed subset.

[0403] 81. A method as described in clause 80, wherein the machine learning model comprises an autoencoder.

[0404] 82. A method as described in any of clauses 79 to 81, wherein at least one hyperparameter is related to the design of the machine learning model.

[0405] 83. A method as described in any of clauses 79 to 82, wherein the objective function comprises at least two different model evaluation metrics.

[0406] 84. A method as described in clause 83, wherein one of the model evaluation metrics includes a loss function used during training of the machine learning model.

[0407] 85. A method as described in clause 83 or 84, wherein one of the model evaluation metrics comprises an Lp-norm loss function, where p≥1.

[0408] 86. A method as described in any of clauses 83 to 85, wherein the training data comprises expert annotations of anomalies on a subset of the imaging dataset, and one of the model evaluation metrics comprises a discriminative loss function for evaluating the difference between the expert annotations of the anomalies and the detected anomalies.

[0409] 87. A method as described in clause 86, wherein the expert annotation contains only a subset of anomalies present in the imaging dataset.

[0410] 88. A method as described in any of clauses 83 to 87, wherein one of the model evaluation metrics comprises a complexity measure of the machine learning model.

[0411] 89. A method as described in clause 88, wherein the measure of the complexity of the machine learning model takes into account the total number of floating point operations in a forward pass of the machine learning model.

[0412] 90. A method as described in any of clauses 79 to 89, wherein the training data comprises expert annotations of anomalies on a subset of the imaging dataset, and the objective function comprises an Lp-norm loss function, a discriminant loss function for evaluating the difference between the expert annotated anomalies and the detected anomalies, and a weighted sum of a complexity measure of the machine learning model, where p ≥ 1.

[0413] 91. A method as described in any of clauses 79 to 90, wherein the objective function includes a quality value to evaluate the quality of the trained machine learning model, and wherein a user interface is configured to present information about the trained machine learning model to a user and allow the user to indicate the quality value.

[0414] 92. A method as described in any of clauses 79 to 91, wherein the sampling strategy for selecting hyperparameter values ​​comprises considering hyperparameter values ​​and corresponding values ​​of the objective function from one or more previous iterations by optimizing a criterion selected from a group containing expected improvement, maximum probability of improvement, and an upper confidence limit.

[0415] 93. The method of any one of clauses 79 to 92, wherein the sampling strategy comprises an early stopping criterion.

[0416] 94. A method as described in any of clauses 79 to 93, wherein at least one set of hyperparameter values ​​is associated with a probability distribution that indicates the likelihood of each hyperparameter value being selected by the sampling strategy.

[0417] 95. A method as described in clause 94, wherein the probability distribution is modeled based on previous application knowledge, preferably based on imaging hardware settings or design knowledge, such as critical distance, critical size or pitch size.

[0418] 96. A method as described in any of clauses 79 to 95, wherein the sampling strategy of at least two iterations is different.

[0419] 97. A method as described in any of clauses 79 to 96, wherein the size of the subset of generated training data increases with the number of iterations.

[0420] 98. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset comprising defects belonging to a plurality of defect classes, the method comprising:

[0421] - Producing anomaly detection images by applying anomaly detection methods to imaging datasets;

[0422] - Perform one or more iterations, at least one of which includes the following steps:

[0423] i. providing one or more samples of anomaly detection image value distribution for each defect category of a subset of defect categories;

[0424] ii. calibrating the anomaly detection image by at least one calibration method, the method comprising the following steps:

[0425] a. Based on one or more samples of the distribution of anomaly detection image values, training a machine learning model for anomaly localization, preferably anomaly segmentation;

[0426] b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image;

[0427] - Applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing noise and prominent defects in the anomaly detection image.

[0428] 99. A method as described in clause 98, wherein the anomaly detection image is a difference image between the imaging data set and a comparison data set, and the comparison data is based on a die-to-die principle or a die-to-database principle.

[0429] 100. A method as described in clause 99, wherein the comparison dataset comprises generating a reconstructed representation of the imaging dataset by training a machine learning autoencoder on the imaging dataset or a subset thereof, and applying the trained autoencoder to the imaging dataset to obtain a reconstructed representation.

[0430] 101. A method as described in any of clauses 98 to 100, wherein the machine learning model for anomaly localization is trained to optimize a loss function based on foreground and background samples.

[0431] 102. A method as described in clause 101, wherein the loss function is a semi-supervised loss function.

[0432] 103. A method as described in clause 101 or 102, wherein samples from the anomaly detection image value distribution for each defect class in the subset of defect classes are used as foreground or background samples.

[0433] 104. The method of clause 103, wherein the sample is used as a foreground sample and background samples are selected from the remaining pixels of the anomaly detection image.

[0434] 105. The method of clause 104, wherein if the value of the anomaly detection image is below a critical value, the remaining pixels of the anomaly detection image may be selected as background samples.

[0435] 106. A method as described in clause 104 or 105, wherein each background sample is weighted by a weighting function w of its anomaly detection image value a, in particular by a negative exponential weighting function w(a)=exp(-a).

[0436] 107. A method as described in any of clauses 98 to 106, wherein the user interface is configured to allow a user to indicate the location of a small number of defects for each category of a subset of defect categories in the anomaly detection image, wherein the user indicates the location of the defect by selecting a single pixel of the defect, preferably a pixel in a central area, and automatically generates an annotation of the defect by selecting an area, preferably a circle, surrounding the selected single pixel, and samples the anomaly detection image values ​​from the area around the selected pixel.

[0437] 108. The method of clause 107, wherein the size of the region is selected based on a critical dimension and / or a critical distance and / or a pitch size.

[0438] 109. A method as described in any of clauses 79 to 108, wherein the imaging dataset comprises one or more multi-sensory images.

[0439] 110. The method of any one of clauses 79 to 109, wherein the imaging dataset is a multi-beam SEM image.

[0440] 111. The method of any one of clauses 79 to 110, wherein the imaging dataset is a focused ion beam SEM image.

[0441] 112. The method of any of clauses 79 to 111, further comprising measuring one or more characteristics of the detected anomaly.

[0442] 113. The method of clause 112, further comprising controlling at least one wafer fabrication process parameter based on the one or more measured characteristics.

[0443] 114. The method of clause 112, further comprising assessing a quality of the wafer based on the one or more measured characteristics and at least one quality assessment rule.

[0444] 115. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations comprising the method of any of clauses 79 to 114.

[0445] 116. A system for controlling the quality of wafers produced in a semiconductor manufacturing plant, the system comprising:

[0446] - an imaging device adapted to provide an imaging data set of the wafer;

[0447] - one or more processing devices;

[0448] - One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the method as recited in clause 114.

[0449] 117. A system for controlling the production of wafers in a semiconductor manufacturing plant, the system comprising:

[0450] - a production wafer assembly for producing wafers controlled by at least one manufacturing process parameter;

[0451] - an imaging device adapted to provide an imaging data set of the wafer;

[0452] - one or more processing devices;

[0453] - One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the method as described in clause 113.

[0454] In summary, particular attention should be paid to the following preferred features of the present invention: The present invention relates to a computer-implemented method 10, 10' for detecting anomalies 15, which includes: selecting an imaging data set 12 of a chip 120 and a hyperparameter value defining a machine learning model 16 for anomaly detection; training and evaluating the machine learning model 16 by calculating an objective function value; selecting one of the trained machine learning models and applying it to detect anomalies 15. The present invention also relates to a computer-implemented method 10" for detecting anomalies 15 in an imaging data set 12 of a chip 120, which includes: providing a sample of anomaly detection image value distribution for each defect category; calibrating the anomaly detection image 20, 26, 26' for anomaly localization by training a machine learning model; applying a critical value to the calibrated anomaly detection image 98, 108 to detect anomalies 15.

[0455] Reference numerals list

[0456] 10. 10', 10", 10'' computer implemented method

[0457] 11 Defective unit structure

[0458] 12 Imaging Datasets

[0459] 14 Input

[0460] 15 Abnormal

[0461] 16 Machine Learning Models

[0462] 18 Reconstruction

[0463] 19 Data Selection Steps

[0464] 20 Anomaly Detection Images

[0465] 21 Training data generation steps

[0466] 22.22' Input image

[0467] 23 Defects

[0468] 24.24'Reconstruction of the image

[0469] 25 Hyperparameter Value Selection Steps

[0470] 26.26' Anomaly Detection Image

[0471] 27 Training Steps

[0472] 28. 28' Anomaly Detection Image

[0473] 29 Model Evaluation Steps

[0474] 30.30' Input image

[0475] 31 Model Selection Steps

[0476] 32.32'Reconstruction of image

[0477] 34 No defect data

[0478] 36 Expert comments

[0479] 38 Expert knowledge

[0480] 40 Hyperparameter Range

[0481] 42 samples

[0482] 43 Hyperparameter Optimization Unit

[0483] 44 Clipper

[0484] 46 Objective Function

[0485] 48 Hyperparameters

[0486] 50 subsampled data set

[0487] 52 Full imaging datasets

[0488] 54 Optimization Model

[0489] 56 Final Model

[0490] 58 Good Group

[0491] 60 Bad group

[0492] 61 Abnormal Detection Image Generation Steps

[0493] 62 Density of good groups

[0494] 63 Sampling steps

[0495] 64 Density of bad group

[0496] 65 Calibration steps

[0497] 66 Expected Improvement

[0498] 67 Training Steps

[0499] 68 Worst Test

[0500] 69 Application Steps

[0501] 70 Best Experiment

[0502] 71 Criticalization Steps

[0503] 72 Anomaly Detection Images

[0504] 73 Iterations

[0505] 74 Calibration steps

[0506] 76 Evaluation Steps

[0507] 77 Yes

[0508] 78 Termination Step

[0509] 79 No answer

[0510] 80 Calibration Method Adaptation Steps

[0511] 82 Critical or Screening Adjustment Steps

[0512] 84 Annotation Adjustment Steps

[0513] 86 Training Steps

[0514] 88 Iterations

[0515] 90 Input image

[0516] 92 Anomaly Detection Images

[0517] 94 Click Points

[0518] 95 partially annotated anomaly detection images

[0519] 96 Comments

[0520] 98 calibrated anomaly detection images

[0521] 100 Overlap

[0522] 102 Reconstructing the Image

[0523] 104 Input Image

[0524] 106 Uncalibrated Anomaly Detection Images

[0525] 108 Calibrated Anomaly Detection Images

[0526] 110 Confusion matrix before calibration

[0527] 112 Confusion matrix after calibration

[0528] 114, 114' system

[0529] 116 Imaging Device

[0530] 118 Processing Device

[0531] 120 chips

[0532] 122CPU

[0533] 124 Interface

[0534] 126 Memory

[0535] 128 User Interface

[0536] 130 Components

Claims

1. A computer-implemented method (10, 10') for detecting an anomaly (15), comprising: - selecting an imaging data set (12) of a wafer (120); - generating training data from the imaging data set (12); -Iterate the following steps: i. selecting a hyperparameter value from an associated set of hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter (48) defining a machine learning model (16) for detecting anomalies (15); ii. training the machine learning model (16) defined by the hyperparameters (48) based on the subset of the training data that has been generated; iii. evaluating the trained machine learning model (16) by calculating the associated objective function value of the objective function (46); - selecting one of the trained machine learning models based on the associated objective function value and applying it to the imaging dataset (12) of the wafer (120) to detect anomalies (15).

2. The method of claim 1, wherein: The machine learning model is trained when presented with a subset of the imaging dataset (12) as input (14) to compute a reconstruction (18) of the subset without abnormalities (15) and to detect abnormalities (15) within the subset based on a comparison between the subset and the reconstructed subset.

3. The method of claim 2, wherein: The machine learning model (16) includes an autoencoder.

4. A method as claimed in any one of the preceding claims, wherein: At least one hyperparameter (48) is a design hyperparameter that is related to the design of the machine learning model (16).

5. A method as claimed in any one of the preceding claims, wherein: All of the at least one hyperparameter (48) are design hyperparameters that are related to the design of the machine learning model.

6. A method as claimed in any one of the preceding claims, wherein: The objective function (46) includes at least two different model evaluation metrics.

7. The method of claim 6, wherein: At least one of the model evaluation metrics measures the quality of the anomaly detection.

8. The method according to claim 6 or 7, wherein: One of the model evaluation metrics includes a loss function used during training of the machine learning model (16).

9. The method of claim 8, wherein: One of the model evaluation metrics is a loss function used during training of the machine learning model (16).

10. The method according to any one of claims 6 to 9, wherein: One of the model evaluation metrics comprises an Lp-norm loss function of the training data sample and the corresponding target data sample, where p≥1.

11. The method according to any one of claims 7 to 10, wherein: The training data comprises expert annotations (36) of anomalies (15) for a subset of the imaging dataset (12), and one of the model evaluation metrics comprises a discriminative loss function that evaluates the difference between the expert annotations (36) of the anomalies (15) and the anomalies (15) that have been detected.

12. The method of claim 11, wherein: The expert annotation (36) assigns labels to only a subset of the abnormality types (15) present in the imaging dataset (12).

13. The method according to any one of claims 7 to 12, wherein: One of the model evaluation metrics includes a complexity measure of the machine learning model (16).

14. A method as claimed in any one of the preceding claims, wherein: The training data comprises expert annotations (36) of anomalies (15) of a subset of the imaging dataset (12), and the objective function (46) comprises a weighted sum of an Lp-norm loss function of the training data samples and corresponding target data samples and a discriminant loss function evaluating the difference between the expert annotations (36) of the anomalies (15), where p≥1.

15. The method of claim 14, wherein: The objective function (46) also includes a complexity measure of the machine learning model (16).

16. The method according to claim 13 or 15, wherein: The complexity measure of the machine learning model (16) takes into account the total number of floating point operations in a forward pass of the machine learning model (16).

17. A method as claimed in any one of the preceding claims, wherein: The objective function (46) includes a quality value to evaluate the quality of the trained machine learning model (16), wherein the user interface (128) is configured to present information about the trained machine learning model (16) to a user and allow the user to indicate the quality value.

18. A method as claimed in any one of the preceding claims, wherein: The sampling strategy for selecting the number of hyperparameter values ​​comprises considering hyperparameter values ​​and corresponding values ​​of the objective function (46) from one or more previous iterations by optimizing a criterion selected from a group consisting of expected improvement, maximum probability of improvement, and an upper confidence limit.

19. A method as claimed in any one of the preceding claims, wherein: The sampling strategy included early stopping criteria.

20. A method as claimed in any one of the preceding claims, wherein: At least one set of hyperparameter values ​​is associated with a probability distribution that indicates the likelihood of each hyperparameter value being selected by the sampling strategy.

21. The method of claim 20, wherein: The probability distribution is modeled based on prior application knowledge about the critical distance, critical dimension, or pitch size.

22. A method as claimed in any one of the preceding claims, wherein: For at least two iterations, the sampling strategy was different.

23. A method as claimed in any one of the preceding claims, wherein: The size of the subset of the training data that has been generated increases with the number of iterations.

24. A computer-implemented method (10", 10"') for detecting anomalies (15) in an imaging data set (12) of a wafer (120), the imaging data set (12) comprising defects (23) belonging to a plurality of defect classes, the method comprising: - generating an anomaly detection image (20, 26, 26', 28, 28', 72, 92) by applying an anomaly detection method to the imaging data set (12); - performing one or more iterations (88) comprising the following steps: i. providing one or more samples of a distribution of abnormal detection image values ​​for each defect category of the subset of defect categories; ii. calibrating the anomaly detection image (20, 26, 26', 28, 28', 72, 92), the at least one calibration method comprising the following steps: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of the anomaly detection image values; b. applying the trained machine learning model to the anomaly detection image (20, 26, 26', 28, 28', 72, 92) to obtain a calibrated anomaly detection image (98, 108); - applying a threshold to the calibrated anomaly detection image (98, 108) to detect anomalies (15), thereby reducing noise and protruding defects (23) in the anomaly detection image (20, 26, 26', 28, 28', 72, 92).

25. The method of claim 24, wherein: The anomaly detection image (20, 26, 26', 28, 28', 72, 92) is a difference image between the imaging data set (12) and a comparison data set, the comparison data set being based on a die-to-die principle or a die-to-database principle.

26. The method of claim 25, wherein: The comparison dataset comprises a reconstructed representation of the imaging dataset (12), which is generated by training a machine learning autoencoder on the imaging dataset (12) or a subset thereof and applying the trained autoencoder to the imaging dataset (12) to obtain the reconstructed representation.

27. The method of any one of claims 24 to 26, wherein: The machine learning model for anomaly localization is trained to optimize the loss function based on anomaly and non-anomalous samples.

28. The method of claim 27, wherein: This loss function is a semi-supervised loss function.

29. The method of claim 27 or 28, wherein: The samples from the distribution of abnormality detection image values ​​for each defect class in the subset of defect classes are used as abnormal samples.

30. The method of claim 29, wherein: The non-abnormal samples are selected from the remaining pixels of the abnormality detection image (20, 26, 26', 28, 28', 72, 92).

31. The method of claim 30, wherein: The remaining pixels of the anomaly detection image (20, 26, 26', 28, 28', 72, 92) are selected as non-abnormal samples if their anomaly detection image values ​​are lower than a critical value.

32. The method of claim 30 or 31, wherein: In this loss function, each abnormal sample is detected by its abnormal image value a i weighted by the weighting function w, in particular by the negative exponential weighting function w(a i )=exp(-a i ) to weight.

33. The method of any one of claims 27 to 32, wherein: The loss function is a semi-supervised weighted cross-entropy loss function.

34. The method of claim 33, wherein: The loss function is defined as follows: Where a represents the abnormal detection image (20, 26, 26', 28, 28', 72, 92), y true represents the pixel-level annotations from the set {unannotated = 0, abnormal = 1}, y pred represents the default label of the machine learning model, w is the weighting factor, and i refers to the pixel index in the anomaly detection image (20, 26, 26', 28, 28', 72, 92).

35. The method of any one of claims 24 to 34, wherein: A single critical value is applied to the calibrated anomaly detection image (98, 108) to detect anomalies (15).

36. The method of any one of claims 24 to 35, wherein: The user interface (128) is configured to allow a user to indicate the locations of a small number of defects (23) of each category of a subset of the defect categories in the anomaly detection image (20, 26, 26', 28, 28', 72, 92), wherein the user indicates the location of the defect (23) by selecting a single pixel of the defect (23), preferably a pixel in a central area, and automatically generates an annotation (96) of the defect by selecting an area, preferably a circular area, surrounding the selected single pixel, and samples the anomaly detection image value from the area surrounding the selected pixel.

37. The method of claim 36, wherein: The size of the region is selected based on the critical dimension and / or critical distance and / or pitch size.

38. A method as claimed in any one of the preceding claims, wherein The imaging dataset (12) comprises one or more multi-sensory images.

39. A method as claimed in any one of the preceding claims, wherein: The imaging data set (12) is a multi-beam SEM image.

40. A method as claimed in any one of the preceding claims, wherein The imaging data set (12) is a focused ion beam SEM image.

41. The method of any preceding claim, further comprising measuring one or more characteristics of the detected anomaly (15).

42. The method of claim 41, further comprising controlling at least one wafer fabrication process parameter based on the one or more measured characteristics.

43. The method of claim 42, further comprising assessing a quality of the wafer (120) based on the one or more measured characteristics and at least one quality assessment rule.

44. One or more machine-readable hardware storage devices comprising instructions executable by one or more processing devices (118) to perform operations comprising the method of any one of claims 1 to 43.

45. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of any preceding claim.

46. ​​A system (114) for controlling the quality of wafers (120) produced in a semiconductor manufacturing plant, the system (114) comprising: - an imaging device (116) adapted to provide an imaging data set (12) of the wafer (120); - one or more processing devices (118); - One or more machine-readable hardware storage devices comprising instructions executable by one or more processing devices (118) to perform operations including the method of claim 43.

47. A system (114') for controlling the production of wafers (120) in a semiconductor manufacturing plant, the system (114') comprising: - a production wafer member (130) for producing a wafer (120) controlled by at least one manufacturing process parameter; - an imaging device (116) adapted to provide an imaging data set (12) of the wafer (120); - one or more processing devices (118); - One or more machine-readable hardware storage devices comprising instructions executable by one or more processing devices (118) to perform operations including the method of claim 42.