Computer-implemented method for detecting anomalies in wafer imaging data sets and system utilizing such method

Abstract

A computer-implemented method for detecting anomalies in an imaging dataset of a wafer is provided. [Solution] A computer-implemented method for detecting anomalies includes steps of selecting a wafer imaging dataset (12) and hyperparameter values ​​that define machine learning models (16) for anomaly detection, training and evaluating the trained machine learning models (16) by calculating an objective function value, and selecting one of the trained machine learning models and applying it to anomaly detection.

Description

[Technical Field]

[0001] Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 324258, filed March 28, 2022, and German Patent Application No. 102022113352.4, filed May 25, 2022, which are incorporated herein by reference in their entireties.

[0002] The present invention relates to a computer-implemented method for detecting anomalies in an imaging dataset of a wafer, and also to a machine-readable hardware storage device, a system for controlling the production of wafers in a semiconductor foundry, and a system for controlling the quality of wafers produced in a semiconductor foundry.

[0003] The present invention is not limited to computer-implemented methods for wafers, but can also be applied to any other manufacturing object. [Background technology]

[0004] Quality control (QC) is used in production processes to iteratively improve the quality of the final product. Due to its central role, QC techniques not only need to be versatile, efficient, and flexible, but also adaptable to changing production conditions and use cases. Due to the lack of information about underlying defects, which may change with application or over time, QC systems must be based on broad assumptions to cover all defect types. Therefore, QC systems must be able to cope with cold-start conditions.

[0005] Cold start refers to the problem of a learning-based system being unable to draw inferences about items for which it has not yet gathered sufficient information. This problem occurs frequently in the semiconductor industry, where production processes and wafer types are constantly being adjusted. Therefore, cold start is a common problem in machine learning systems that involve automated data modeling, because machine learning models must be retrained from scratch every time system parameters are modified.

[0006] Unsupervised machine learning techniques, such as autoencoders, can successfully tackle the QC problem. During training, such techniques learn a compressed internal representation of abundantly available "clean" or "defect-free" data. As a result, the model is able to perfectly reconstruct defect-free image samples. During inspection, defects in the input image are not faithfully reconstructed. Spatial regions with large reconstruction errors indicate outliers with respect to the training data, also known as anomalies. Anomalies are localized deviations of an imaging dataset from an a priori defined norm, here deviations from the normed semiconductor structure.

[0007] However, not all anomalies are defects. For example, anomalies may include imaging artifacts, image acquisition noise, changes in imaging conditions, variations in semiconductor structures within the norm, rare semiconductor structures, or variations due to imperfect lithography, changes in manufacturing conditions, or changes in wafer processing. Such anomalies that are not defects but are detected by some anomaly detection method are called nuisances.

[0008] Machine learning models based on unsupervised learning derive information solely from data without requiring human input, such as annotation, which can be laborious, noisy, or impractical. However, expert knowledge is still required to supervise model training, i.e., to define various model parameters, called hyperparameters, such as the model design and its complexity (e.g., number of layers, number of filters per layer, size of the autoencoder bottleneck), its regularization (e.g., strategy and amplitude), data preprocessing techniques, dataset diversity, and learning strategy (e.g., learning rate and number of epochs). Expert selection of such hyperparameters is crucial for obtaining high-quality machine learning models.

[0009] In the prior art, hyperparameter optimization methods have been proposed that optimize the hyperparameter values ​​of a machine learning model by finding an optimal set of hyperparameter values ​​that minimize the expected validation loss of the machine learning model. An example of such a machine learning model suitable for anomaly detection is an autoencoder.

[0010] Hyperparameter optimization methods have been proposed to automatically search for optimal hyperparameter values ​​for machine learning models. Among them, the Neural Architecture Search (NAS) method has been proposed to automatically search for optimal hyperparameter values ​​that define the architecture of a neural network.

[0011] Among these hyperparameter optimization methods, various techniques can be used to predict good hyperparameter candidates, as disclosed in, for example, "Max-value Entropy Search for Multi-Objective Bayesian Optimization; S. Belakaria, A. Deshwal, J. Doppa; Conference on Neural Information Processing Systems 2019."

[0012] A known hyperparameter optimization framework called Optuna was disclosed, for example, in "Akiba, T., Sano, S., Yanase, T., Ohta, T. and Koyama, M., 2019, Optuna: A next-generation hyperparameter optimization framework, in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631."

[0013] Another known method for hyperparameter optimization is disclosed, for example, in U.S. Patent Application Publication No. 2020 / 0342329. In this disclosure, the autoencoder algorithm hyperparameters are optimized by evaluating the quality of the training progress, i.e., the history of the loss function values. The hyperparameter optimization method is unsupervised and does not use labeled training data.

[0014] US Patent Application Publication No. 2021 / 0256392 disclosed another hyperparameter optimization method in which hyperparameters are optimized by exploring a search space of hyperparameters and comparing the performance difference of corresponding anomaly detection networks.

[0015] However, applying the disclosed hyperparameter optimization method to anomaly detection machine learning models is not straightforward. The anomaly detection model must successfully reconstruct the defect-free portions of the image, but not the defects, for example, via mean squared error loss (L2 loss). Known hyperparameter optimization methods, such as Optuna or U.S. Patent Application Publication No. 2020 / 0342329, evaluate machine learning models using the same metric used during model training, often defaulting to the L2 metric, i.e., the mean squared error of predicted values. However, Optuna and similar prior art methods are not suitable for defect or anomaly detection because they converge toward a perfect reconstruction of the input image, including all possible defects, making anomaly or defect detection impossible.

[0016] Machine learning models for anomaly detection often suffer from low precision rates due to noise and high nuisance rates because not all anomalies are defects. Therefore, anomaly detection methods applied to wafer imaging datasets can face a very high nuisance rate n. This nuisance rate n is the inverse of the precision rate p, i.e., n = 1 - p, because so many, largely irrelevant, deviations on the wafer surface are discovered. Consequently, prior art anomaly detection methods require extensive post-processing to be useful for detecting defects on wafer surfaces. Additionally, many anomaly detection methods fail to provide pixel-level accurate anomaly detection results.

[0017] An improved anomaly detection method was presented in "Attention Guided Anomaly Detection and Localization in Images, S. Venkataramanan, KC. Peng, RV Singh, A. Mahalanobis, ECCV 2020." The proposed method is based on the idea of ​​using an attention mechanism to distinguish between non-anomalous and anomalous areas, so that the anomaly detection model is trained on non-anomalous data, which can provide improved results. However, this method is not applicable to semiconductor images. Furthermore, the generated attention map does not have pixel-level accuracy, which does not meet the high-precision requirements in the semiconductor field.

[0018] Another improved anomaly detection method was presented in "ESAD: End-to-end Deep Semi-supervised Anomaly Detection, C. Huang, F. Ye, Y. Zhang, Y Wang, Q. Tian, ​​Arxiv 2020." This paper proposes semi-supervised training of anomaly detection based on a large unannotated dataset in conjunction with a small annotated dataset. However, this method performs image-level anomaly detection, only indicating whether anomalies exist in an image without identifying their location. Again, this method does not meet the accuracy requirements in the semiconductor field.

[0019] A method for semantic segmentation was presented in "What's the point: semantic segmentation with Point Supervision; A. Bearman, O. Russakovsky, V. Ferrari, L. Fei-Fei; European Conference on Computer Vision 2016." This method addresses the problem of reducing user effort when annotating images by showing a single point of an object instead of pixel-accurate segmentation.

[0020] Methods for automated defect detection that meet precision requirements in the semiconductor industry include anomaly detection algorithms based on the die-to-die principle or the die-to-database principle. The die-to-die principle finds deviations from a typical or average wafer design by comparing a portion of the wafer with other portions of the same wafer. The die-to-database principle finds deviations from the ideal data by comparing a portion of the wafer with ideal simulation data from a database (e.g., a CAD file of the wafer).

[0021] Therefore, in defect detection methods such as die-to-die or die-to-database, the intermediate output is typically a difference image that shows the difference between the expected image (comparison data set) and the actual image. Anomaly suggestions are usually obtained by setting a threshold on the difference image. In this way, accurate anomaly detection can be obtained at the pixel level. However, setting the threshold is a balancing act between maximizing the capture rate (actual defects flagged as anomalies) and minimizing the nuisance rate (imaging artifacts, noise, defects of no interest, etc. flagged as anomalies). This step can be tedious, especially when defects with variations in shape, size, and appearance are to be detected.

[0022] Traditionally, threshold setting has been a manual process in which an expert probes through increasing thresholds. At each step, newly flagged anomalies are analyzed and threshold windows for various defect classes are selected. Alternatively, an expert provides several annotations for each defect class and then extracts thresholds. However, both approaches are search-intensive and wasteful for low-contrast defects, resulting in a high nuisance rate.

[0023] Therefore, it is an object of the present invention to improve anomaly detection in semiconductor wafers. A further object of the present invention is to improve the accuracy of machine learning methods for anomaly detection in semiconductor wafers. A further object of the present invention is to reduce user interaction required for selecting hyperparameter values ​​of machine learning models for anomaly detection. A further object of the present invention is to provide an anomaly detection method that includes hyperparameter optimization. A further object of the present invention is to make hyperparameter optimization applicable to machine learning methods for anomaly detection. A further object of the present invention is to provide an evaluation metric for determining the performance of machine learning models for anomaly detection. A further object of the present invention is to provide an improved anomaly detection method that includes improved post-processing methods, such as improved threshold setting methods. Summary of the Invention

[0024] The present invention addresses the problem of improving the accuracy of machine learning methods for anomaly detection, particularly for wafer imaging datasets. Furthermore, the present invention aims to improve the accuracy of machine learning techniques for anomaly detection without compromising recall.

[0025] This object is achieved by the invention as set forth in the independent claims. Advantageous embodiments and further developments of the invention are set forth in the dependent claims.

[0026] According to a first embodiment of the present invention, there is provided a method for reducing the effort required for selecting hyperparameter values ​​of a machine learning model for detecting anomalies or defects in semiconductor wafers. The method provides a way to automatically optimize at least one hyperparameter value of the machine learning model, thereby obtaining an optimized machine learning model for anomaly detection. Thus, according to the first embodiment of the present invention, a computer-implemented method for detecting anomalies comprises: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - selecting an optimized machine learning model from one of the at least two trained machine learning models based on an associated objective function value; - applying the optimized machine learning model to the wafer imaging dataset to detect anomalies; and the method step of selecting an optimized machine learning model includes, for each of the at least two trained machine learning models: - selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection; - training a machine learning model controlled by hyperparameter values ​​on an input subset of training data; - evaluating the trained machine learning model by calculating an objective function value associated with the objective function; Includes:

[0027] By selecting an optimized machine learning model from among multiple different machine learning models, the accuracy of the anomaly detection method is improved, while the user effort required to train the machine learning model is reduced.

[0028] According to a second embodiment of the present invention, a computer-implemented method for anomaly detection includes: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - repeating the following steps: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection; ii. training a machine learning model defined by hyperparameters based on the generated subset of training data; iii. Evaluating the trained machine learning model by calculating the associated objective function value of the objective function. - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; Includes:

[0029] By selecting one of the trained machine learning models based on the associated objective function value, the accuracy of anomaly detection is improved by optimizing the hyperparameters, e.g., architecture, of the machine learning model. At the same time, the user effort required to train the machine learning model is reduced. The trained machine learning model can be used to detect anomalies in an imaging dataset of a wafer. The imaging dataset can be obtained, for example, by means of a structured electron microscope.

[0030] In a machine learning model, a hyperparameter is a parameter whose value is used to control the learning process but is not learned from data. In contrast, the values ​​of other parameters (typically node weights) are derived from training data by training. According to an example of the first or second embodiment, the hyperparameter includes at least one of design hyperparameters or algorithmic hyperparameters. Design hyperparameters refer to the design of a machine learning model. Design hyperparameters include all hyperparameters related to the architecture of the machine learning model, such as the number of layers, layer size, layer type, filter size, kernel size of a convolutional layer, the type of convolution used, upsampling scheme, connections between layers, bottleneck size, bottleneck filter size, etc. An example of a model or design hyperparameter is the topology and size of a neural network. Algorithmic hyperparameters do not, in principle, affect the performance of the model, but affect the speed and quality of the learning process. Examples of algorithmic hyperparameters include the learning rate, mini-batch size, dropout rate, the size and content of the training dataset, and the type of loss function. A further example of an algorithm hyperparameter is the optimization algorithm used (stochastic gradient descent, Adam, RmsProp, etc.).

[0031] According to the first or second embodiment, the at least one hyperparameter defining the machine learning model for the detection of anomalies may include, but is not limited to, at least one of the following examples: Design hyper parameters: - Bottleneck size, - the size of the bottleneck filter (the number of features in the bottleneck), - initial filter size (number of filters in the first layer of the network; other network functions are scaled proportionally to the first layer), - the type of convolution used, - upsampling scheme, - connections between layers, - the number of layers in the model, - the size of the layers in the model, - the type of layers in the model, - filter size, - kernel size of convolutional layers, Algorithm hyperparameters: - Loss function type and / or parameters - initial learning rate, - learning rate decay factor, - Use of momentum, - number of epochs, - regularization scale, - the size and content of the training set (number of images), - a sample representing the dataset, - dropout rate, - Use of Nesterov's accelerated gradient, - Type of optimization algorithm.

[0032] In one example according to the first or second embodiment, the method according to the first or second embodiment further includes selecting a plurality of hyperparameters that jointly define a machine learning model for anomaly detection. For each selected hyperparameter, a set of associated hyperparameter values ​​is selected. Preferably, at least one hyperparameter to be optimized is related to the design of the machine learning model.

[0033] In one example of the method of the first or second embodiment, generating training data includes expert annotation of anomalies for a subset of the imaging dataset, which allows defining an objective function suitable for selecting an optimized machine learning model or for optimizing hyperparameters.

[0034] According to an example of the first or second embodiment, the subset of training data includes some samples of the training data, or may include the entire generated training data. The size of the subset of training data increases with the number of iterations, thereby reducing the calculation time.

[0035] In one example of the method according to the first or second embodiment, the input data and training data for the machine learning model may consist of tiles of a particular size of the imaging dataset. In one example according to the first or second embodiment, tiles (e.g., 2D images or 3D voxel arrays) are extracted from the imaging dataset and input to the machine learning model. The tiles may include sufficient spatial context for the anomaly to be detected. In one example according to the first or second embodiment, the tiles are at least as large as the expected anomaly and also incorporate spatial neighborhood context.

[0036] In one example according to the first or second embodiment, an autoencoder model is selected as the machine learning model, and the autoencoder machine learning model is trained to calculate a reconstructed subset of the imaging dataset of a wafer without anomalies.

[0037] An autoencoder machine learning model is an unsupervised machine learning method that requires minimal user input. An autoencoder model is a type of artificial neural network used in unsupervised learning that learns to efficiently code unlabeled data. An autoencoder includes two main parts: an encoder that maps inputs to codes and a decoder that maps codes to reconstructions of the inputs. The encoder and decoder neural networks can be trained to minimize the difference between the reconstructed representation of the input data and the input data itself. The codes are low-dimensional representations of the input data and can therefore be thought of as compressed versions of the input data. In one example according to the first or second embodiment, the autoencoder roughly reconstructs the input and preserves only the most relevant aspects of the data in the reconstruction. Therefore, autoencoders can be used to detect anomalies. Anomalies generally relate to rare deviations from a norm in an imaging dataset. Because such occurrences are rare, the autoencoder does not reconstruct this type of information, thereby suppressing anomalies in the imaging dataset.

[0038] In one example according to the first or second embodiment, in the anomaly detection step according to the first or second embodiment, anomalies in the imaging dataset are detected based on a comparison between the imaging dataset and a comparison dataset. The comparison dataset includes information that enables detection of anomalies when compared to the imaging dataset. In one example according to the first or second embodiment, the comparison dataset includes a reconstructed imaging dataset that is ideally free of defects. In one example according to the first or second embodiment, the reconstructed imaging dataset is reconstructed by an autoencoder or by principal component analysis. In one example according to the first or second embodiment, anomaly detection is performed using a distance measure or threshold operation applied to the comparison dataset.

[0039] Anomalies can be detected by comparing an imperfect reconstruction of the imaging dataset with the original imaging dataset. Differences between the input image and the reconstructed representation of the input image indicate an anomaly. A distance measure between the input image and the reconstructed representation of the input image can be used to quantify whether an anomaly exists. The larger the difference, the more likely that tile contains an anomaly. Detecting anomalies can include applying one or more thresholds to the difference image between the original and reconstructed images. In one example according to the first or second embodiment, a local threshold can be applied to a subset of the difference image. Additional measurements, such as the magnitude, location, or shape of the difference, or their local distribution, can also be used to detect anomalies. This allows for the detection of defects of interest while minimizing deviations due to noise.

[0040] In one example according to the first or second embodiment, the method for anomaly detection further includes defining an objective function suitable for selecting an optimized machine learning model or for optimizing hyperparameters. The objective function according to this example can include at least one model evaluation measure.

[0041] The model evaluation metric is a quantifiable expression measuring a property of the trained machine learning model, and may include measuring at least one property related to the performance of the trained machine learning model, the quality of anomaly detection, the complexity of the trained machine learning model, the effort or cost to apply the trained machine learning model, etc. In one example according to the first or second embodiment, multiple model evaluation metric can be defined and appropriately combined in the objective function. In one example, previously generated validation data is provided to evaluate the model evaluation metric.

[0042] In one example according to the first or second embodiment, the objective function includes at least two model evaluation measures. A method using an objective function including at least two model evaluation measures is particularly applicable to anomaly detection because the combination of different model evaluation measures prevents a trained machine learning model, such as an autoencoder, from reconstructing defects and ensures that the comparison dataset deviates sufficiently from the imaging dataset in the presence of defects. In this way, human effort is reduced, anomaly detection performance is maximized, and the quality, reproducibility, and stability of the results are improved. Additionally, cold starts are possible.

[0043] The objective function is a measure of the quality of the trained machine learning model and therefore of the quality of the hyperparameter values ​​selected for this model. By applying an objective function that includes at least two different model evaluation measures, it can be ensured that the optimization of the hyperparameter values ​​according to the first or second embodiment does not result in a perfect reconstruction of the training data, including anomalies, since all model evaluation measures are calculated simultaneously and therefore contribute to the objective function value of the objective function.

[0044] The objective function for hyperparameter optimization can include or be a loss function used during training of a machine learning model as a model evaluation measure. The model evaluation measure can also include or be an Lp-norm loss function (p≧1), e.g., an L2 loss function. The Lp-norm loss function can measure the deviation of training data samples from corresponding target data samples, e.g., the deviation of training data samples from an anomaly-free reconstruction, e.g., the deviation of training data samples from an autoencoder reconstruction. This ensures correct reconstruction of anomaly-free image regions and improves anomaly detection results.

[0045] In one example according to the first or second embodiment, at least one of the model evaluation measures includes a discriminative loss function that evaluates the difference between expert annotations of the anomalies and the detected anomalies. The expert annotations include labels assigned to pixels or regions in the imaging dataset, the labels indicating, for example, anomaly or no anomaly. Thus, also or alternatively, the objective function may be a discriminative loss function L for defect / non-defect using a small number of pixel-wise annotations provided by an expert user. CE In one example of the first or second embodiment, the difference between the original sample and the encoded sample (original-encoded) is kept as a continuous value but clipped to the range (0,1). This value is directly compared to a binary label (anomaly / no anomaly) through cross-entropy loss. Note that the provided example need not encompass all defects in the dataset. Preferably, less than 10%, and more preferably less than 1%, of the training data are expert annotations. In addition, the expert annotations assign labels only to a subset of anomalies and / or types of anomalies present in the imaging dataset. In this way, new defects introduced into the imaging dataset do not specifically require expert user annotation, thereby minimizing the required expert user input and enabling a cold start. Note also that these amounts of training data are not sufficient to train a single model.

[0046] According to one example according to the first or second embodiment, expert annotations can be provided via multiple formats (e.g., bounding boxes, click points, auxiliary process information, etc.), and the discrimination loss (L CE ) is adapted accordingly. For example, the bounding box annotation is computed using an overlap measure, i.e., the intersection-over-union loss L IOU is related to.

[0047] In one example of the first or second embodiment of the present invention, the expert annotations are provided, for example, as pixel-by-pixel annotations or as bounding boxes. These expert annotations are facilitated using the expert's knowledge, for example, based on a critical distance, critical dimension, or pitch size associated with the minimum size of a structure on the wafer. In this way, the expert only needs to click on the center of a defect in the imaging data set, and a large area corresponding to the critical distance, critical dimension, or pitch size is automatically assigned as a defect around the click point.

[0048] To avoid overfitting, one of the model evaluation measures may further include a measure of the complexity of the machine learning model, for example, an Occam's razor penalty for the complexity of the machine learning model. This model evaluation measure may be the total number of floating-point operations (L FLOP ), e.g., consider the logarithm of this number. Alternatively, the number and / or size of layers of the trained neural network, and / or the number of connections between neurons, or other suitable hyperparameter values, can also be used by model evaluation metrics to measure the complexity of machine learning models.

[0049] In one example of the first or second embodiment of the present invention, the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1) that measures the deviation of training data samples from corresponding target data samples (e.g., the deviation of training data samples from their autoencoder reconstructions) and a discriminative loss function that evaluates the difference between the model predictions of anomalies and the expert annotations. Considering this particular combination of model evaluation measures simultaneously 1) evaluates the accuracy of predictions on anomaly-free data (Lp-norm loss), 2) evaluates the accuracy of predictions on anomalous data (discriminative loss), and 3) specifically selects the expert annotations for anomalies to prevent class imbalance, thereby ensuring very high accuracy of anomaly detection results.

[0050] In one example of the first or second embodiment of the present invention, the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1) that measures the deviation of training data samples from corresponding target data samples (e.g., the deviation of training data samples from their autoencoder reconstructions), a discriminative loss function that evaluates the difference between the expert annotations of anomalies and the model predictions, and a measure of machine learning model complexity. Considering this particular combination of model evaluation measures simultaneously: 1) evaluates the accuracy of predictions on anomaly-free data (Lp-norm loss), 2) evaluates the accuracy of predictions on anomalous data (discriminative loss), 3) prevents class imbalance by specifically selecting the expert annotations for anomalies, and 4) prevents overfitting (complexity measure), thereby ensuring particularly high accuracy of anomaly detection results. In one example, the objective function includes, for example, three model evaluation measures, for example: f=w1L2+w2L CE +w3L FLOP where the weights w1, w2, and w3 are user-configurable. Typically, the first two weights are similar in magnitude, while the third weight is at least ten times lower, e.g., w1=10, w2=1, and w3=0.01. One or more of the weights can be set to 0; for example, w3 can be 0, so that the objective function does not penalize the model complexity. Other L norms can be used instead of the L2 norm. p We can also use the norm. CE is an arbitrary loss function that penalizes the deviation of the difference image for anomalies in the imaging dataset from some ground truth data, e.g., the intersection-over-union loss L IOU Additional model performance measures can also be added. In addition, other options from the Optuna library may be chosen.

[0051] In one example according to the first or second embodiment, the objective function includes a quality value as a model evaluation measure for evaluating the quality of the trained machine learning model, similarly or alternatively to any of the above sections, and the user interface is configured to present information about the trained machine learning model to a user and allow the user to indicate the quality value. The information presented to the user may include a precision rate, a recall rate, sample anomaly detection, hyperparameter values ​​associated with the trained machine learning model, such as the model's design, filters learned by the model for one or some of its layers, etc. When the objective function is minimized, if the quality of the trained machine learning model is high, the user can select a low value as the quality value. If the quality of the trained machine learning model is low, the user can select a high value as the quality value.

[0052] In one example according to the first or second embodiment, the hyperparameter values ​​are selected according to a sampling strategy that determines which hyperparameter values ​​to select next based on previously selected hyperparameter values ​​and associated objective function values. In one example according to the first or second embodiment, the sampling strategy includes the use of a sampling algorithm.

[0053] An example of a sampling strategy is the tree-structured Parzen estimator (TPE), which handles categorical hyperparameters in a tree-structured manner. For example, the number of layers in a neural net and the number of neurons in each layer define the tree structure. For example, there cannot be a third layer without a second layer, and setting the number of neurons only makes sense if this layer is present in the graph. Another example is the selection of an optimizer for a machine learning model, since each optimizer can have its own set of parameters.

[0054] Preferably, the sampling strategy for selecting the hyperparameter values ​​includes taking into account the hyperparameter values ​​and corresponding values ​​of the objective function from one or more prior iterations by optimizing one criterion selected from the group including expected improvement, maximum probability of improvement, and upper confidence interval, thereby improving the performance of the method for selecting an optimized machine learning model and improving anomaly detection results.

[0055] In one example of the first or second embodiment, the method further includes evaluating an improved gain function. The gain function is a surrogate model of validation loss as a function of hyperparameter values ​​that can be fitted to previously obtained objective function values, also called prior observations, and configured to predict where a local optimum of the objective function is located. These methods are further referred to as sequential model-based optimization (SMBO). The surrogate model can function as at least a portion of the objective function. Using the surrogate model simplifies the evaluation of the objective function and reduces computational time because the costly step of evaluating the objective function is performed less often.

[0056] For example, we can use the probability of improvement (PI) as the improvement gain function, which evaluates the objective function f at the point that is most likely to improve this value. The objective function f is to be minimized. Let f' be the smallest value of f observed so far, and let D be the previous observation, i.e., the objective function value obtained previously. This then corresponds to the following utility function (corresponding to a set of hyperparameters) associated with the evaluation of f at a given point x:

[0057]

number

number

[0058] An alternative improvement capture function that takes into account the magnitude of improvement is expected improvement (EI). Expected improvement evaluates f at the point that most improves the expectation of f'. This corresponds to the following utility function: u(x)=max(0,f'-f(x))

[0059] The expected gain function is then the expected utility as a function of x:

number

[0060] The point with the most expected improvement (maximum expected utility) is selected. The expected improvement has two components. The first component can be increased by decreasing the mean function μ(x). The second component can be increased by increasing the variance K(x;x). These two terms can be interpreted as explicitly encoding the trade-off between exploitation (valuing at a point with a low mean value) and exploration (valuing at a point with high uncertainty). The exploitation-exploration trade-off is a classical consideration in such problems, and as a consequence of the Bayesian decision-theoretic treatment, the expected improvement criterion automatically captures both.

[0061] An alternative improvement gain function is commonly known as the upper bound confidence interval (UCB). It is generally described in terms of maximizing f rather than minimizing f, but in a minimization context, the improvement gain function takes the form a UCB (x;β)=μ(x)-βσ(x) where β>0 is a trade-off parameter,

[0062]

number

[0063] For each of the described sampling strategies, at least one set of hyperparameter values ​​can be associated with a probability distribution indicating the likelihood of each hyperparameter value being selected by the sampling strategy. The probability distribution can be predefined and therefore independent of the objective function value. The probability distribution indicating the likelihood of each hyperparameter value being selected by the sampling strategy can be modeled based on application-based prior knowledge, preferably based on design knowledge such as imaging hardware settings or critical distances. In this way, prior knowledge about the parameter space can be integrated into the method step of selecting an optimized machine learning model in the first embodiment or the iterative step in the second embodiment. Therefore, the improved sampling strategy reduces computation time, thereby improving anomaly detection results.

[0064] In one example, the sampling strategy includes at least two different sampling strategies for the at least two trained machine learning models of the first embodiment, or the machine learning models trained during the iterative step of the second embodiment, resulting in improved results through a more thorough exploration of the hyperparameter value space.

[0065] Optionally, the step configured to select an optimized machine learning model according to the first embodiment or the iterative step according to the second embodiment includes a pruning strategy, e.g., a pruning algorithm that determines whether to continue or interrupt training of a given machine learning model. In one example, the pruning strategy includes an early stopping criterion. In a further example, the pruning strategy includes an asynchronous successive halvings strategy. In a further example, a machine learning model generated by sampling a set of hyperparameter values ​​may first be tested based on an objective function for a small subset of training data samples. If performance is poor, the sampled hyperparameter values ​​can be discarded early; otherwise, the size of the training data subset can be increased. This pruning strategy reduces computation time. By applying the pruning strategy, hyperparameter values ​​are tested and discarded early if the objective function does not show good results. This saves computation time and makes the method applicable to cold-start scenarios, where retraining is often required.

[0066] According to a third embodiment of the present invention, there is provided a computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method including the steps of: - generating an anomaly-detected image by applying an anomaly detection method to the imaging dataset; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of the subset of defect classes; ii. calibrating the anomaly detection image by at least one calibration method comprising the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly-detected image; b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - Applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing nuisance and highlighting defects in the anomaly detection image.

[0067] In a fourth embodiment of the present invention, a computer-implemented method for detecting anomalies in an imaging dataset of a wafer is provided, the imaging dataset including defects belonging to multiple defect classes. - generating an anomaly-detected image by applying an anomaly detection method to the imaging dataset; - providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of the subset of defect classes; - detecting anomalies in the anomaly detection image by at least one calibration method, the calibration method comprising the steps of: training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly-detected image; Applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; Applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing nuisance and highlighting defects in the anomaly detection image.

[0068] The method according to the third or fourth embodiment of the present invention further enables the application of machine learning methods for anomaly detection to defect detection, requiring minimal user input, thereby reducing annoyances in the anomaly detection image while simultaneously highlighting anomalies or defects. Based on this enhanced or calibrated anomaly detection image, robust anomaly detection can be achieved by applying a single threshold.

[0069] According to aspects of the third or fourth embodiment, the method further comprises automatically setting the threshold by using available side information (e.g., information related to the size of wafer structures, such as critical distances, critical dimensions, pitch sizes, etc.) or by providing annotated defects of at least a subset of defect classes. To this end, the method according to the third or fourth embodiment relates to a solution workflow that calibrates anomaly detection images, e.g., difference images, to be robust to defect size and / or contrast and to highlight defects of interest while suppressing defects due to noise.

[0070] The concept of "anomaly localization" refers to any method that computes the location of an anomaly, such as anomaly segmentation methods, semantic anomaly segmentation methods, anomaly detection methods, classification methods, regression methods, methods for finding out-of-distribution samples, etc.

[0071] The number of defect classes can relate to a single defect class, multiple defect classes, or all defect classes occurring in the imaging data set.

[0072] The term "anomaly detection image" refers to the output of the anomaly detection method in the form of an image showing an anomaly, for example by pixel-wise labeling or bounding boxes. In one example of the third or fourth embodiment, the anomaly detection image is a difference image between an imaging dataset and a comparison dataset, for example a reconstruction of the imaging dataset. In one example, the comparison dataset is based on a die-to-die or die-to-database principle.

[0073] In one example of the third or fourth embodiment, the comparison dataset may include a reconstructed representation of the imaging dataset generated by training a machine-learning autoencoder on the imaging dataset or a subset thereof and applying the trained autoencoder to the imaging dataset to obtain a reconstructed imaging dataset. The generation of such an autoencoder and the reconstructed imaging dataset is discussed in the description of the first or second embodiment of the present invention.

[0074] In this way, the quality of the anomaly detection images is improved. Instead of an autoencoder, principal component analysis can also be used to generate the reconstructed dataset.

[0075] In one example of an embodiment of the third or fourth embodiment, the anomaly detection method is a method according to the first or second embodiment of the present invention.

[0076] In one example of the third or fourth embodiment, the at least one calibration method is selected from a group of calibration methods, and in one example, the steps of determining calibrated anomaly-detection images and applying a single threshold to the calibrated anomaly-detection images are repeated iteratively for each calibration method in the group of calibration methods.

[0077] In one example of the third or fourth embodiment, the method further includes selecting, e.g., by user input, one or more values ​​in a domain of a distribution of values ​​of the anomaly-detected image as thresholds. The thresholds may include minimum and maximum values ​​in the domain of the distribution of values ​​of the anomaly-detected image, e.g., minimum and maximum intensity values ​​of the anomaly or defect. This method allows multiple adapted thresholds to be applied in combination with filters.

[0078] For example, the anomaly detection image can be calibrated by applying at least one calibration method that includes the following steps: - for each defect class of the subset of defect classes, calculating an intermediate calibrated anomaly detection image by adapting the anomaly detection image value based on one or more selected thresholds of the anomaly detection image value for the current defect class; - applying one or more filters for the current defect class, e.g., a size filter; and finally - applying the operator to all intermediate calibrated anomaly detection images to generate calibrated anomaly detection images.

[0079] Throughout this specification, the term "subset" of a set refers to a single member, some members, or all members of the set.

[0080] In one example of the third or fourth embodiment, the operator is selected from the group including pixel-wise sum, pixel-wise average, pixel-wise minimum, pixel-wise maximum, and pixel-wise scaling. In this way, the final calibrated anomaly-detected image includes defects from different defect classes, each of which is extracted from the original anomaly-detected image based on a different set of thresholds and filters. Using the maximum operator preserves as many anomalies as possible (provided the anomalies are marked with values ​​higher than background), while the minimum operator further reduces noise and nuisance.

[0081] According to a second example of the third or fourth embodiment, a method includes providing annotated defects. The annotated defects are used to automatically set required thresholds to be applied to the calibrated anomaly detection image. According to the second example, only a small number of annotated defects are required for a small number of defect classes, and other defect classes can remain unannotated. The method according to this example relies on the assumption that the annotated defects cover the appearance spectrum of all defect classes. A small number of defect classes means that the subset of defect classes includes less than 50% of all defect classes, preferably less than 30% of all defect classes, more preferably less than 20% of all defect classes, and most preferably less than 10% of all defect classes. A small number of annotated defects per class means more than 5 but less than 20, preferably more than 5 but less than 10, annotated defects per defect class. The method according to the second example addresses the issue of defect annotation and can automatically set thresholds for the calibrated anomaly detection image.

[0082] In the calibration method according to the second example, the task of automatically determining the threshold can be formulated as a pixel-wise localization problem, e.g., a pixel-wise segmentation problem. Many different designs of the pixel-wise localization of the second embodiment are possible depending on various assumptions, such as the type of annotation (bounding box, click point, pixel-level or image-level annotation, multi-user annotation, annotation obtained from secondary sources, etc.), loss function (addressing model complexity through regularization, incorporating prior knowledge, handling class imbalance, etc.), or problem formulation (semantic segmentation, object detection, classification, regression, handling out-of-distribution samples, etc.).

[0083] In one example of the third or fourth embodiment, a machine learning model for anomaly localization is trained to optimize a loss function based on anomalous and non-anomalous samples. This allows for pixel-by-pixel detection of important anomalies in the semiconductor industry with high accuracy. Samples from the distribution of anomaly detection image values ​​for each defect class of a subset of defect classes can be used as anomalous samples. In a further example, the loss function is a semi-supervised loss function. Based on the semi-supervised loss function, expert annotations can be taken into account during training of the calibration method, thereby improving the quality of anomaly detection. However, user effort is kept low because only a small amount of annotation is required and most of the samples are selected automatically. Throughout this specification, the term "foreground sample" is used synonymously with "anomalous sample," and the term "background sample" is used synonymously with "non-anomalous sample."

[0084] Specifically, a sample from the distribution of values ​​of the anomaly-detected image for each defect class of the subset of defect classes can be used as the anomaly sample, while a non-anomaly sample can be (automatically) selected from the remaining pixels of the anomaly-detected image. A remaining pixel of the anomaly-detected image can be selected as a non-anomaly sample if its anomaly-detected image value is below a threshold. Additionally or alternatively, each non-anomaly sample can be selected from the distribution of values ​​of its anomaly-detected image a i In particular, the negative exponential weighting function w(a i )=exp(-a i) can be used to weight the non-anomalous pixels. This allows for automatic selection of a large number of non-anomalous pixels with minimal user effort. Furthermore, high weights can be assigned to pixels with low anomaly detection image values, i.e., pixels with very small autoencoder reconstruction errors, which are least likely to belong to an anomaly, ensuring high accuracy in the selection of non-anomalous pixels. Therefore, by selecting only a few samples from the distribution of anomaly detection image values ​​for each defect class in the subset of defect classes, it is possible to prevent samples from these unselected distributions from being mistakenly used as non-anomalous samples. Additionally, a small number of selected samples is sufficient to obtain high accuracy in anomaly detection, thereby reducing user effort.

[0085] The user is required to provide only a few pixel-level annotations for a subset of defect classes. The subset can include all defect classes. Preferably, the subset does not include all defect classes. More preferably, the subset includes only a small number of defect classes, e.g., less than 10% of the defect classes. The annotation process can be facilitated by leveraging available meta-information (e.g., critical dimensions, critical distances, or pitch sizes). For example, the user can prepare click points that are processed into pixel-level annotations by expanding the click points into circles of a size corresponding to the critical dimensions, critical distances, or pitch sizes that represent the lower or upper limits of the minimum feature size on the wafer. Alternatively, the user can use a brush to mark defective pixels. Based on these annotations, a machine learning anomaly localization model, e.g., an anomaly segmentation or anomaly detection model, can be trained. When applied to the anomaly detection image, this localization model labels each pixel as an anomaly of interest or a nuisance. In this way, anomalies are highlighted and nuisances are mitigated. Additional post-processing techniques, such as histogram equalization, location-based or size-based filtering, can be used to further reduce nuisance. The calibrated anomaly detection image is finally thresholded using a single threshold, e.g., 0.5, to obtain defects in the imaging dataset.

[0086] In one example of the third or fourth embodiment of the present invention, during training, the machine learning model for anomaly localization is a machine learning model for anomaly segmentation. This model takes as input a partially annotated anomaly detection image, such as a difference image. As output, the model labels each pixel as either a target anomaly or a nuisance. Since not all input pixels are annotated by the user, model training is based on the following semi-supervised loss function:

[0087]

number

[0088]

number

number

number

[0089] Alternatively, various other loss functions can be used to train machine learning models for anomaly localization, such as the Kullback-Leibler divergence loss function, L1 or L2 loss function, etc.

[0090] An example of the third or fourth embodiment provides a method for calibrating differential images, allowing high recall and manageable precision to be obtained by applying a single threshold to the calibrated anomaly detection images. Thus, a machine learning model according to the third or fourth embodiment includes method steps for setting up, training, and applying a nonlinear filter configured to amplify defects while suppressing nuisances in differential images used for semiconductor defect detection.

[0091] In one example according to the third or fourth embodiment of the present invention, a user interface is configured to allow a user to indicate the location of a small number of defects of each class of a subset of defect classes in the anomaly detection image. To this end, the user interface is configured for user indication of the defect location by selecting a single pixel, preferably a pixel in a central region, of the defect. An annotation of the defect is automatically generated by selecting an area, preferably a circle, surrounding the selected single pixel, and a value in the anomaly detection image is sampled from the area surrounding the selected pixel. Preferably, the size of the area is selected based on side information such as a critical dimension and / or a critical distance and / or a pitch size. The critical distance and critical dimension relate to the minimum size of a structure on the wafer, and the pitch size relates to the minimum distance between structures, and therefore can be understood as an upper limit on the minimum size of a wafer structure. In other words, an anomaly can be assumed to be at least as large as the smallest structure on the wafer. Therefore, the generated anomaly area surrounding the selected pixel can be automatically selected as the minimum size of a wafer structure in terms of the critical dimension, critical distance, or pitch size (converted to pixels). In this way, the number of labeled samples is automatically increased, thereby improving the accuracy of anomaly detection. In addition, by limiting the size of the anomaly region surrounding the selected pixel based on the size of the wafer structure, it is possible to prevent erroneous labeling, especially for very small structures on the wafer. Furthermore, a single click is sufficient for annotating defects, significantly reducing the user's annotation effort. Thus, despite the limited availability of training data at the beginning of training, a cold start is feasible because only a very small number of selected samples are sufficient to train a machine learning model for anomaly localization.

[0092] The anomaly detection method according to the first or second embodiment and the anomaly detection method according to the third or fourth embodiment can be trained in many variations. They can be trained jointly as a single module, which makes maintenance and evaluation easier. On the other hand, independent modules have the advantages of versatility and isolation of expert opinion.

[0093] It should be noted that a computer-implemented method for the detection of anomalies in an imaging dataset can simultaneously include both hyperparameter optimization according to either the first or second example embodiment and anomaly detection image calibration according to either the third or fourth example embodiment, and all features according to the embodiments or examples can also be applied to this combined approach.

[0094] Anomaly detection is the first step in the workflow of defect detection (and classification, if possible). In particular, to have a method suitable for cold starts, anomaly detection is a valuable first step to scan large amounts of data, so that only samples that are likely to contain defects are presented to the user for annotation.

[0095] Additionally, one or more properties of the detected anomalies can be measured, such as their size, location or shape parameters, or the density of anomalies over a specific region or over the entire imaging data set. Based on such measurements, at least one wafer manufacturing process parameter can be controlled based on the one or more measured properties, or the quality of the wafer can be assessed based on the one or more measured properties and at least one quality assessment rule.

[0096] In this way, the detected anomalies can be used to control the quality of wafers produced in a semiconductor foundry or to control the wafer production process in a semiconductor foundry.

[0097] Additionally, one or more machine-readable hardware storage devices may include instructions executable by one or more processing devices to perform operations including any of the methods disclosed herein.

[0098] An inspection system for controlling the quality of wafers produced in a semiconductor manufacturing facility comprises the following features: an imaging device adapted to provide an imaging dataset of the wafer; an optional graphical user interface configured to present data to a user and obtain input data from the user; one or more processing devices; and one or more machine-readable hardware storage devices containing instructions executable by the one or more processing devices to perform operations comprising one of the methods disclosed herein, including assessing the quality of the wafer based on one or more measurements and at least one quality assessment rule.

[0099] A system for controlling production of wafers in a semiconductor fabrication factory comprises: means for producing wafers controlled by at least one manufacturing process parameter; an imaging device adapted to provide an imaging dataset of said wafers; an optional graphical user interface configured to present data to a user and obtain input data from the user; one or more processing devices; and one or more machine-readable hardware storage devices containing instructions executable by the one or more processing devices to perform operations including a method comprising controlling at least one wafer manufacturing process parameter based on one or more measurements.

[0100] According to the embodiments described herein, various imaging modalities can be used to acquire imaging datasets for defect detection and classification. Different imaging datasets may be acquired with various imaging modalities. The imaging dataset may include one or more multisensory images. The imaging dataset may be a multibeam SEM image or a focused ion beam image, for example, generated by a helium ion microscope (HIM). The imaging dataset may include a two-dimensional image, a three-dimensional image, a slice-by-slice three-dimensional image, or a multisensory fusion image. For example, the imaging dataset may include a 2D image. Here, a multibeam SEM may be employed. A multibeam SEM uses multiple beams to simultaneously acquire images at multiple fields of view. For example, 50 or more or 90 or more beams may be used. Each beam covers a different portion of the wafer surface. This allows for the acquisition of a large imaging dataset in a short time. Typically, 4.5 gigapixels are acquired per second. For example, imaging one square centimeter of a wafer with a pixel size of 2 nm results in 25 terapixels of data. Other examples of imaging datasets including 2D images relate to imaging modalities such as optical imaging, phase contrast imaging, and X-ray imaging. It is also possible for the imaging dataset to be a volumetric 3D dataset that can be processed slice by slice or as a three-dimensional volume. Cross-beam imaging devices including a focused ion beam (FIB) source, an atomic force microscope (AFM), or a scanning electron microscope (SEM) can be used here. Multimodal imaging datasets may also be used, for example, combining X-ray imaging and SEM. The imaging dataset 22 may additionally or alternatively include an aerial image obtained by an aerial imaging system. The aerial image is a radiation intensity distribution at substrate level. The aerial image can be used to simulate the radiation intensity distribution generated by the photolithography mask 14 during a photolithography process.The aerial image measurement system may comprise, for example, a staring array sensor, or a line scan sensor, or a time delay integration (TDI) sensor.

[0101] Although examples and embodiments of the present invention are described with respect to semiconductor wafers, it should be understood that the present invention is not limited to semiconductor wafers, but may also be applied, for example, for semiconductor manufacturing masks, or in a variety of other fields, such as anomaly detection in manufacturing parts or biological samples.

[0102] It should be noted that the present invention described by way of examples and embodiments is not limited to these embodiments and examples, and that those skilled in the art can implement various combinations or modifications thereof. In the following, advantageous exemplary embodiments of the present invention will be described and are shown diagrammatically in the drawings. [Brief explanation of the drawings]

[0103] [Figure 1] FIG. 1 is a diagram of a schematic defective cell structure containing multiple anomalies due to various defects. [Figure 2] 1A-1C are a flowchart and results of applying an anomaly detection method such as an autoencoder to a wafer imaging dataset. [Figure 3] 3 is a flowchart of an anomaly detection method according to the first or second embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing the results of an anomaly detection method obtained by a general hyperparameter optimization method. [Figure 5] 4 is a flowchart of an anomaly detection method according to an example of the first or second embodiment of the present invention. [Figure 6] FIG. 2 illustrates the results of an anomaly detection method according to an example of the first or second embodiment of the present invention, based on an objective function including at least two different model evaluation measures. [Figure 7] FIG. 1 illustrates the sampling strategy of the tree-structured Parzen estimator. [Figure 8]FIG. 1 illustrates the pruning strategy of the asynchronous successive halvings algorithm. [Figure 9] FIG. 1 shows the evolution of objective function values ​​for different machine learning models defined by hyperparameters selected according to the sampling strategy of the tree-structured Parzen estimator. [Figure 10] 10 is a flowchart of a computer-implemented method for detecting anomalies in an imaging dataset of a wafer according to a third or fourth embodiment of the present invention. [Figure 11] 10 is a flowchart of a computer-implemented method for detecting anomalies in an imaging dataset of a wafer according to an example of the third or fourth embodiment of the present invention. [Figure 12] 5A-5C illustrate steps of a computer-implemented method for the detection of anomalies in an imaging dataset of a wafer according to a third or fourth embodiment of the present invention. [Figure 13] 10A-10C show a comparison of an anomaly-detection image obtained by a computer-implemented method for detecting anomalies in an imaging dataset of a wafer according to the third or fourth embodiment of the present invention with a calibrated anomaly-detection image; [Figure 14] 10A-10D show confusion matrices for anomaly-detected images and calibrated anomaly-detected images obtained by a computer-implemented method for detecting anomalies in an imaging dataset of a wafer according to the third or fourth embodiment of the present invention; [Figure 15] 1 is a schematic diagram of a system for controlling the quality of wafers in a semiconductor manufacturing factory. [Figure 16] 1 is a schematic diagram of a system for controlling wafer production in a semiconductor manufacturing factory. DETAILED DESCRIPTION OF THE INVENTION

[0104] 1 shows a schematic defective cell structure 11 that includes multiple anomalies 15. The anomalies 15 are localized deviations of an imaging data set 12 from an a priori defined norm, here deviations from a normed semiconductor structure.

[0105] FIG. 2 illustrates a flowchart and results of applying an anomaly detection method, such as an autoencoder, to an imaging dataset 12 of a wafer 120. The input 14 of the method includes the imaging dataset 12 of the wafer 120, which includes one or more images of the wafer 120. A machine learning model 16 is trained based on the imaging dataset 12. This machine learning model 16 can be based on a die-to-database principle or a die-to-die principle. The autoencoder model is trained based on the imaging dataset 12 based on a die-to-die principle. The autoencoder learns a compressed internal representation of abundantly available "clean" or "defect-free" data. As a result, the machine learning model 16 can perfectly reconstruct defect-free image samples. During inspection, contaminated input images are not faithfully reconstructed. Spatial regions with large reconstruction errors indicate outliers with respect to the training data, also known as anomalies 15. Based on the autoencoder model, a comparison dataset in the form of a reconstruction 18 of the input 14 is calculated. Because defects 23 in the imaging dataset 12 are not reconstructed by the autoencoder, the difference between the reconstruction 18 (comparison dataset) and the input 14 includes defects 23 in the imaging dataset 12. Not all such differences are attributable to defects 23, as imaging artifacts such as noise, semiconductor structure variations, or lithography imperfections also cause differences in the difference image. Thus, defects 23 only form a subset of anomalies 15. Anomalies 15 that are not defects 23 but are detected by some anomaly detection method are referred to as nuisances. The difference image shows the anomalies 15 in the imaging dataset 12. Thus, the difference image is the anomaly detection image 20.

[0106] Based on a defect-free input image 22, a reconstructed image 24 is obtained that differs only due to noise as shown in difference image 26. An input image 22' containing a defect 23 is reconstructed into reconstructed image 24', except for the area containing the defect 23, which is only partially reconstructed. Thus, difference image 26' contains deviations from zero at the defect locations. The defects can be localized by applying a threshold to difference image 26'.

[0107] Because autoencoders are machine learning models based on unsupervised learning, they derive information solely from input data without human input. However, expert knowledge is still required to supervise model training by defining hyperparameter values ​​(e.g., bottleneck size) for the hyperparameters that define the machine learning model. To reduce required user interaction and improve the reconstruction results of an autoencoder, a computer-implemented method 10 for detecting anomalies 15 according to the first or second embodiment of the present invention, as shown in FIG. 3 , includes the following steps: in a data selection step 19, selecting an imaging dataset 12 of a wafer 120; in a training data generation step 21, generating training data from the imaging dataset 12 and repeating the following steps: in a hyperparameter value selection step 25, selecting hyperparameter values ​​from a related hyperparameter value set or selecting multiple hyperparameters from multiple related hyperparameter value sets based on a sampling strategy; A selecting step, wherein the hyperparameter values ​​correspond to at least one hyperparameter that defines a machine learning model 16 for detecting anomalies 15; a training step 27, training the machine learning model 16 defined by the hyperparameters 48 based on a subset of the generated training data, and a model evaluation step 29, evaluating the trained machine learning model 16 by calculating an associated objective function value of the objective function; a model selection step 31, selecting one of the trained machine learning models based on the associated objective function value and applying it to the imaging dataset 12 of the wafer 120 to detect anomalies 15.

[0108] The objective function ensures that the hyperparameter or hyperparameters are selected such that the machine learning model defined by them correctly reconstructs the imaging dataset 12 excluding regions containing defects 23. To this end, the objective function may include a model evaluation measure in the form of an Lp-norm loss function (p≧1), which ensures correct reconstruction of the anomaly-free imaging dataset 12 by penalizing deviations. For example, the L2-norm loss function, which is a mean squared error loss function, is the default choice as the objective function for most hyperparameter optimization methods.

[0109] However, when the objective function of the hyperparameter optimization includes only the Lp-norm-based model evaluation measure, the autoencoder learns to also reconstruct the defects 23 contained in the imaging dataset 12. Thus, the defects 23 contained in the input image are at least partially reconstructed, and the difference image shows only small or no deviation from 0 at the defect locations, as shown in Figure 4.

[0110] In Figure 4, the results of the anomaly detection method obtained by a general hyperparameter optimization technique are shown. An input image 30 is reconstructed by an autoencoder machine learning method 16, which is p It was obtained by a standard hyperparameter optimization method based on an objective function consisting only of norm measures, which results in a comparison data set in the form of a reconstructed image 32 that includes all defects 23, so that an anomaly detection image 28 in the form of a difference image does not indicate as anomalies 15 any of the defects 23 present in the imaging data set 12.

[0111] 5 is a flowchart of an anomaly detection method 10′ according to an example of the first or second embodiment of the present invention. The method comprises a hyperparameter optimization unit 43 that performs a training data generation step 21 for generating training data from the imaging dataset 12, a hyperparameter value selection step 25 for selecting hyperparameter values, a training step 27 for training machine learning models 16 based on the selected hyperparameter values, a model evaluation step 29 for evaluating the trained machine learning models according to an objective function 46, and a model selection step 31 for selecting one of the trained machine learning models 16 based on an associated objective function value.

[0112] The machine learning model for anomaly detection trained according to the first or second embodiment of the present invention is based on the die-to-die principle or the die-to-database principle. In either case, training the machine learning model 16, e.g., an autoencoder or principal component analysis, requires abundant, preferably defect-free, data 34.

[0113] The hyperparameter optimization unit 43 requires an objective function 46 that is used to evaluate each of the trained machine learning models 16. The objective function 46 includes one or more model evaluation measures. In one example of the first or second embodiment of the present invention, the objective function includes at least two different model evaluation measures 46 to prevent the trained machine learning model from reconstructing not only the defects 23 but also the anomalies 15.

[0114] L p Apart from the norm loss function, the preferred model evaluation measure is the discriminative loss function L CE, which also requires a small number of expert annotations 36. The expert annotations 36 may be provided, for example, as pixel-by-pixel annotations or as bounding boxes. These expert annotations 36 may be facilitated using the expert's knowledge 38, for example, based on a critical distance, critical dimension, or pitch size associated with the smallest size of a structure on the wafer. In this way, the expert need only click on the center of a defect 23 in the anomaly detection image or imaging dataset 12, and a large area around the click point corresponding to the critical distance, critical dimension, or pitch size is automatically assigned as the defect 23.

[0115] The hyperparameter optimization unit 43 includes a sampler 42 and a pruner 44. The sampler 42 is used to execute the sampling strategy by selecting hyperparameter values ​​for hyperparameters 48 from a set of associated hyperparameter values. In one example, expert knowledge 38 is applied by introducing hyperparameter ranges 40 for the hyperparameters. From these ranges, hyperparameter values ​​are selected by the sampler 42. Additionally or alternatively, the expert knowledge 38 can be applied by introducing probability distributions indicating the likelihood of each hyperparameter value being selected by the sampling strategy based on imaging hardware settings or design knowledge, such as critical distance, critical dimension, and / or pitch size. Each hyperparameter value corresponds to at least one hyperparameter 48 that defines a machine learning model for anomaly 15 detection. The hyperparameter 48 may refer, for example, to the architecture of the neural network underlying the machine learning model (e.g., the number and size of layers, or the size of the bottleneck of an autoencoder), or the learning rate. The optional pruner 44 determines whether training of the machine learning model 16 should continue by selecting different hyperparameter values ​​according to a sampling strategy, or whether training should be interrupted, for example, by an early stopping criterion. In the case of interruption, new hyperparameter values ​​may be selected randomly, if possible, with respect to a given hyperparameter range and / or probability distribution for hyperparameter values. Training then begins again according to the sampling strategy.

[0116] The machine learning model 16 is trained on a subsampled dataset 50 sampled from the full imaging dataset 52. The size of the subsampled dataset 50 increases with the number of iterations performed by the hyperparameter optimization unit 43. In this way, a cold start on a small subsampled dataset 50 can be achieved with little effort and time for the user. In addition, new training data (e.g., training data containing new types of defects) can be easily incorporated into the training process, enabling online learning and optimization. This makes the training process quick and easy to adapt. After selecting the trained machine learning model that achieves the best objective function value, the optimized machine learning model 54 is finally trained on the full imaging dataset 52 to obtain a final machine learning model 56 for anomaly detection.

[0117] 6 shows the results of an anomaly detection method in the first or second embodiment of the present invention, where at least two different model evaluation measures are combined in an objective function 46 to obtain improved results. In addition to the Lp norm (e.g., the L2 norm, which penalizes deviations of the reconstructed image 32' from the input image 30'), the objective function 46 includes a discriminative loss function L, which is a defect / non-defect based on a small number of pixel-wise annotations provided by an expert. CE The expert indicates the locations of defects 23 that should not be reconstructed by the autoencoder. Thus, the anomaly detection image 28' (difference image) can be compared with the binary labels indicated by the user by additionally using cross-entropy loss as an additional model evaluation measure in the objective function 46. The expert annotations 36 are not required to encompass all types of defects 23 that occur in the imaging dataset 12, and the number of annotations can be very small, for example, less than 10%, preferably less than 1%, of the training data samples. Since annotations can be provided via multiple formats (e.g., bounding boxes, click points, auxiliary process information, etc.), the discrimination loss (L CE) needs to be adapted accordingly. For example, the bounding box annotation is computed using an overlap measure, i.e., the intersection-over-union loss L IOU is related to.

[0118] In one example of the first or second embodiment of the present invention, the objective function 46 also includes a model metric that includes an Occam's razor penalty on complexity to avoid overfitting. This model metric is the total number of floating-point operations (L FLOP ) Alternatively, the number and / or size of layers of the trained neural network, and / or the number of connections between neurons, or other suitable hyperparameter values, can be used as a model evaluation metric to measure the complexity of the machine learning model. This complexity-based model evaluation metric is optional.

[0119] Thus, in one example of the first or second embodiment of the present invention, the objective function is based on three model evaluation measures, e.g. f=w1L2+w2L CE +w3L FLOP where the weights w1, w2, and w3 are chosen by experts. Typically, the first two weights are similar in magnitude, while the third weight is at least ten times lower, e.g., w1=10, w2=1, and w3=0.01. One or some of the weights, e.g., weight w3, can be set to 0. Instead of the L2 norm, other L p We can also use the norm. CE is an arbitrary loss function that penalizes the deviation of the difference image for anomalies in the imaging dataset from some ground truth data, e.g., the intersection-over-union loss L IOUcan be replaced by. In one example, the objective function also or alternatively includes a quality value as a model evaluation measure for evaluating the quality of the trained machine learning model 16, and the user interface 128 is configured to present information about the trained machine learning model 16 to the user and have the user indicate the quality value. Additional model evaluation measures may be added or used instead of one or more of the model evaluation measures described above. In addition, all options in the backbone Optuna library may be added.

[0120] Figure 6 shows an input image 30' containing the same defect 23 as Figure 4. In reconstructed image 32', defect 23 is not reconstructed because the objective function includes at least two model evaluation measures. While the background image is correctly reconstructed, the reconstruction does not replicate any of the defects 23 (open circles) as desired. Consequently, the difference image, as anomaly detection image 28', contains defect 23.

[0121] FIG. 7 illustrates a sampling strategy for a tree-structured Parzen estimator (TPE) that can be used to sample hyperparameter values ​​by sampler 42 .

[0122] The Parzen density estimator is the density of the observations x=x1,...,x n is a nonparametric kernel density estimator used to estimate the probability density function (hyperparameter values) of a random variable given a set of:

number

[0123] The Parzen estimator is conceptually similar to Bayesian optimization, but is the opposite in theory: while Bayesian optimization seeks to find p(y|x) (where y is the objective function value and x is a hyperparameter value), Parzen estimator trees model p(x|y) and p(y).

[0124] With respect to Bayesian optimization, the first step in TPE is to start sampling the objective function 46 by random search to initialize the algorithm.

[0125] The observations are then split into two groups: those that perform best according to the objective function 46 (good group 58, e.g., top quartile), and y as the split value for the two groups. * The remainder (poor group 60) defines:

[0126] The probability of being in each of these groups is y <y * If p(x|y)=l(x), y≧y * If so, it is modeled as p(x|y)=g(x) (the Gaussian process models the posterior probability).

[0127] Two densities, l and g, are modeled using the Parzen density estimator, where g denotes the density of the good group 62 and l denotes the density of the poor group 64.

[0128] p(y) defines the percentile split between two categories. <y * ) = δ (i.e., if g models the top quartile, δ = 0.75).

[0129] Using Bayes' law (i.e., p(x,y)=p(y)p(x|y)), the definition of expected improvement (EI)66 can be shown to be equivalent to l(x) / g(x).

[0130] The sampling strategy then selects the next point as the one that maximizes l(x) / g(x), as shown in Figure 7.

[0131] FIG. 8 illustrates the pruning strategy of the asynchronous successive halvings algorithm used by pruner 44 to determine whether to continue or stop model training in one example of the first or second embodiment of the present invention. The vertical axis shows the training loss function, and the horizontal axis shows training time. The lower the objective function value, the better the performance of the model defined by the associated hyperparameter value or values. Only the hyperparameter value or values ​​that achieve the lowest objective function value are continuously improved, with the remaining set being discarded in early iterations.

[0132] Asynchronous successive halving algorithm (ASHA) The ASHA algorithm, shown in Figure 8, is a pruning strategy that combines random search with normative early stopping in an asynchronous manner.

[0133] The successive halving algorithm (SHA) is a well-known multi-armed bandit algorithm for prescriptive early stopping. The successive halving algorithm starts with all candidate configurations in the base rung and proceeds as follows: 1. Uniformly assign a budget (value on the horizontal axis) to the set of candidate hyperparameter configurations at a given stage; 2. Evaluate the performance of all candidate configurations; 3. Promote the top 1 / η ​​of the candidate configurations to the next stage, where η is the elimination rate selected by the user; 4. In the next stage, double the budget per configuration and repeat until one configuration remains.

[0134] Higher η indicates a more aggressive elimination rate, where all configurations except the top 1 / η ​​are eliminated.

[0135] The SHA algorithm is difficult to parallelize because the algorithm takes a set of configurations as input and waits for all configurations in one stage to complete before promoting a configuration to the next stage.

[0136] To remove the bottleneck created by synchronous promotion, the asynchronous successive halving algorithm (ASHA) grows bottom-up, promoting as many configurations as possible, instead of starting with a broad set of configurations and narrowing them down.

[0137] ASHA starts by assigning a worker to add configurations to the bottom row. When a worker finishes one job and requests a new one, it checks the rows from top to bottom, checking whether the top 1 / η ​​of each row has a configuration that can be promoted to the next row. If not, the worker is assigned to add one configuration to the bottom row, widening the range so that more configurations can be promoted.

[0138] The output of the ASHA algorithm is shown in Figure 9, which summarizes the results of all trials. Here, 17 trials were performed for hyperparameter optimization. The worst trials, 68, which achieved the worst objective function values, were dropped in the first iteration. In each subsequent iteration, the best model from the previous iteration was seeded and trained for longer. Finally, the best trials, 70, were retrained on the entire original training dataset.

[0139] FIG. 10 shows a flowchart of a computer-implemented method 10″ for anomaly detection in an imaging dataset 12 of a wafer 120, the imaging dataset 12 including defects 23 belonging to multiple defect classes according to the third or fourth embodiment of the present invention, the method comprising the following steps: in an anomaly-detection image generation step 61, generating an anomaly-detection image 72 by applying an anomaly detection method to the imaging dataset 12; performing one or more iterations 73, at least one of which comprises the following steps: in a sampling step 63, sampling one or more of the distributions of values ​​of the anomaly-detection image for each defect class of a subset of defect classes; providing a sample or samples; in a calibration step 65, calibrating the anomaly detection image 72 by at least one calibration method including the following steps: in a training step 67, training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​in the anomaly detection image; in an application step 69, applying the trained machine learning model to the anomaly detection image 72 to obtain a calibrated anomaly detection image; in a thresholding step 71, applying a threshold to the calibrated anomaly detection image to detect anomalies 15, thereby reducing nuisances and highlighting defects 23 in the anomaly detection image 72.

[0140] 11 shows a computer-implemented method 10''' for anomaly detection in an imaging dataset 12 of a wafer 120 according to an example of the third or fourth embodiment of the present invention. The anomaly-detection image 72 is calibrated in a calibration step 74 in each iteration 88. In the calibration step 74, a calibration method is selected from a set of calibration methods.

[0141] The set of calibration methods includes a first calibration method for training a machine learning model for anomaly localization based on one or more samples of the distribution of values ​​of the anomaly detection images for all defect classes of the subset of defect classes. The trained machine learning model is then applied to the anomaly detection images 72. Finally, a threshold is applied to the calibrated anomaly detection images to obtain anomalies 15 or defects 23.

[0142] The set of calibration methods also includes a second calibration method based on global or local thresholding and / or filtering. For this purpose, one or more lower and / or upper thresholds are selected for the anomaly-detected image 72, and the anomaly-detected image 72 is adapted based on these thresholds. The filtering may include, for example, morphological cleaning, cluster size filtering, size filtering, etc. The size filter may indicate, for example, minimum and maximum sizes for each defect type, so that only anomalies or defects exhibiting sizes within the indicated size range are detected. The size of an anomaly or defect may be measured by the number of connected pixels, the length of the anomaly or defect in a particular direction, its diameter, etc. The calibration method may combine thresholding and filtering. The user interface may be configured to allow the user to select the thresholds and / or filters. Finally, the thresholds are applied to the calibrated anomaly-detected image to obtain the anomalies 15 or defects 23.

[0143] In an evaluation step 76 it is checked whether all (or a sufficiently large number) of defects 23 have been detected. If the answer is affirmative 77, the iteration 88 is ended in an end step 78. Otherwise, if the answer is negative 79, the calibration method is adapted in an adapt calibration method step 80.

[0144] To adapt the first calibration method, annotations are added or adapted in an annotation adjustment step 84 by expert annotation, for example, by clicking on defects and applying expert knowledge such as critical dimensions and / or critical distances and / or pitch sizes associated with minimum feature sizes. To adapt the second calibration method, thresholds and / or filters are added or adapted in a threshold or filter adjustment step 82, for example, by user input. The calibration method can be adapted based on additional information, for example, a machine learning model for anomaly localization is retrained in a training step 86. In each iteration 88, a different calibration method can be selected and applied to the calibrated anomaly-detected images from the previous iteration.

[0145] In one example of the third or fourth embodiment of the present invention, the second calibration method includes the following steps: for each defect class in the subset of defect classes, calculating an intermediate calibrated anomaly detection image by adapting the values ​​of the anomaly detection image based on one or more selected thresholds of the values ​​of the anomaly detection image for the current defect class; applying one or more filters, such as a size filter, for the current defect class so that only anomalies 15 within a certain size range are retained by the calibration method; and finally, generating a calibrated anomaly detection image by applying an operator to all the intermediate calibrated anomaly detection images, where the operator is selected from the group including pixel-wise sum, pixel-wise average, pixel-wise minimum, pixel-wise maximum, and pixel-wise scaling. In this way, the final calibrated anomaly detection image includes defects 23 from different defect classes, each of which was extracted from the anomaly detection image 72 based on a different set of thresholds and filters. Using the maximum operator preserves as many anomalies 15 as possible, while the minimum operator further reduces noise and nuisance.

[0146] The first calibration method is used to normalize the value a of the anomaly detection image based on an upper threshold u and a lower threshold l, i.e.,

number

[0147] Alternatively, the adaptation of the anomaly detection image values ​​may involve clipping the values ​​to the range [l,u]. a new =min(max(a,l),u)

[0148] In one example of the third or fourth embodiment of the present invention, the calibration method is formulated as a pixel-wise segmentation problem, as shown in FIG. 12 . The input image 90 to the anomaly detection method contains two defect classes 23, called hollow diamonds and triangles. The output of the anomaly detection method is an anomaly-detection image 92, which is a difference image between the input image 90 and a comparison dataset (here, the output of an autoencoder that reconstructs the input image 90). The mottled background is due to noise and high reconstruction error near the defects 23. A simple threshold that ensures high defect recall would result in a high nuisance rate. Therefore, the anomaly-detection image 92 is calibrated using a machine learning model for anomaly localization. To this end, the user interface is configured to allow the user to provide expert annotations 96 for several samples of a subset of defect classes. To facilitate the annotation process, the user provides click points 94 that are automatically processed into pixel-level annotations 96 by utilizing available meta-information (e.g., critical distances and / or critical dimensions and / or pitch sizes). Based on this meta-information, the click points are expanded to cover the area covering the smallest feature size on the wafer. In the partially annotated anomaly detection image 95, annotated pixels are set to 1 and unannotated pixels are set to 0. It is sufficient for the user to provide only a small number of click points (5–10, less than 20) for a subset of defect classes. Even if the remaining defects 23 are not part of the training data, they can still be segmented by the machine learning model for anomaly segmentation, assuming that the annotated classes cover the appearance range of all defect classes. Based on the annotations, the machine learning model for anomaly segmentation is trained. In this way, new defects 23 can be detected based on limited training data, and the user effort required for the annotation process is minimized, minimizing user effort and enabling a cold start.

[0149] According to an example of the third or fourth embodiment of the present invention, during training, a machine learning model for anomaly segmentation considers as input partially annotated anomaly detection images, such as difference images. As output, the model labels each pixel as a target anomaly or a nuisance. Since not all input pixels are annotated by the user, model training is based on the following semi-supervised loss function:

[0150]

number

[0151]

number

number

number

[0152] FIG. 13 shows a qualitative analysis of a method according to the third or fourth embodiment of the present invention. In the top row, the reconstructed image 102 differs from the input image 104 in regions corresponding to defects 23. The bottom row shows an uncalibrated anomaly-detected image 106 before applying the calibration method. This uncalibrated image 106 exhibits high recall (all defects 23 are highlighted) but low precision (a lot of reflective highlighting corresponding to noise), i.e., a lot of false positives. Furthermore, the intensity of highlighting varies for each defect class. The right column shows a calibrated anomaly-detected image 108 after applying the calibration method. The calibrated anomaly-detected image 108 exhibits both high recall and precision, with all defect classes exhibiting a uniform intensity of 1.0. These can be easily and automatically detected based on a single uniform threshold of 0.5. Therefore, applying the calibration method maintains high recall, significantly improves precision, and ensures uniform intensity values ​​for defects 23, so applying a single threshold to the calibrated anomaly-detected image 108 is sufficient for anomaly detection.

[0153] The benefit of applying the calibration method according to the third or fourth embodiment of the present invention is quantified in Figure 14, which shows the confusion matrix before and after calibration 110 and 112, i.e., before and after segmenting the anomaly-detected image 108 and applying a threshold of 0.5. In this way, high recall (0.93 vs. 1.0) is maintained, a significant reduction in false positives (from 0.61 to 0.25) is achieved, and precision is significantly improved.

[0154] 15 schematically illustrates a system 114 that can be used to control the quality of wafers 120 produced in a semiconductor foundry. System 114 includes an imaging device 116 and a processing device 118. Imaging device 116 is coupled to processing device 118. Imaging device 116 is configured to acquire an imaging dataset 12 of wafer 120. Wafer 120 may include semiconductor structures, e.g., transistors such as field effect transistors, memory cells, etc. Exemplary embodiments of imaging device 116 are an SEM or multi-beam SEM, a helium ion microscope (HIM) or a cross-beam device including a FIB and SEM, or any charged particle imaging device.

[0155] The imaging device 116 can provide the imaging dataset 12 to a processing device 118. The processing device 118 includes a processor, for example, implemented as a CPU 122 or a GPU. The processor can receive the imaging dataset 12 via an interface 124. The processor can load program code from a memory 126. The processor can execute the program code. Executing the program code causes the processor to perform techniques such as those described herein, such as hyperparameter optimization, training an anomaly detection method, performing the anomaly detection method to detect one or more anomalies 15 in the imaging dataset 12 of the wafer 120, calibrating the anomaly-detection image based on a sample of the value distribution of the anomaly-detection image, etc. For example, upon loading the program code from the memory 126, the processor can execute the computer-implemented method shown in FIG. 3, FIG. 6, or FIG. 10, respectively. Optionally, the processing device can include a user interface 128 for inputting user input, such as a click point, a bounding box, or a characteristic of the value distribution of the anomaly-detection image.

[0156] FIG. 16 schematically illustrates a system 114′ that can be used to control the production of wafers 120 in a semiconductor foundry. The system 114′ includes the same components as those shown in FIG. 14 , and the foregoing applies to each of the components in this figure. Additionally, the system 114′ includes a means 130 for producing wafers 120 controlled by at least one wafer manufacturing process parameter. To this end, the imaging data set 12 is provided to a processing device 118 by means of an imaging device 116. A processor of the processing device 118 is configured to perform one of the disclosed methods, which includes controlling at least one wafer manufacturing process parameter based on one or more measured properties of detected anomalies 15 in the imaging data set 12 of the wafer 120. For example, a detected anomaly 15 due to a bridge defect indicates insufficient etching, thereby increasing the etching amount; a detected anomaly 15 due to an open line indicates excessive etching, thereby decreasing the etching amount; consistently occurring anomalies 15 indicate a mask defect, thereby requiring the mask to be checked; and an anomaly 15 due to a missing structure indicates that the material deposition is not ideal, thereby modifying the material deposition.

[0157] The following clauses include preferred embodiments of the present invention:

[0158] 1a. A computer-implemented method for anomaly detection, comprising: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - repeating the following steps: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection; ii. training a machine learning model defined by hyperparameters based on the generated subset of training data; iii. Evaluating the trained machine learning model by calculating the associated objective function value of the objective function. - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; 20. A computer-implemented method comprising:

[0159] 1b. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes; - generating training data from the imaging dataset; - repeating the following steps: i. selecting a plurality of hyperparameter values ​​from a set of associated hyperparameter values ​​based on a sampling strategy, each hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection; ii. training a machine learning model for anomaly detection based on the generated subset of training data; iii. evaluating the trained machine learning model according to an objective function including at least two different model evaluation measures; - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging dataset to detect anomalies, thereby providing an anomaly-detected image; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of the subset of defect classes; ii. calibrating an anomaly detection image based on a sample of said distribution by a calibration method selected from a group of calibration methods, thereby reducing nuisance and highlighting defects in the anomaly detection image, wherein at least one of the selected calibration methods includes the steps of: c. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly-detected image for all defect classes of the subset of defect classes; d. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a single threshold to the calibrated anomaly detection image to detect anomalies; 20. A computer-implemented method comprising:

[0160] 1c. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method comprising: - providing an anomaly-detected image generated by applying an anomaly detection model to an imaging dataset; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of the subset of defect classes; ii. calibrating an anomaly detection image based on a sample of said distribution by a calibration method selected from a group of calibration methods, thereby reducing nuisance and highlighting defects in the anomaly detection image, wherein at least one of the selected calibration methods includes the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly detection image for all defect classes of the subset of defect classes; b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a single threshold to the calibrated anomaly detection image to detect anomalies; 20. A computer-implemented method comprising:

[0161] 1. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, comprising: - generating training data from the imaging dataset; - repeating the following steps: i. selecting a plurality of hyperparameter values ​​from a set of associated hyperparameter values ​​based on a sampling strategy, each hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection; ii. training a machine learning model for anomaly detection based on the generated subset of training data; iii. evaluating the trained machine learning model according to an objective function including at least two different model evaluation measures; - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; 20. A computer-implemented method comprising:

[0162] 2. The method of any one of the preceding clauses, wherein a machine learning model is trained to, when presented with a subset of an imaging dataset as input, calculate an anomaly-free reconstruction of the subset, and anomalies in the subset are detected based on a comparison between the subset and the reconstructed subset.

[0163] 3. The method of clause 2, wherein the machine learning model comprises an autoencoder.

[0164] 4. The method of any one of the preceding clauses, wherein at least one hyperparameter is related to the design of the machine learning model.

[0165] 5. The method of any one of the preceding clauses, wherein one of the model evaluation measures includes a loss function used during training of the machine learning model.

[0166] 6. The method of any one of the preceding clauses, wherein one of the model evaluation measures comprises an Lp-norm loss function, where p≧1.

[0167] 7. The method of any one of the preceding clauses, wherein the training data includes expert annotations of anomalies for a subset of the imaging dataset, and one of the model evaluation measures includes a discriminative loss function that assesses the difference between the expert-annotated anomalies and the detected anomalies.

[0168] 8. The method of clause 7, wherein the expert annotations account for less than 10%, preferably less than 1%, of the training data.

[0169] 9. The method of clause 7 or 8, wherein the expert annotations include only a subset of the anomalies present in the imaging dataset.

[0170] 10. The method of any one of the preceding clauses, wherein one of the model evaluation measures includes a measure of the complexity of the machine learning model.

[0171] 11. The method of clause 10, wherein the measure of complexity of the machine learning model includes a total number of floating-point operations in one forward propagation of the machine learning model.

[0172] 12. The method of any one of the preceding clauses, wherein the objective function comprises a weighted sum of at least two model evaluation measures.

[0173] 13. The method of clause 12, wherein the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1), a discriminative loss function that evaluates the difference between the expert-annotated anomalies and the detected anomalies, and a measure of the complexity of the machine learning model.

[0174] 14. The method of any one of the preceding clauses, wherein the sampling strategy for selecting the plurality of hyperparameter values ​​includes taking into account hyperparameter values ​​and corresponding values ​​of the objective function from one or more prior iterations.

[0175] 15. The method of clause 14, wherein the sampling strategy selects hyperparameter values ​​by optimizing one criterion selected from the group including expected improvement, maximum probability of improvement, and upper confidence interval.

[0176] 16. The method of any one of the preceding clauses, wherein the sampling strategy includes early stopping criteria.

[0177] 17. The method of any one of the preceding clauses, wherein the sampling strategy comprises randomly selecting hyperparameter values ​​from the relevant set of hyperparameter values.

[0178] 18. The method of any one of the preceding clauses, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy.

[0179] 18a. The method of the preceding clause, wherein the probability distribution is modeled based on knowledge from previous applications, preferably based on design knowledge such as imaging hardware settings or critical distances.

[0180] 19. The method of any one of the preceding clauses, wherein the sampling strategy is different in at least two replicates.

[0181] 20. The method of any one of the preceding clauses, wherein the size of the generated subset of training data varies based on a sampling strategy.

[0182] 21. The method of clause 20, wherein the size of the generated subset of training data increases with the number of iterations.

[0183] 22. The method of any one of the preceding clauses, wherein prior knowledge based on a particular application is used to select initial hyperparameter values.

[0184] 23. The sampling strategy is - in the first iteration: selecting initial hyperparameter values ​​from a set of associated hyperparameter values; - in a subsequent iteration: selecting, based on the hyperparameter values ​​and corresponding values ​​of the objective function from one or more previous iterations, hyperparameter values ​​from the associated set of hyperparameter values ​​that maximizes a criterion of expected improvement; - if an early stopping criterion is met in a certain iteration: selecting initial hyperparameter values ​​from a set of relevant hyperparameter values; 10. The method of any one of the preceding clauses, including:

[0185] 24. The method of any one of the preceding clauses, wherein the iterations are performed in parallel by multiple threads.

[0186] 25. The method of any one of the preceding clauses, wherein the selected machine learning model is trained on a full imaging dataset of the wafer before being applied to the imaging dataset of the wafer to detect anomalies.

[0187] 26. The method according to any one of the preceding clauses, wherein the trained machine learning model that achieves the lowest objective function value is selected.

[0188] 27. The method according to any one of the preceding clauses, wherein the trained machine learning model that achieves the highest objective function value is selected.

[0189] 28. The method of any one of the preceding clauses, wherein the sampling strategy criteria implements at least one component selected from the group consisting of an exploratory scheme and an exploitative scheme.

[0190] 28a. A method according to any one of the preceding clauses, wherein the objective function includes a quality value for evaluating the quality of the trained machine learning model, and the user interface is configured to present information about the trained machine learning model to a user and allow the user to indicate the quality value.

[0191] 29. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method comprising: - providing an anomaly-detected image generated by applying an anomaly detection model to an imaging dataset; - performing one or more iterations, the iterations including the steps of: i. providing one or more characteristics of a distribution of values ​​of the anomaly detection image for each defect class of the subset of defect classes; ii. calibrating the anomaly detection image based on characteristics of the distribution by a calibration method selected from a set of calibration methods, thereby reducing nuisances and highlighting defects in the anomaly detection image. 20. A computer-implemented method comprising:

[0192] 30. The method of clause 29, 1a, 1b or 1c, wherein the anomaly detection image is a difference image between the imaging data set and the comparison data set.

[0193] 31. The method according to clause 30, wherein the comparative dataset is based on the die-to-die principle or the die-to-database principle.

[0194] 32. The method of clause 30 or 31, wherein the comparative dataset is generated by a machine learning model.

[0195] 33. The method of clause 32, wherein the comparison dataset includes a reconstructed representation of the imaging dataset generated by training an autoencoder on the imaging dataset or a subset thereof and applying the trained autoencoder to the imaging dataset to obtain a reconstructed representation.

[0196] 34. The method of clause 29, 1a, 1b or 1c, wherein the anomaly detection image is generated by a machine learning model.

[0197] 35. The method of clause 34, wherein anomaly detection and calibration of the anomaly detection image are jointly learned by a machine learning model applied to the imaging dataset to directly obtain calibrated anomaly detection images.

[0198] 36. The method of any one of clauses 29-35, 1a, 1b or 1c, wherein the one or more characteristics comprise a sample from said distribution.

[0199] 37. The method of any one of clauses 29-36, 1a, 1b or 1c, wherein the one or more characteristics comprise the top quartile and / or bottom quartile of said distribution.

[0200] 38. The method of any one of clauses 29-37, 1a, 1b or 1c, wherein the one or more characteristics include moments of said distribution.

[0201] 39. The method of any one of clauses 29-36, 1a, 1b or 1c, wherein the one or more characteristics include a minimum and / or a maximum of a domain of said distribution.

[0202] 40. The method of any one of clauses 36 to 39, further providing a filter for each defect class of the subset of defect classes.

[0203] 41. The method of clause 40, wherein the filter comprises a size filter.

[0204] 42. The method of clause 40 or 41, wherein the set of calibration methods comprises a calibration method for calibrating the anomaly detection image by applying the following steps: - calculating, for each defect class of the subset of defect classes, an intermediate calibrated image obtained by: i. adapting the values ​​of the anomaly detection image based on one or more characteristics of the distribution of values ​​of the anomaly detection image for the current defect class; ii. applying one or more filters for the current defect class; - generating a calibrated anomaly detection image by applying an operator to all intermediate calibrated images, the operator being selected from the group comprising pixel-wise sum, pixel-wise mean, pixel-wise minimum, pixel-wise maximum, pixel-wise scaling.

[0205] 43. The method of clause 42, wherein the values ​​of the anomaly detection image are adapted by normalization.

[0206] 44. The method of clause 42, wherein the values ​​of the anomaly detection image are adapted by clipping.

[0207] 45. The method of clauses 29-44, 1a, 1b or 1c, wherein the set of calibration methods comprises a calibration method for calibrating the anomaly detection image by applying the following steps: - training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more characteristics of the distribution of values ​​of the anomaly detection image for all defect classes of the subset of defect classes; - applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image.

[0208] 46. ​​The method of clause 45, wherein the machine learning model is trained to optimize a loss function for anomaly localization based on foreground samples and background samples.

[0209] 47. The method of clause 46, wherein the loss function is a weighted cross-entropy loss function.

[0210] 48. The method of clause 46, wherein the loss function is a Kullback-Leibler divergence loss function, or an L1 loss function or an L2 loss function.

[0211] 49. The method of any one of clauses 46 to 48, wherein the loss function is a semi-supervised loss function.

[0212] 50. The method of any one of clauses 45 to 49, wherein a machine learning model for anomaly localization is trained on partially labeled anomaly detection images derived from properties of the distribution.

[0213] 51. The method of any one of clauses 45 to 50, wherein one or more characteristics include a sample from said distribution being used as a foreground sample or a background sample.

[0214] 52. The method of clause 51, wherein the sample is used as a foreground sample and background samples are selected from the remaining pixels of the anomaly detection image.

[0215] 53. The method of clause 52, wherein the remaining pixels of the anomaly-detected image are selected as background samples if the value of the anomaly-detected image is below a threshold.

[0216] 54. The method according to clause 52 or 53, wherein each background sample is weighted by a weighting function w of its anomaly detection image value a, in particular by a negative exponential weighting function w(a)=exp(-a).

[0217] 55. The method of any one of clauses 45 to 54, wherein the machine learning model has been pre-trained with training data from a similar application.

[0218] 56. The method of any one of clauses 45 to 55, wherein the machine learning model addresses class imbalance by using focal loss functions and / or defect rate priors.

[0219] 57. The method of any one of clauses 29-56, 1a, 1b or 1c, further comprising presenting the anomaly detection image to a user via a user interface, the user interface configured to allow the user to input information regarding each defect class of the subset of defect classes, from which one or more characteristics of the distribution of values ​​in the anomaly detection image for each defect class are derived.

[0220] 58. The method of clause 57, wherein the user interface is configured to allow a user to indicate in the anomaly detection image the location of a small number of defects of each class of a subset of defect classes, and one or more characteristics of the distribution of anomaly detection image values ​​for each of these classes are prepared by sampling anomaly detection image values ​​from the defects indicated by the user for this class.

[0221] 59. The method of clause 58, wherein the user indicates the location of the defect by selecting a single pixel, preferably a pixel in the central region, of said defect.

[0222] 60. The method of clause 59, wherein the defect annotation is automatically generated by selecting an area, preferably a circle, surrounding a selected single pixel.

[0223] 61. The method of clause 60, wherein the anomaly detection image value is sampled from an area surrounding the selected pixel.

[0224] 62. The method of clause 60 or 61, wherein the regions are selected based on application-specific knowledge, in particular from the field of wafer manufacturing.

[0225] 63. The method of clause 62, wherein the size of the region is selected based on a critical dimension.

[0226] 64. The method of clause 62 or 63, wherein the size of the region is selected based on the pitch size.

[0227] 65. The method of any one of clauses 29-64, 1a, 1b or 1c, wherein the calibrated anomaly detection image is post-processed to further reduce nuisance.

[0228] 66. The anomaly detection method of any one of clauses 29-65, 1a, 1b or 1c, wherein one or more characteristics of the distribution of values ​​of the anomaly detection image for each defect class of the subset of defect classes are prepared based on similar applications.

[0229] 67. The method of any one of clauses 29-66, 1a, 1b or 1c, further generating an uncertainty estimate for each anomaly in the calibrated anomaly detection image.

[0230] 68. The method according to any one of clauses 29 to 67, 1a, 1b or 1c, further comprising a final step after performing one or more iterations. - thresholding the calibrated difference image with a single threshold to obtain a binary anomaly detection image.

[0231] 69. The method of clauses 29-68, 1a, 1b or 1c, wherein the subset of defect classes includes less than 50% of the defect classes, preferably less than 30% of the defect classes, more preferably less than 20% of the defect classes, and most preferably less than 10% of the defect classes.

[0232] 70. The method of any one of clauses 29-69, 1a, 1b or 1c, wherein multiple replicates are performed and the calibration method is different for at least two of the multiple replicates.

[0233] 70a. The method of any one of the preceding clauses, wherein the imaging dataset includes one or more multisensory images.

[0234] 70b. ​​The method of any one of the preceding clauses, wherein the imaging dataset comprises a two-dimensional image, a three-dimensional image, a slice-by-slice three-dimensional image, or a multi-sensory fusion image.

[0235] 71. A method according to any one of the preceding clauses, wherein the imaging data set is a multi-beam SEM image.

[0236] 72. A method according to any one of the preceding clauses, wherein the imaging data set is a focused ion beam SEM image.

[0237] 73. The method of any one of the preceding clauses, further comprising measuring one or more properties of the detected anomaly.

[0238] 74. The method of clause 73, further comprising controlling at least one wafer manufacturing process parameter based on the one or more measured properties.

[0239] 75. The method of clause 73, further comprising assessing the quality of the wafer based on the one or more measured properties and at least one quality assessment rule.

[0240] 76. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the method described in any one of clauses 1 to 75.

[0241] 77. A system for controlling the quality of wafers produced in a semiconductor manufacturing facility, comprising: - an imaging device adapted to provide an imaging data set of said wafer; - one or more processing devices; - one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods of clause 75.

[0242] 78. A system for controlling wafer production in a semiconductor manufacturing facility, comprising: - means for producing wafers controlled by at least one manufacturing process parameter; - an imaging device adapted to provide an imaging data set of the wafer; - one or more processing devices; - one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods of clause 74; Including, the system.

[0243] 79. A computer-implemented method for anomaly detection, comprising: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - repeating the following steps: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for anomaly detection; ii. training a machine learning model defined by hyperparameters based on the generated subset of training data; iii. Evaluating the trained machine learning model by calculating the associated objective function value of the objective function. - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; 20. A computer-implemented method comprising:

[0244] 80. The method of clause 79, wherein a machine learning model, when presented with a subset of an imaging dataset as input, is trained to compute an anomaly-free reconstruction of the subset, and anomalies within the subset are detected based on a comparison between the subset and the reconstructed subset.

[0245] 81. The method of clause 80, wherein the machine learning model includes an autoencoder.

[0246] 82. The method of any one of clauses 79 to 81, wherein at least one hyperparameter is related to the design of the machine learning model.

[0247] 83. The method of any one of clauses 79 to 82, wherein the objective function includes at least two different model evaluation measures.

[0248] 84. The method of clause 83, wherein one of the model evaluation measures includes a loss function used during training of the machine learning model.

[0249] 85. The method of clause 83 or 84, wherein one of the model evaluation measures includes an Lp norm loss function (p≧1).

[0250] 86. The method of any one of clauses 83 to 85, wherein the training data includes expert annotations of anomalies for a subset of the imaging dataset, and one of the model evaluation measures includes a discriminative loss function that evaluates the difference between the expert annotations of anomalies and the detected anomalies.

[0251] 87. The method of clause 86, wherein the expert annotations include only a subset of the anomalies present in the imaging dataset.

[0252] 88. The method of any one of clauses 83 to 87, wherein one of the model evaluation measures includes a measure of the complexity of the machine learning model.

[0253] 89. The method of clause 88, wherein the measure of complexity of the machine learning model takes into account the total number of floating-point operations in one forward propagation of the machine learning model.

[0254] 90. The method of any one of clauses 79 to 89, wherein the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1), a discriminative loss function that evaluates the difference between the expert annotations of the anomalies and the detected anomalies, and a measure of the complexity of the machine learning model.

[0255] 91. A method according to any one of clauses 79 to 90, wherein the objective function includes a quality value for evaluating the quality of the trained machine learning model, and the user interface is configured to present information about the trained machine learning model to a user and to allow the user to indicate the quality value.

[0256] 92. The method of any one of clauses 79 to 91, wherein the sampling strategy for selecting the plurality of hyperparameter values ​​comprises taking into account the hyperparameter values ​​and corresponding values ​​of the objective function from one or more prior iterations by optimizing one criterion selected from the group including expected improvement, maximum probability of improvement, and upper confidence interval.

[0257] 93. The method of any one of clauses 79 to 92, wherein the sampling strategy includes early stopping criteria.

[0258] 94. The method of any one of clauses 79 to 93, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy.

[0259] 95. The method of clause 94, wherein the probability distribution is modeled based on knowledge from previous applications, preferably based on imaging hardware settings or design knowledge such as critical distances, critical dimensions, or pitch sizes.

[0260] 96. The method of any one of clauses 79 to 95, wherein the sampling strategy is different in at least two replicates.

[0261] 97. The method of clauses 79-96, wherein the size of the generated subset of training data increases with the number of iterations.

[0262] 98. A computer-implemented method for detecting anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method comprising: - generating an anomaly-detected image by applying an anomaly detection method to the imaging dataset; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of the subset of defect classes; ii. calibrating the anomaly detection image by at least one calibration method comprising the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly-detected image; b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing nuisance and highlighting defects in the anomaly detection image; 20. A computer-implemented method comprising:

[0263] 99. The method according to clause 98, wherein the anomaly detection image is a difference image between an imaging data set and a comparison data set, and the comparison data set is based on the die-to-die principle or the die-to-database principle.

[0264] 100. The method of clause 99, wherein the comparison dataset includes a reconstructed representation of the imaging dataset generated by training a machine learning autoencoder on the imaging dataset or a subset thereof and applying the trained autoencoder to the imaging dataset to obtain a reconstructed representation.

[0265] 101. The method of any one of clauses 98 to 100, wherein a machine learning model for anomaly localization is trained to optimize a loss function based on foreground samples and background samples.

[0266] 102. The method of clause 101, wherein the loss function is a semi-supervised loss function.

[0267] 103. The method of clause 101 or 102, wherein a sample from said distribution of anomaly detection image values ​​for each defect class of a subset of defect classes is used as a foreground sample or a background sample.

[0268] 104. The method of clause 103, wherein the sample is used as a foreground sample and background samples are selected from the remaining pixels of the anomaly detection image.

[0269] 105. The method of clause 104, wherein the remaining pixels of the anomaly-detected image are selected as background samples if the value of the anomaly-detected image is below a threshold.

[0270] 106. The method according to clause 104 or 105, wherein each background sample is weighted by a weighting function w of its anomaly detection image value a, in particular by a negative exponential weighting function w(a)=exp(-a).

[0271] 107. A method according to any one of clauses 98 to 106, wherein the user interface is configured to allow the user to indicate the location of a small number of defects of each class of a subset of defect classes in the anomaly detection image, the user indicates the location of the defect by selecting a single pixel, preferably a pixel in a central region, of the defect, an annotation of the defect is automatically generated by selecting an area, preferably a circle, surrounding the selected single pixel, and values ​​of the anomaly detection image are sampled from the area surrounding the selected pixel.

[0272] 108. The method of clause 107, wherein the size of the region is selected based on a critical dimension and / or a critical distance and / or a pitch size.

[0273] 109. The method of any one of clauses 79 to 108, wherein the imaging dataset includes one or more multisensory images.

[0274] 110. The method of any one of clauses 79 to 109, wherein the imaging data set is a multi-beam SEM image.

[0275] 111. The method of any one of clauses 79 to 110, wherein the imaging data set is a focused ion beam SEM image.

[0276] 112. The method of any one of clauses 79 to 111, further comprising measuring one or more properties of the detected anomaly.

[0277] 113. The method of clause 112, further comprising controlling at least one wafer fabrication process parameter based on the one or more measured properties.

[0278] 114. The method of clause 112, further comprising assessing the quality of the wafer based on the one or more measured properties and at least one quality assessment rule.

[0279] 115. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the method described in any one of clauses 79 to 114.

[0280] 116. A system for controlling the quality of wafers produced in a semiconductor manufacturing facility, comprising: - an imaging device adapted to provide an imaging data set of said wafer; - one or more processing devices; - One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the method of clause 114.

[0281] 117. A system for controlling wafer production in a semiconductor manufacturing facility, comprising: - means for producing wafers controlled by at least one manufacturing process parameter; - an imaging device adapted to provide an imaging data set of the wafer; - one or more processing devices; - one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods of clause 113; and Including, the system.

[0282] In summary, the following preferred features of the present invention are, inter alia: the present invention relates to a computer-implemented method 10, 10′ for detecting anomalies 15, comprising the steps of selecting an imaging dataset 12 of a wafer 120 and hyperparameter values ​​defining a machine learning model 16 for anomaly detection, training and evaluating the machine learning model 16 by calculating objective function values, and selecting and applying one of the trained machine learning models to detect anomalies 15. The present invention also relates to a computer-implemented method 10″ for detecting anomalies 15 in an imaging dataset 12 of a wafer 120, comprising the steps of providing a sample of a distribution of values ​​of the anomaly-detection images for each defect class, calibrating the anomaly-detection images 20, 26, 26′ by training a machine learning model for anomaly localization, and applying a threshold to the calibrated anomaly-detection images 98, 108 to detect the anomalies 15. [Explanation of symbols]

[0283] 10,10',10'',10'''' Computer-Implemented Method 11 Defective cell structure 12 Imaging Datasets 14 Input 15 Abnormality 16 Machine Learning Models 18 Reconstruction 19 Data Selection Steps 20 Abnormal detection image 21 Training data generation step 22,22' Input image 23 Defects 24,24' Reconstructed image 25 Hyperparameter value selection step 26,26' Abnormality detection image 27 Training Steps 28,28' Abnormality detection image 29 Model Evaluation Steps 30,30' input image 31 Model Selection Step 32,32' Reconstructed image 34 Flawless Data 36 Expert annotation 38 Expert Knowledge 40 Hyperparameter Range 42 Sampler 43 Hyperparameter Optimization Unit 44 Pruner 46 Objective Function 48 Hyperparameters 50 subsampled datasets 52 complete imaging datasets 54 Optimization Model 56 Final Model 58 Good group 60 Poor Group 61 Anomaly detection image generation step 62 Good flock density 63 Sampling Steps 64 Poor herd density 65 calibration steps 66 Expected Improvements 67 Training Steps 68 Worst Attempt 69 Application Steps 70 Best Attempts 71 Thresholding Step 72 Abnormal detection image 73 repetitions 74 Calibration Steps 76 Evaluation Steps 77 affirmative answers 78 End Step 79 negative answers 80 Calibration Method Adaptation Steps 82 Threshold or filter adjustment steps 84 Annotation Adjustment Steps 86 Training Steps 88 repetitions 90 Input image 92 Abnormal detection image 94 Click Points 95 Partially annotated anomaly detection images 96 Annotations 98 Calibrated anomaly detection image 100 Overlay 102 Reconstructed image 104 Input image 106 Uncalibrated Anomaly Detection Image 108 Calibrated anomaly detection image 110 Uncalibrated confusion matrix 112 Calibrated confusion matrix 114,114' system 116 Imaging Devices 118 Processing Device 120 wafers 122 CPU 124 Interface 126 memory 128 User Interface 130 means

Claims

1. A computer-implemented method (10, 10') for the detection of an anomaly (15), comprising: - selecting an imaging data set (12) of a wafer (120); generating training data from said imaging dataset (12); repeating the following steps: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter (48) defining a machine learning model (16) for the detection of anomalies (15); ii. training the machine learning model (16) defined by the hyperparameters (48) based on the generated subset of training data; iii. Evaluating the trained machine learning model (16) by calculating an associated objective function value of the objective function (46). selecting one of the trained machine learning models based on the associated objective function value and applying it to the imaging data set (12) of the wafer (120) to detect anomalies (15); A computer-implemented method (10, 10') comprising:

2. 2. The method of claim 1, wherein the machine learning model, when presented with a subset of the imaging dataset as input, is trained to compute an anomaly-free reconstruction of the subset, and the anomaly in the subset is detected based on a comparison between the subset and the reconstructed subset.

3. The method of claim 2 , wherein the machine learning model (16) comprises an autoencoder.

4. The method of any one of claims 1 to 3, wherein at least one hyperparameter (48) is a design hyperparameter related to the design of the machine learning model (16).

5. The method of any one of claims 1 to 4, wherein all of the at least one hyperparameter (48) are design hyperparameters related to the design of the machine learning model.

6. The method of any one of claims 1 to 5, wherein the objective function (46) comprises at least two different model evaluation measures.

7. The method of claim 6 , wherein at least one of the model evaluation metrics measures the quality of the anomaly detection.

8. The method of claim 6 or 7, wherein one of the model evaluation measures comprises a loss function used during training of the machine learning model (16).

9. The method of claim 8 , wherein one of the model evaluation measures is a loss function used during training of the machine learning model (16).

10. The method of any one of claims 6 to 9, wherein one of the model evaluation measures comprises an Lp-norm loss function, where p≧1, of the training data samples and corresponding target data samples.

11. 11. The method of claim 7, wherein the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and wherein one of the model evaluation measures comprises a discriminative loss function that evaluates the difference between the expert annotations of the anomalies and the detected anomalies.

12. The method of claim 11 , wherein the expert annotation (36) assigns labels to only a subset of types of anomalies (15) present in the imaging dataset (12).

13. The method of any one of claims 7 to 12, wherein one of the model evaluation measures comprises a measure of the complexity of the machine learning model (16).

14. 14. The method of claim 1, wherein the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and the objective function comprises a weighted sum of an Lp-norm loss function (p≧1) of the training data samples and corresponding target data samples and a discriminative loss function that evaluates differences between the expert annotations of the anomalies.

15. The method of claim 14 , wherein the objective function (46) further comprises a measure of complexity of the machine learning model (16).

16. 16. The method of claim 13 or 15, wherein the measure of the complexity of the machine learning model (16) takes into account the total number of floating-point operations in one forward propagation of the machine learning model (16).

17. 17. The method of claim 1, wherein the objective function (46) includes a quality value for evaluating a quality of the trained machine learning model (16), and a user interface (128) is configured to present information about the trained machine learning model (16) to a user and to allow the user to indicate the quality value.

18. 18. The method of any one of claims 1 to 17, wherein the sampling strategy for selecting the plurality of hyperparameter values ​​comprises taking into account hyperparameter values ​​from one or more prior iterations and corresponding values ​​of the objective function (46) by optimizing one criterion selected from the group consisting of expected improvement, maximum probability of improvement, and upper confidence interval.

19. The method of any one of claims 1 to 18, wherein the sampling strategy includes an early stopping criterion.

20. 20. The method of any one of claims 1 to 19, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy.

21. 21. The method of claim 20, wherein the probability distribution is modeled based on previous application knowledge of critical distances, critical dimensions, or pitch sizes.

22. The method of any one of claims 1 to 21, wherein the sampling strategy is different in at least two replicates.

23. The method of any one of claims 1 to 22, wherein the size of the generated subset of training data increases with the number of iterations.

24. 1. A computer-implemented method (10'', 10''') for detecting anomalies (15) in an imaging dataset (12) of a wafer (120), the imaging dataset (12) including defects (23) belonging to a plurality of defect classes, the method comprising: - applying an anomaly detection method to the imaging dataset (12) to generate an anomaly-detected image (20, 26, 26', 28, 28', 72, 92); - performing one or more iterations (88) comprising the steps of: i. providing one or more samples of a distribution of anomaly-detection image values ​​for each defect class of the subset of defect classes; ii. Calibrating said anomaly detection image (20, 26, 26', 28, 28', 72, 92) by at least one calibration method comprising the steps of: a) training a machine learning model for anomaly localization, preferably anomaly segmentation, based on the one or more samples of the distribution of values ​​of the anomaly-detected image; b. applying the trained machine learning model to the anomaly-detection images (20, 26, 26', 28, 28', 72, 92) to obtain calibrated anomaly-detection images (98, 108); - applying a threshold to said calibrated anomaly detection images (98, 108) to detect anomalies (15) to reduce nuisance and highlight defects (23) in said anomaly detection images (20, 26, 26', 28, 28', 72, 92); A computer-implemented method (10'', 10''') comprising:

25. 25. The method of claim 24, wherein the anomaly detection image (20, 26, 26', 28, 28', 72, 92) is a difference image between the imaging dataset (12) and a comparison dataset, and the comparison dataset is based on the die-to-die principle or the die-to-database principle.

26. 26. The method of claim 25, wherein the comparison dataset comprises the reconstructed representation of the imaging dataset (12) generated by training a machine learning autoencoder on the imaging dataset (12) or a subset thereof and applying the trained autoencoder to the imaging dataset (12) to obtain a reconstructed representation.

27. 27. The method of any one of claims 24 to 26, wherein the machine learning model for anomaly localization is trained to optimize a loss function based on anomalous and non-anomalous samples.

28. 28. The method of claim 27, wherein the loss function is a semi-supervised loss function.

29. 29. The method of claim 27 or 28, wherein the sample from the distribution of anomaly detection image values ​​for each defect class of the subset of defect classes is used as an anomaly sample.

30. 30. The method of claim 29, wherein the non-anomalous samples are selected from the remaining pixels of the anomaly-detected image (20, 26, 26', 28, 28', 72, 92).

31. 31. The method of claim 30, wherein the remaining pixels of the anomaly-detected image (20, 26, 26', 28, 28', 72, 92) are selected as non-anomalous samples if their anomaly-detected image values ​​are below a threshold.

32. Each abnormal sample is calculated by the loss function as the value a i In particular, the negative exponential weighting function w(a i ) = exp(-a i 32. The method of claim 30 or 31, wherein the weighting is performed by

33. The method of any one of claims 27 to 32, wherein the loss function is a semi-supervised weighted cross-entropy loss function.

34. The loss function is [Equation 1] where a is the abnormality detection image (20, 26, 26', 28, 28', 72, 92), y true is a pixel-level annotation from the set {unannotated=0, anomaly=1}, y pred 34. The method of claim 33, wherein: ∑ i denotes the label predicted by the machine learning model, w denotes a weighting coefficient, and i denotes a pixel index in the anomaly-detected image (20, 26, 26', 28, 28', 72, 92).

35. The method of any one of claims 24 to 34, wherein a single threshold is applied to the calibrated anomaly detection image (98, 108) to detect anomalies (15).

36. 36. The method of claim 24, wherein a user interface (128) is configured to allow a user to indicate locations of a small number of defects (23) of each class of the subset of defect classes in the anomaly detection image (20, 26, 26', 28, 28', 72, 92), the user indicating the location of the defect (23) by selecting a single pixel, preferably a pixel in a central region, of the defect (23), an annotation (96) of the defect (23) is automatically generated by selecting an area, preferably a circle, surrounding the selected single pixel, and values ​​of the anomaly detection image are sampled from the area surrounding the selected pixel.

37. 37. The method of claim 36, wherein the size of the region is selected based on a critical dimension and / or a critical distance and / or a pitch size.

38. The method of any one of claims 1 to 37, wherein the imaging dataset (12) comprises one or more multisensory images.

39. The method of any one of claims 1 to 38, wherein the imaging dataset (12) is a multi-beam SEM image.

40. The method of any one of claims 1 to 39, wherein the imaging data set (12) is a focused ion beam SEM image.

41. The method of any one of claims 1 to 40, further comprising measuring one or more properties of the detected anomaly (15).

42. 42. The method of claim 41, further comprising controlling at least one wafer manufacturing process parameter based on the one or more measured properties.

43. 43. The method of claim 42, further comprising assessing a quality of the wafer (120) based on the one or more measured properties and at least one quality assessment rule.

44. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices (118) to perform operations including the method of any one of claims 1-43.

45. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 43.

46. A system (114) for controlling the quality of wafers (120) produced in a semiconductor manufacturing facility, comprising: an imaging device (116) adapted to provide an imaging data set (12) of said wafer (120); one or more processing devices (118); one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices (118) to perform operations including the method of claim 43; A system (114) comprising:

47. A system (114') for controlling production of wafers (120) in a semiconductor fabrication factory, comprising: means (130) for producing wafers (120) controlled by at least one manufacturing process parameter; an imaging device (116) adapted to provide an imaging data set (12) of said wafer (120); one or more processing devices (118); one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices (118) to perform operations including the method of claim 42; The system (114') comprises:

Images