COMPUTER-IMPLEMENTED METHOD FOR DETECTION OF ANOMALIES IN IMAGING DATASETS OF WAFERS AND SYSTEMS EMPLOYING SUCH METHOD - Patent application

Abstract

The present invention relates to a computer-implemented method (10, 10') for detection of an anomaly (15) comprising the steps of selecting an imaging data set (12) of a wafer (120) and hyperparameter values ​​defining a machine learning model (16) for anomaly detection, training and evaluating the trained machine learning model (16) by calculating an objective function value, and selecting one of the trained machine learning models and applying it to the detection of the anomaly (15). The present invention also relates to a computer-implemented method (10'') for detection of an anomaly (15) in an imaging data set (12) of a wafer (120) comprising the steps of providing a sample of a distribution of values ​​of anomaly-detection images for each defect class, calibrating the anomaly-detection images (20, 26, 26') by training a machine learning model for anomaly localization, and applying a threshold to the calibrated anomaly-detection images (98, 108) to detect the anomaly (15).

Description

[Technical field]

[0001] Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 324258, filed March 28, 2022, and German Patent Application No. 102022113352.4, filed May 25, 2022, which are incorporated by reference in their entireties into this application.

[0002] The present invention relates to a computer-implemented method for detecting anomalies in an imaging dataset of a wafer, and to a machine-readable hardware storage device, a system for controlling the production of wafers in a semiconductor foundry, and a system for controlling the quality of wafers produced in a semiconductor foundry.

[0003] The present invention is not limited to computer-implemented methods for wafers, but can also be applied to any other manufactured object. [Background technology]

[0004] Quality control (QC) is used in the production process to iteratively improve the quality of the final product. Due to its central role, QC techniques not only need to be versatile, efficient and flexible, but also must be able to adapt to changing production conditions and use cases. Due to the lack of information on underlying defects that may vary with application or over time, QC systems must be based on broad assumptions to cover all defect types. Therefore, QC systems must be able to cope with cold start conditions.

[0005] Cold start refers to the problem that a learning-based system cannot draw inferences about items about which it has not yet gathered enough information. This problem occurs frequently in the semiconductor industry, as production processes and wafer types are constantly adjusted. Cold start is therefore a common problem in machine learning systems with automated data modeling, because machine learning models must be retrained from scratch every time system parameters are modified.

[0006] Unsupervised machine learning techniques, e.g. autoencoders, can successfully tackle the QC problem. During training, such techniques learn a compressed internal representation of abundantly available "clean" or "defect-free" data. As a result, the model is able to perfectly reconstruct defect-free image samples. During inspection, defects in the input image are not faithfully reconstructed. Spatial regions with large reconstruction errors indicate outliers with respect to the training data, also known as anomalies. Anomalies are localized deviations of an imaging dataset from an a priori defined norm, here the deviations from the normed semiconductor structure.

[0007] However, not all anomalies are defects. For example, anomalies may include imaging artifacts, image acquisition noise, changes in imaging conditions, variations in semiconductor structures within the norm, rare semiconductor structures, or variations due to imperfect lithography, changes in manufacturing conditions, or changes in wafer processing. Such anomalies that are not defects but are detected by some anomaly detection method are called nuisances.

[0008] Machine learning models based on unsupervised learning derive information only from the data, without requiring any human input, e.g. annotation, which may be laborious, noisy, or impractical. However, expert knowledge is still required to supervise the training of the model, i.e. to define various model parameters, called hyperparameters, such as the model design and its complexity (e.g. number of layers, number of filters per layer, size of the autoencoder bottleneck), its regularization (e.g. strategy and amplitude), data preprocessing techniques, diversity of the dataset, and learning strategy (e.g. learning rate and number of epochs). Expert selection of such hyperparameters is crucial to obtain high-quality machine learning models.

[0009] In the prior art, hyperparameter optimization methods have been proposed that optimize the hyperparameter values ​​of a machine learning model by finding an optimal set of hyperparameter values ​​that minimize the expected validation loss of the machine learning model. An example of such a machine learning model suitable for anomaly detection is an autoencoder.

[0010] Hyperparameter optimization methods have been proposed to automatically search for optimal hyperparameter values ​​for machine learning models. Among them, the Neural Architecture Search (NAS) method has been proposed to automatically search for optimal hyperparameter values ​​that define the architecture of a neural network.

[0011] Among these hyperparameter optimization methods, various techniques can be used to predict good hyperparameter candidates, as disclosed, for example, in "Max-value Entropy Search for Multi-Objective Bayesian Optimization; S. Belakaria, A. Deshwal, J. Doppa; Conference on Neural Information Processing Systems 2019".

[0012] A known hyperparameter optimization framework called Optuna was disclosed, for example, in "Akiba, T., Sano, S., Yanase, T., Ohta, T. and Koyama, M., 2019, Optuna: A next-generation hyperparameter optimization framework, in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631".

[0013] Another known method for hyperparameter optimization was disclosed, for example, in US Patent Application Publication No. 2020 / 0342329. In this disclosure, the autoencoder algorithm hyperparameters are optimized by evaluating the quality of the training progress, i.e., the history of the loss function values. The hyperparameter optimization method is unsupervised and does not use labeled training data.

[0014] In U.S. Patent Application Publication No. 2021 / 0256392, another hyperparameter optimization method was disclosed in which the hyperparameters are optimized by exploring a search space of hyperparameters and comparing the performance difference of corresponding anomaly detection networks.

[0015] However, applying the disclosed hyperparameter optimization method to an anomaly detection machine learning model is not straightforward. The anomaly detection model needs to reconstruct well the defect-free parts of the image, for example via mean squared error loss (L2 loss), but not the defects. Known hyperparameter optimization methods, such as Optuna or US Patent Application Publication No. 2020 / 0342329, evaluate the machine learning model using the same metric used during model training, and in most cases default to the L2 metric, i.e., the mean squared error of the predicted values. However, Optuna and similar methods of the prior art are not suitable for defect or anomaly detection, as they converge towards a perfect reconstruction of the input image, including all possible defects, which makes anomaly or defect detection impossible.

[0016] Machine learning models for anomaly detection often suffer from low precision rates due to noise, and high nuisance rates because not all anomalies are defects. Therefore, anomaly detection methods applied to wafer imaging datasets can face a very high nuisance rate n, which is the inverse of the precision rate p, i.e., n=1-p, because so many and mostly irrelevant deviations on the wafer surface are discovered. As a result, prior art anomaly detection methods require extensive post-processing to be useful for wafer surface defect detection. In addition, many anomaly detection methods do not provide accurate anomaly detection results at the pixel level.

[0017] An improved anomaly detection method was disclosed in "Attention Guided Anomaly Detection and Localization in Images, S. Venkataramanan, KC. Peng, RV Singh, A. Mahalanobis, ECCV 2020". The method proposed there is based on the idea of ​​using attention mechanism to distinguish between non-anomaly and anomalous areas, so that the training of the anomaly detection model is based on non-anomaly data, which can give improved results. However, this method is not applicable to semiconductor images. Moreover, the generated attention map does not have pixel-level accuracy, which does not meet the high-precision requirements in the semiconductor field.

[0018] Another improved anomaly detection method was disclosed in "ESAD: End-to-end Deep Semi-supervised Anomaly Detection, C. Huang, F. Ye, Y. Zhang, Y Wang, Q. Tian, ​​Arxiv 2020". This paper proposes semi-supervised training of anomaly detection based on a large unannotated dataset together with a small annotated dataset. However, this method only performs image-level anomaly detection to indicate whether anomalies exist in an image without localizing the anomaly. Again, this method does not meet the accuracy requirements in the semiconductor field.

[0019] A method for semantic segmentation was presented in "What's the point: semantic segmentation with Point Supervision; A. Bearman, O. Russakovsky, V. Ferrari, L. Fei-Fei; European Conference on Computer Vision 2016". The method addresses the problem of reducing the user effort in annotating images by showing a single point of an object instead of a pixel-level accurate segmentation.

[0020] Methods for automatic detection of defects that meet the precision requirements in the semiconductor industry include anomaly detection algorithms based on the die-to-die principle or the die-to-database principle. The die-to-die principle finds deviations from a typical or average wafer design by comparing a portion of the wafer with other portions of the same wafer. The die-to-database principle finds deviations from the ideal data by comparing a portion of the wafer with ideal simulation data from a database (e.g. the CAD file of the wafer).

[0021] Thus, in defect detection methods such as die-to-die or die-to-database, the intermediate output is typically a difference image showing the difference between the expected image (comparison data set) and the actual image. Anomaly suggestions are typically obtained by thresholding the difference image. In this way, accurate anomaly detection at pixel level can be obtained. However, setting the threshold is a balancing act between maximizing the capture rate (real defects flagged as anomalies) and minimizing the nuisance rate (imaging artifacts, noise, non-interesting defects, etc. flagged as anomalies). This step can be tedious, especially when defects with variations in shape, size, and appearance are to be detected.

[0022] Traditionally, threshold setting is a manual process in which an expert probes with increasing thresholds. At each step, a newly flagged anomaly is analyzed and threshold windows for different defect classes are selected. Alternatively, an expert provides several annotations for each defect class and extracts thresholds. However, both approaches are search-intensive and wasteful for low-contrast defects, resulting in a high nuisance rate.

[0023] Therefore, it is an object of the present invention to improve anomaly detection in semiconductor wafers. A further object of the present invention is to improve the accuracy of machine learning methods for anomaly detection in semiconductor wafers. A further object of the present invention is to reduce the user interaction required for the selection of hyperparameter values ​​of a machine learning model for anomaly detection. A further object of the present invention is to provide an anomaly detection method comprising hyperparameter optimization. A further object of the present invention is to make hyperparameter optimization applicable to machine learning methods for anomaly detection. A further object of the present invention is to provide an evaluation metric for determining the performance of a machine learning model for anomaly detection. A further object of the present invention is to provide an improved anomaly detection method comprising improved post-processing methods, e.g. improved threshold setting methods. Summary of the Invention

[0024] The present invention addresses the problem of improving the accuracy of machine learning methods for anomaly detection, in particular for wafer imaging datasets. Furthermore, the present invention aims to improve the accuracy of machine learning techniques for anomaly detection without compromising recall.

[0025] This object is achieved by the invention as defined in the independent claims. Advantageous embodiments and further developments of the invention are defined in the dependent claims.

[0026] According to a first embodiment of the present invention, a method is provided for reducing the effort required for selecting hyperparameter values ​​of a machine learning model for the detection of anomalies or defects in semiconductor wafers. The method provides for automatically optimizing at least one hyperparameter value of the machine learning model, thereby obtaining an optimized machine learning model for anomaly detection. Thus, according to a first embodiment of the present invention, a computer-implemented method for the detection of anomalies comprises: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - selecting an optimized machine learning model from one of the at least two trained machine learning models based on an associated objective function value; - applying the optimized machine learning model to the wafer imaging data set to detect anomalies; and the method step of selecting an optimized machine learning model comprises, for each of the at least two trained machine learning models: - selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for detection of anomalies; - training a machine learning model controlled by hyperparameter values ​​on an input subset of training data; - evaluating the trained machine learning model by calculating associated objective function values ​​of the objective function; Includes.

[0027] By selecting an optimized machine learning model from among multiple different machine learning models, the accuracy of the anomaly detection method is improved, and at the same time, the user effort required to train the machine learning model is reduced.

[0028] According to a second embodiment of the present invention, a computer-implemented method for detection of anomalies comprises: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - repeating the steps of: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for detection of anomalies; ii. training a machine learning model defined by the hyperparameters based on the generated subset of training data; iii. Evaluating the trained machine learning model by calculating associated objective function values ​​of the objective function. - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; Includes.

[0029] By selecting one of the trained machine learning models based on the associated objective function value, the optimized hyperparameters, e.g., architecture, of the machine learning model improve the accuracy of anomaly detection. At the same time, the user effort required to train the machine learning model is reduced. The trained machine learning model can be used for the detection of anomalies in an imaging data set of a wafer. The imaging data set can be obtained, for example, by means of a structured electron microscope.

[0030] In a machine learning model, a hyperparameter is a parameter whose value is used to control the learning process but is not learned from data. In contrast, the values ​​of other parameters (typically node weights) are derived by training from training data. According to an example of the first or second embodiment, the hyperparameter includes at least one of design hyperparameters or algorithmic hyperparameters. Design hyperparameters refer to the design of the machine learning model. Design hyperparameters include all hyperparameters related to the architecture of the machine learning model, such as the number of layers, the size of layers, the type of layers, the size of filters, the kernel size of convolutional layers, the type of convolution used, the upsampling scheme, the connections between layers, the bottleneck size, the size of the bottleneck filter, etc. An example of a model or design hyperparameter is the topology and size of the neural network. Algorithmic hyperparameters do not in principle affect the performance of the model, but affect the speed and quality of the learning process. Examples of algorithmic hyperparameters include the learning rate, the mini-batch size, the dropout rate, the size and content of the training dataset, and the type of loss function. A further example of an algorithm hyperparameter is the optimization algorithm used (stochastic gradient descent, Adam, RmsProp, etc.).

[0031] According to the first or second embodiment, at least one hyperparameter defining a machine learning model for the detection of anomalies may include, but is not limited to, at least one of the following examples: Design Hyperparameters: - Bottleneck size, - the size of the bottleneck filter (the number of features at the bottleneck), - initial filter size (the number of filters in the first layer of the network; other network functions are scaled proportionally to the first layer), - the type of convolution used, - upsampling scheme, - connections between layers, - the number of layers in the model, - the size of the layer in the model, - the type of layers in the model, - filter size, - kernel size of the convolutional layer, Algorithm hyper parameters: - Loss function type and / or parameters - the initial learning rate, - learning rate decay factor, - Use of momentum, - number of epochs, - regularization scale, - the size and content of the training set (number of images), - a sample representing the dataset, - Dropout rate, - Use of Nesterov's accelerated gradient, - Type of optimization algorithm.

[0032] In one example according to the first or second embodiment, the method according to the first or second embodiment further comprises selecting a plurality of hyperparameters that jointly define a machine learning model for detecting anomalies. For each selected hyperparameter, a set of associated hyperparameter values ​​is selected. Preferably, at least one hyperparameter to be optimized is related to the design of the machine learning model.

[0033] In one example of the method of the first or second embodiment, generating training data includes expert annotation of anomalies on a subset of the imaging dataset, which allows defining an objective function suitable for selecting an optimized machine learning model or for optimizing hyperparameters.

[0034] According to an example of the first or second embodiment, the subset of training data includes some samples of training data. It can also include the entire training data generated. The size of the subset of training data increases with the number of iterations, thereby reducing the calculation time.

[0035] In one example of the method according to the first or second embodiment, the input data and training data of the machine learning model may consist of tiles of a particular size of the imaging dataset. In one example according to the first or second embodiment, tiles (e.g., 2D images or 3D voxel arrays) are extracted from the imaging dataset and input to the machine learning model. The tiles may include sufficient spatial context of the anomaly to be detected. In one example according to the first or second embodiment, the tiles are at least as large as the expected anomaly and also incorporate spatial neighborhood context.

[0036] In one example according to the first or second embodiment, an autoencoder model is selected as the machine learning model, and the autoencoder machine learning model is trained to calculate a reconstructed subset of the imaging dataset of a wafer without anomalies.

[0037] An autoencoder machine learning model is a method of unsupervised machine learning that requires minimal user input. An autoencoder model is a type of artificial neural network used in unsupervised learning that learns to efficiently code unlabeled data. An autoencoder includes two main parts: an encoder that maps the input to a code, and a decoder that maps the code to a reconstruction of the input. The encoder and decoder neural networks can be trained to minimize the difference between the reconstructed representation of the input data and the input data itself. The code is a low-dimensional representation of the input data, and can therefore be thought of as a compressed version of the input data. In one example according to the first or second embodiment, the autoencoder is adapted to roughly reconstruct the input and preserve only the most relevant aspects of the data in the reconstruction. Thus, autoencoders can be used to detect anomalies. Anomalies generally relate to rare deviations from a norm in an imaging dataset. Because such occurrences are rare, the autoencoder does not reconstruct this type of information, and therefore suppresses anomalies in the imaging dataset.

[0038] In one example according to the first or second embodiment, in the step of anomaly detection according to the first or second embodiment, anomalies in the imaging dataset are detected based on a comparison between the imaging dataset and a comparison dataset. The comparison dataset includes information that allows detection of anomalies when compared to the imaging dataset. In one example according to the first or second embodiment, the comparison dataset includes a reconstructed imaging dataset that is ideally free of defects. In one example according to the first or second embodiment, the reconstructed imaging dataset is reconstructed by an autoencoder or by principal component analysis. In one example according to the first or second embodiment, the anomaly detection is performed using a distance measure or threshold operation applied to the comparison dataset.

[0039] Anomalies can be detected by comparing an imperfect reconstruction of the imaging dataset with the original imaging dataset. Any difference between the input image and the reconstructed representation of the input image indicates an anomaly. A distance measure between the input image and the reconstructed representation of the input image can be used to quantify whether an anomaly exists. The larger the difference, the more likely the tile contains an anomaly. Detecting anomalies can include applying one or more thresholds to the difference image between the original and the reconstructed image. In one example according to the first or second embodiment, a local threshold can be applied to a subset of the difference images. Additional measurements, such as the magnitude, location, or shape of the difference, or their local distribution, can also be used to detect anomalies. This detects defects of interest while reducing deviations due to noise.

[0040] In one example according to the first or second embodiment, the method for anomaly detection further comprises a step of defining an objective function suitable for selecting an optimized machine learning model or suitable for optimizing hyperparameters. The objective function according to this example can include at least one model evaluation measure.

[0041] The model evaluation metric is a quantifiable expression that measures a property of the trained machine learning model, and may include measuring at least one property related to the performance of the trained machine learning model, the quality of anomaly detection, the complexity of the trained machine learning model, the effort or cost to apply the trained machine learning model, etc. In one example according to the first or second embodiment, multiple model evaluation metric can be defined and appropriately combined in the objective function. In one example, previously generated validation data is provided to evaluate the model evaluation metric.

[0042] In one example according to the first or second embodiment, the objective function includes at least two model evaluation measures. The method using the objective function including at least two model evaluation measures is particularly applicable to anomaly detection. This is because the combination of different model evaluation measures can prevent the trained machine learning model, e.g., autoencoder, from reconstructing defects and ensure that the comparison data set deviates sufficiently from the imaging data set in the presence of defects. In this way, human effort is reduced, the performance of anomaly detection is maximized, and the quality, reproducibility, and stability of the results are improved. In addition, cold start is possible.

[0043] The objective function is a measure of the quality of the trained machine learning model and therefore of the hyperparameter values ​​selected for this model. By applying an objective function that includes at least two different model evaluation measures, it can be ensured that the optimization of the hyperparameter values ​​according to the first or second embodiment does not give a perfect reconstruction of the training data including anomalies, since all model evaluation measures are calculated simultaneously and therefore contribute to the objective function value of the objective function.

[0044] The objective function of the hyperparameter optimization can include or be a loss function used during training of the machine learning model as a model evaluation measure. The model evaluation measure can also include or be an Lp-norm loss function (p≧1), e.g., an L2 loss function. The Lp-norm loss function can measure the deviation of the training data samples from the corresponding target data samples, e.g., the deviation of the training data samples from an anomaly-free reconstruction, e.g., the deviation of the training data samples from an autoencoder reconstruction. This ensures correct reconstruction of anomaly-free image regions, improving anomaly detection results.

[0045] In one example according to the first or second embodiment, at least one of the model evaluation measures includes a discriminative loss function that evaluates the difference between expert annotations of the anomalies and the detected anomalies. The expert annotations include labels assigned to pixels or regions in the imaging dataset, the labels indicating, for example, anomaly or no anomaly. Thus, also or alternatively, the objective function may evaluate the defect / non-defect discriminative loss L using a small number of pixel-wise annotations provided by an expert user. CE In one example of the first or second embodiment, the difference between the original sample and the encoded sample (original-encoded) is kept as a continuous value, but clipped in the range (0,1). This value is directly compared to the binary label (anomaly / no anomaly) through cross-entropy loss. Note that the provided example does not need to encompass all defects in the dataset. Preferably less than 10%, more preferably less than 1%, of the training data are expert annotations. In addition, the expert annotations assign labels only to a subset of the anomalies present in the imaging dataset and / or to a subset of the types of anomalies present in the imaging dataset. In this way, expert user input is required to be minimized and cold start is possible, since expert user annotation is not specifically required when new defects occur in the imaging dataset. Note also that these amounts of training data are not sufficient to train a model alone.

[0046] According to one example according to the first or second embodiment, the expert annotations can be provided via multiple formats (e.g., bounding boxes, click points, auxiliary process information, etc.), and the discrimination loss (L CE ) is adapted accordingly. For example, the bounding box annotations are scaled by an overlap measure, i.e., the intersection-over-union loss L IOU is related to.

[0047] In one example of the first or second embodiment of the present invention, the expert annotations are provided, for example, as pixel-wise annotations or as bounding boxes. These expert annotations are facilitated using the expert's knowledge, for example based on a critical distance, a critical dimension, or a pitch size related to the minimum size of the structures on the wafer. In this way, the expert only needs to click on the center of the defect in the imaging data set, and a large area corresponding to the critical distance, the critical dimension, or the pitch size is automatically assigned as a defect around the click point.

[0048] To avoid overfitting, one of the model evaluation measures may further include a measure of the complexity of the machine learning model, e.g., an Occam's razor penalty for the complexity of the machine learning model. The model evaluation measure may be the total number of floating-point operations in one forward pass of the machine learning model (L FLOP ), e.g., consider the logarithm of this number. Alternatively, the number and / or size of layers of the trained neural network and / or the number of connections between neurons, and / or other suitable hyperparameter values ​​can be used by the model evaluation metric to measure the complexity of the machine learning model.

[0049] In one example of the first or second embodiment of the present invention, the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1) that measures the deviation of the training data samples from the corresponding target data samples (e.g., the deviation of the training data samples from their autoencoder reconstructions) and a discriminative loss function that evaluates the difference between the model predictions of anomalies and the expert annotations. Considering this particular combination of model evaluation measures, it simultaneously 1) evaluates the accuracy of the predictions on anomaly-free data (Lp-norm loss), 2) evaluates the accuracy of the predictions on anomalous data (discriminative loss), and 3) ensures very high accuracy of anomaly detection results, since the expert annotations for anomalies are specifically selected to prevent class imbalance.

[0050] In one example of the first or second embodiment of the present invention, the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1) that measures the deviation of the training data samples from the corresponding target data samples (e.g., the deviation of the training data samples from their autoencoder reconstructions), a discriminative loss function that evaluates the difference between the expert annotations of anomalies and the model predictions, and a measure of the complexity of the machine learning model. Considering this particular combination of model evaluation measures, a particularly high accuracy of anomaly detection results is ensured, since at the same time: 1) the accuracy of predictions on anomaly-free data is evaluated (Lp-norm loss), 2) the accuracy of predictions on anomalous data is evaluated (discriminative loss), 3) the expert annotations for anomalies are specifically selected to prevent class imbalance, and 4) overfitting is prevented (complexity measure). In one example, the objective function includes, for example, three model evaluation measures, for example: f=w1L2+w2L CE +w3L FLOP where the weights w1, w2, and w3 can be set by the user. Typically, the first two weights are similar in magnitude, while the third weight is at least ten times lower, e.g. w1=10, w2=1, and w3=0.01. It is also possible to set one or more of the weights to zero. For example, w3 can be 0, so that the objective function does not penalize the model complexity. The other L norms can be used instead of the L2 norm. p We can also use the norm. CE is an arbitrary loss function that penalizes the deviation of the difference image for anomalies in the imaging dataset from some ground truth data, e.g., the intersection-over-union loss L IOU Additional model performance measures can also be added. In addition, other options from the Optuna library may be chosen.

[0051] In one example according to the first or second embodiment, the objective function includes a quality value as a model evaluation measure for evaluating the quality of the trained machine learning model, similarly or alternatively to any of the above sections, and the user interface is configured to present information about the trained machine learning model to the user and have the user indicate the quality value. The information presented to the user may include precision rate, recall rate, sample anomaly detection, hyperparameter values ​​associated with the trained machine learning model, such as the design of the model, filters learned by the model for one or some of its layers, etc. When the objective function is minimized, if the quality of the trained machine learning model is high, the user may select a low value as the quality value. If the quality of the trained machine learning model is low, the user may select a high value as the quality value.

[0052] In one example according to the first or second embodiment, the hyperparameter values ​​are selected according to a sampling strategy that determines which hyperparameter values ​​to select next based on previously selected hyperparameter values ​​and associated objective function values. In one example according to the first or second embodiment, the sampling strategy includes the use of a sampling algorithm.

[0053] One example of a sampling strategy is the tree-structured Parzen estimator (TPE), which deals with categorical hyperparameters in a tree-structured manner. For example, the number of layers in a neural net and the number of neurons in each layer define the tree structure. For example, there cannot be a third layer without a second layer, and setting the number of neurons only makes sense if this layer is present in the graph. Another example is the choice of optimizer for a machine learning model, since each optimizer can have its own set of parameters.

[0054] Preferably, the sampling strategy for selecting the hyperparameter values ​​includes taking into account the hyperparameter values ​​and corresponding values ​​of the objective function from one or more prior iterations by optimizing one criterion selected from the group including expected improvement, maximum probability of improvement, upper confidence interval, etc. In this way, the performance of the method for selecting an optimized machine learning model is improved, and the anomaly detection results are improved.

[0055] In one example of the first or second embodiment, the method further includes evaluating an improved gain function. The gain function is a surrogate model of validation loss as a function of hyperparameter values ​​that can be fitted to previously obtained objective function values, also called previous observations, and is configured to predict where the local optimum of the objective function is. These methods are further referred to as Sequential Model-Based Optimization (SMBO). The surrogate model can function as at least a part of the objective function. Using the surrogate model simplifies the evaluation of the objective function and reduces computation time because the costly step of evaluating the objective function is performed less times.

[0056] For example, we can use the probability of improvement (PI) as improvement gain function, which evaluates the objective function f at the point that is most likely to improve this value. The objective function f is to be minimized. Let f' be the minimum value of f observed so far and let D be the previous observation, i.e. the objective function value obtained previously. This then corresponds to the following utility function associated with the evaluation of f at a given point x (corresponding to a set of hyperparameters):

[0057]

number

number

[0058] An alternative improvement capture function that takes into account the magnitude of improvement is expected improvement (EI). Expected improvement evaluates f at the point that best improves the expectation of f'. This corresponds to a utility function like this: u(x)=max(0,f'-f(x))

[0059] The expected gain function is then the expected utility as a function of x:

number

[0060] The point that is expected to produce the most improvement (maximum expected utility) is selected. The expected improvement has two components. The first can be increased by decreasing the mean function μ(x). The second can be increased by increasing the variance K(x;x). These two terms can be interpreted as explicitly encoding a trade-off between exploitation (valuing at a point with a low mean) and exploration (valuing at a point with high uncertainty). The exploitation-exploration trade-off is a classical consideration in such problems, and as a consequence of the Bayesian decision-theoretic treatment, the expected improvement criterion automatically captures both.

[0061] An alternative improvement gain function is commonly known as the upper bound confidence interval (UCB). It is typically described in terms of maximizing f rather than minimizing f, but in a minimization context, the improvement gain function takes the form a UCB (x;β)=μ(x)-βσ(x) where β>0 is a trade-off parameter,

[0062]

number

[0063] For each of the described sampling strategies, at least one set of hyperparameter values ​​can be associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy. The probability distribution can be predefined and therefore does not depend on the objective function value. The probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy can be modeled based on application-based prior knowledge, preferably based on design knowledge such as imaging hardware settings or critical distances. In this way, prior knowledge on the parameter space can be integrated into the method step of selecting an optimized machine learning model in the first embodiment, or the step of iteration in the second embodiment. Thus, the sampling strategy is improved to reduce the computation time, and therefore the anomaly detection results are improved.

[0064] In one example, the sampling strategy includes at least two different sampling strategies for the at least two trained machine learning models of the first embodiment, or the machine learning models trained during the iterative step of the second embodiment, resulting in improved results through a more exhaustive exploration of the hyperparameter value space.

[0065] Optionally, the step configured for selecting the optimized machine learning model according to the first embodiment or the step of iteration of the second embodiment includes a pruning strategy, e.g., a pruning algorithm that determines whether to continue or interrupt the training of a given machine learning model. In one example, the pruning strategy includes an early stopping criterion. In a further example, the pruning strategy includes an asynchronous successive halvings strategy. In a further example, the machine learning model generated by sampling a set of hyperparameter values ​​may be first tested based on an objective function on a small subset of training data samples. If performance is poor, the sampled hyperparameter values ​​can be discarded early, otherwise the size of the subset of training data can be increased. This pruning strategy reduces computation time. By applying the pruning strategy, the hyperparameter values ​​are tested and discarded early if the objective function does not show good results. Thus, computation time is saved and the method becomes applicable to cold start scenarios where retraining is often required.

[0066] According to a third embodiment of the present invention, there is provided a computer-implemented method for detection of anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method comprising the steps of: - generating an anomaly-detected image by applying an anomaly detection method to the imaging dataset; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of a subset of defect classes; ii. calibrating the anomaly detection image by at least one calibration method comprising the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly detection image; b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a threshold to the calibrated anomaly-detection image to detect anomalies, thereby reducing nuisances and highlighting defects in the anomaly-detection image.

[0067] In a fourth embodiment of the present invention, a computer-implemented method for detection of anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, is provided. - generating an anomaly-detected image by applying an anomaly detection method to the imaging dataset; - providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of the subset of defect classes; - detecting anomalies in the anomaly detection image by at least one calibration method, the calibration method comprising the steps of: training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly detection image; applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; Applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing nuisances and highlighting defects in the anomaly detection image.

[0068] The method according to the third or fourth embodiment of the present invention further enables the application of machine learning methods for anomaly detection to defect detection, requiring minimal user input, thereby reducing nuisance in the anomaly detection image while simultaneously highlighting anomalies or defects. Based on such enhanced or calibrated anomaly detection image, it is possible to robustly detect anomalies by applying a single threshold.

[0069] According to aspects of the third or fourth embodiment, the method further comprises automatically setting the threshold by using available side information (e.g., information related to the size of the wafer structures, such as critical distances, critical dimensions, pitch sizes, etc.) or by providing annotated defects of at least a subset of the defect classes. To this end, the method according to the third or fourth embodiment relates to a solution workflow that is robust to the size and / or contrast of the defects and calibrates the anomaly detection images, e.g., difference images, to highlight defects of interest while suppressing defects due to noise.

[0070] The concept of "anomaly localization" refers to any method that computes the location of anomalies, such as anomaly segmentation methods, semantic anomaly segmentation methods, anomaly detection methods, classification methods, regression methods, methods for finding out-of-distribution samples, etc.

[0071] The number of defect classes can relate to a single defect class, multiple defect classes, or all defect classes occurring in the imaged data set.

[0072] The term "anomaly detection image" refers to the output of the anomaly detection method in the form of an image indicative of an anomaly, for example by pixel-wise labeling or bounding boxes. In one example of the third or fourth embodiment, the anomaly detection image is a difference image between an imaging dataset and a comparison dataset, for example a reconstruction of the imaging dataset. In one example, the comparison dataset is based on a die-to-die principle or a die-to-database principle.

[0073] In one example of the third or fourth embodiment, the comparison dataset may include a reconstructed representation of the imaging dataset generated by training a machine learning autoencoder on the imaging dataset or a subset thereof and applying the trained autoencoder to the imaging dataset to obtain a reconstructed imaging dataset. The generation of such an autoencoder and the reconstructed imaging dataset is discussed in the description of the first or second embodiment of the present invention.

[0074] In this way, the quality of the anomaly detection images is improved. Instead of an autoencoder, principal component analysis can also be used to generate the reconstruction dataset.

[0075] In one example of embodiment of the third or fourth embodiment, the anomaly detection method is a method according to the first or second embodiment of the present invention.

[0076] In one example of the third or fourth embodiment, the at least one calibration method is selected from a group of calibration methods, and in one example, the steps of determining a calibrated anomaly-detection image and applying a single threshold to the calibrated anomaly-detection image are repeated iteratively for each calibration method in the group of calibration methods.

[0077] In one example of the third or fourth embodiment, the method further includes selecting, e.g., by user input, one or more values ​​of the domain of the distribution of values ​​of the anomaly-detected image as thresholds. The thresholds may include minimum and maximum values ​​of the domain of the distribution of values ​​of the anomaly-detected image, e.g., minimum and maximum intensity values ​​of the anomaly or defect. This method allows multiple adapted thresholds to be applied in combination with the filters.

[0078] For example, the anomaly detection image can be calibrated by applying at least one calibration method including the following steps: - for each defect class of the subset of defect classes, calculating an intermediate calibrated anomaly detection image by adapting the value of the anomaly detection image based on the selected one or more thresholds of the anomaly detection image value of the current defect class; - applying one or more filters for the current defect class, e.g. a size filter; and finally - applying the operator to all intermediate calibrated anomaly-detection images to generate calibrated anomaly-detection images.

[0079] Throughout this specification, the term "subset" of a set refers to a single member, some members, or all of the members of the set.

[0080] In one example of the third or fourth embodiment, the operator is selected from a group including pixel-wise sum, pixel-wise average, pixel-wise minimum, pixel-wise maximum, pixel-wise scaling. In this way, the final calibrated anomaly detection image contains defects from different defect classes, each of which is extracted from the original anomaly detection image based on a different set of thresholds and filters. Using the maximum operator preserves as many anomalies as possible (provided the anomalies are marked with a value higher than the background), whereas the minimum operator further reduces noise and nuisance.

[0081] According to a second example of the third or fourth embodiment, the method includes a step of providing annotated defects. The annotated defects are used to automatically set a required threshold to be applied to the calibrated anomaly detection image. According to the second example, only a small number of annotated defects are required for a small number of defect classes, and other defect classes may remain unannotated. The method according to this example relies on the assumption that the annotated defects cover the appearance spectrum of all defect classes. A small number of defect classes means that the subset of defect classes includes less than 50% of all defect classes, preferably less than 30% of all defect classes, more preferably less than 20% of all defect classes, and most preferably less than 10% of all defect classes. A small number of annotated defects of each class means more than 5 but less than 20, preferably more than 5 but less than 10 annotated defects per defect class. In the method according to the second example, the problem of defect annotation is addressed, and the threshold of the calibrated anomaly detection image can be automatically set.

[0082] In the calibration method according to the second example, the task of automatically determining the threshold can be formulated as a pixel-wise localization problem, for example, a pixel-wise segmentation problem. Many different designs of the pixel-wise localization of the second embodiment are possible depending on various assumptions, for example, annotation types (bounding boxes, click points, pixel-level or image-level annotations, annotations by multiple users, annotations obtained from secondary sources, etc.), loss functions (dealing with model complexity through regularization, incorporating prior knowledge, handling class imbalance, etc.), or problem formulations (semantic segmentation, object detection, classification, regression, handling out-of-distribution samples, etc.).

[0083] In one example of the third or fourth embodiment, a machine learning model for anomaly localization is trained to optimize a loss function based on anomalous and non-anomalous samples. This allows for pixel-by-pixel detection of anomalies of importance in semiconductors with high accuracy. Samples from the distribution of anomaly detection image values ​​for each defect class of a subset of defect classes can be used as anomaly samples. In a further example, the loss function is a semi-supervised loss function. Based on the semi-supervised loss function, expert annotations can be taken into account during training of the calibration method, thus improving the quality of anomaly detection. However, only a few annotations are required and most of the samples are selected automatically, so that the user effort is kept at a low level. Throughout this specification, the term "foreground sample" is used as a synonym for "anomalous sample" and the term "background sample" is used as a synonym for "non-anomalous sample".

[0084] Specifically, a sample from the distribution of values ​​of the anomaly-detected image for each defect class of the subset of defect classes may be used as an anomaly sample, while a non-anomaly sample may be (automatically) selected from the remaining pixels of the anomaly-detected image. A remaining pixel of the anomaly-detected image may be selected as a non-anomaly sample if its anomaly-detected image value is below a threshold. Additionally or alternatively, each non-anomaly sample may be selected based on its anomaly-detected image value a i In particular, the negative exponential weighting function w(a i )=exp(-a i) to automatically select a large number of non-anomalous pixels while minimizing user effort. Also, high weights can be assigned to pixels with low anomaly detection image values, i.e. pixels with the lowest likelihood of belonging to an anomaly, such as pixels with very small reconstruction errors of the autoencoder, ensuring high accuracy in the selection of non-anomalous pixels. Thus, by selecting only a few samples from the distribution of anomaly detection image values ​​for each defect class in the subset of defect classes, samples from these distributions that were not selected can be prevented from being erroneously used as non-anomalous samples. In addition, a small number of selected samples is sufficient to obtain high accuracy in anomaly detection, reducing user effort.

[0085] The user is requested to provide only a few pixel-level annotations for a subset of defect classes. The subset can include all defect classes. Preferably, the subset does not include all defect classes. More preferably, the subset includes only a few defect classes, e.g., less than 10% of the defect classes. The annotation process is facilitated by leveraging available meta-information (e.g., critical dimension, critical distance, or pitch size). The user can provide a click point that is processed into a pixel-level annotation, e.g., by expanding the click point into a circle of a size corresponding to the critical dimension, critical distance, or pitch size that represents a lower or upper limit of the minimum feature size on the wafer. Alternatively, the user can use a brush to mark the defective pixels. Based on these annotations, a machine learning anomaly localization model, e.g., anomaly segmentation or anomaly detection model, can be trained. This localization model, when applied to the anomaly detection image, labels each pixel as an anomaly-of-interest or a nuisance. In this way, anomalies are highlighted and nuisances are reduced. Additional post-processing techniques, such as histogram equalization, location-based or size-based filtering, etc., can be used to further reduce nuisances. The calibrated anomaly detection images are finally thresholded using a single threshold, e.g., 0.5, to obtain the defects in the imaging data set.

[0086] In one example of the third or fourth embodiment of the present invention, during training, the machine learning model for anomaly localization is a machine learning model for anomaly segmentation. The model considers as input a partially annotated anomaly detection image, such as a difference image. As output, the model labels each pixel as a target anomaly or as a nuisance. Since not all input pixels are annotated by the user, the model learning is based on the following semi-supervised loss function:

[0087]

number

[0088]

number

number

number

[0089] Alternatively, various other loss functions can be used to train the machine learning model for anomaly localization, such as the Kullback-Leibler divergence loss function, L1 or L2 loss function, etc.

[0090] An example of the third or fourth embodiment provides a method for calibrating differential images, allowing for high recall and manageable precision by applying a single threshold to the calibrated anomaly detection images. Thus, the machine learning model according to the third or fourth embodiment includes method steps for setting up, training and applying a non-linear filter configured to amplify defects while suppressing nuisances in differential images used for semiconductor defect detection.

[0091] In one example according to the third or fourth embodiment of the present invention, the user interface is configured to let the user indicate in the anomaly detection image the location of a small number of defects of each class of the subset of defect classes. For this purpose, the user interface is configured for user indication of the location of the defect by selecting a single pixel, preferably a pixel in a central region, of said defect, an annotation of the defect is automatically generated by selecting an area, preferably a circle, surrounding the selected single pixel, and a value of the anomaly detection image is sampled from the area surrounding the selected pixel. Preferably, the size of the area is selected based on side information such as a critical dimension and / or a critical distance and / or a pitch size. The critical distance and the critical dimension relate to the smallest size of the structures on the wafer, and the pitch size relates to the smallest distance between the structures, and therefore can be understood as an upper limit of the smallest size of the wafer structures. That is, it can be assumed that the anomaly is at least as large as the smallest structure on the wafer. Thus, the generated anomaly area surrounding the selected pixel can be automatically selected as the smallest size of the wafer structures in terms of the critical dimension, the critical distance or the pitch size (converted to pixels). In this way, the accuracy of the anomaly detection is improved by automatically increasing the number of samples to be labeled. In addition, by limiting the size of the anomaly region surrounding the selected pixel based on the size of the wafer structure, false labeling can be prevented, especially for very small structures on the wafer. Moreover, a single click is sufficient for annotating the defect, which significantly reduces the annotation effort of the user. In this way, despite the limited availability of training data at the beginning of training, a cold start is feasible since very few selected samples are sufficient to train the machine learning model for anomaly localization.

[0092] The anomaly detection method according to the first or second embodiment and the anomaly detection method according to the third or fourth embodiment can be trained in many variations. They can be trained jointly as a single module, which makes maintenance and evaluation easier. On the other hand, independent modules have the advantages of generality and separation of expert opinions.

[0093] It should be noted that a computer-implemented method for detection of anomalies in an imaging dataset can simultaneously include both hyperparameter optimization according to either the first or second example embodiment and anomaly detection image calibration according to either the third or fourth example embodiment. All features according to the embodiments or examples can be applied to this combined approach as well.

[0094] Anomaly detection is the first step in the workflow of defect detection (and classification, if possible). In particular, to have a method suitable for cold starts, anomaly detection is a valuable first step to scan large amounts of data, so that only samples that are likely to contain defects are presented to the user for annotation.

[0095] In addition, one or more properties of the detected anomalies can be measured, such as their size, location or shape parameters, or the anomaly density for a particular area or for the entire imaged data set. Based on such measurements, at least one wafer manufacturing process parameter can be controlled based on the one or more measured properties, or the quality of the wafer can be evaluated based on the one or more measured properties and at least one quality assessment rule.

[0096] In this manner, the detected anomalies can be utilized to control the quality of wafers produced in a semiconductor foundry or to control the production process of wafers in a semiconductor foundry.

[0097] Additionally, one or more machine-readable hardware storage devices may include instructions executable by one or more processing devices to perform operations including any of the methods disclosed herein.

[0098] An inspection system for controlling the quality of wafers produced in a semiconductor manufacturing facility comprises the following features: an imaging device adapted to provide an imaging dataset of the wafer, an optional graphical user interface configured to present data to a user and to obtain input data from the user, one or more processing devices, and one or more machine-readable hardware storage devices including instructions executable by the one or more processing devices to perform operations including one of the methods disclosed herein including assessing the quality of the wafer based on one or more measurements and at least one quality assessment rule.

[0099] A system for controlling production of wafers in a semiconductor manufacturing factory comprises the following features: means for producing wafers controlled by at least one manufacturing process parameter; an imaging device adapted to provide an imaging dataset of said wafers; an optional graphical user interface configured to present data to a user and to obtain input data from the user; one or more processing devices; and one or more machine-readable hardware storage devices including instructions executable by the one or more processing devices to perform operations including a method including controlling at least one wafer manufacturing process parameter based on one or more measurements.

[0100] According to the embodiments described herein, various imaging modalities may be used to acquire imaging data sets for defect detection and classification. With various imaging modalities, it may be possible to obtain different imaging data sets. The imaging data set may include one or more multisensory images. The imaging data set may be a multibeam SEM image, or a focused ion beam image, for example, generated by a Helium Ion Microscope (HIM). The imaging data set may include a two-dimensional image, a three-dimensional image, a slice-by-slice three-dimensional image, or a multisensory fusion image. For example, the imaging data set may include a 2D image. Here, it may be possible to employ a multibeam SEM. A multibeam SEM simultaneously acquires images in multiple fields of view using multiple beams. For example, 50 or more, or 90 or more beams may be used. Each beam covers a separate portion of the wafer surface. This results in a large amount of imaging data set in a short time. Typically, 4.5 gigapixels are acquired per second. For example, imaging one square centimeter of a wafer with a pixel size of 2 nm results in 25 terapixels of data. Other examples of imaging datasets including 2D images are for imaging modalities such as optical imaging, phase contrast imaging, X-ray imaging, etc. It is also possible that the imaging dataset is a volumetric 3D dataset that can be processed slice by slice or as a three-dimensional volume. Here, cross-beam imaging devices including a focused ion beam (FIB) source, an atomic force microscope (AFM) or a scanning electron microscope (SEM) can be used. Multimodal imaging datasets may also be used, for example combining X-ray imaging and SEM. The imaging dataset 22 may additionally or alternatively include an aerial image obtained by an aerial imaging system. The aerial image is a radiation intensity distribution at the substrate level. The aerial image can be used to simulate the radiation intensity distribution generated by the photolithographic mask 14 during the photolithographic process.The aerial image measurement system may comprise, for example, a staring array sensor, or a line scan sensor, or a time delay integration (TDI) sensor.

[0101] Although examples and embodiments of the invention are described with respect to semiconductor wafers, it should be understood that the invention is not limited to semiconductor wafers, but may also be applied, for example, for masks for semiconductor manufacturing, or in a variety of other fields, such as anomaly detection in manufacturing parts or biological samples.

[0102] It should be noted that the present invention described by way of examples and embodiments is not limited to these embodiments and examples, but can be implemented by those skilled in the art through various combinations or improvements thereof. In the following, advantageous exemplary embodiments of the present invention are described and are illustrated diagrammatically in the drawings. [Brief description of the drawings]

[0103] [Figure 1] FIG. 1 illustrates a schematic defective cell structure containing multiple anomalies due to various defects. [Diagram 2] FIG. 1 shows a flowchart and results of applying an anomaly detection method such as an autoencoder to a wafer imaging dataset. [Diagram 3] 4 is a flowchart of an anomaly detection method according to the first or second embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing the results of an anomaly detection method obtained by a general hyperparameter optimization method. [Diagram 5] 4 is a flowchart of an anomaly detection method according to an example of the first or second embodiment of the present invention. [Figure 6] FIG. 2 illustrates the results of an anomaly detection method according to an example of the first or second embodiment of the present invention, based on an objective function that includes at least two different model evaluation measures. [Figure 7] FIG. 1 illustrates the sampling strategy of the tree-structured Parzen estimator. [Figure 8]FIG. 1 illustrates a pruning strategy for the asynchronous successive halvings algorithm. [Figure 9] FIG. 13 shows the evolution of objective function values ​​for different machine learning models defined by hyperparameters selected according to the sampling strategy of the tree-structured Parzen estimator. [Figure 10] 11 is a flowchart of a computer-implemented method for detection of anomalies in an imaging data set of a wafer according to a third or fourth embodiment of the present invention. [Figure 11] 11 is a flowchart of a computer-implemented method for detection of anomalies in an imaging dataset of a wafer according to an example of the third or fourth embodiment of the present invention. [Figure 12] FIG. 11 illustrates steps of a computer-implemented method for detection of anomalies in an imaging dataset of a wafer according to a third or fourth embodiment of the present invention. [Figure 13] FIG. 11 shows a comparison of an anomaly detection image obtained by a computer-implemented method for detection of anomalies in an imaging dataset of a wafer according to the third or fourth embodiment of the present invention with a calibrated anomaly detection image. [Figure 14] FIG. 11 shows confusion matrices for anomaly-detection images and calibrated anomaly-detection images obtained by a computer-implemented method for detection of anomalies in an imaging dataset of a wafer according to the third or fourth embodiment of the present invention. [Figure 15] 1 is a schematic diagram of a system for controlling the quality of wafers in a semiconductor manufacturing factory. [Figure 16] 1 is a schematic diagram of a system for controlling the production of wafers in a semiconductor foundry; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0104] 1 shows a schematic defective cell structure 11 that includes a number of anomalies 15. The anomalies 15 are localized deviations of an imaging data set 12 from an a priori defined norm, here a deviation from a normed semiconductor structure.

[0105] FIG. 2 is a flow chart and result diagram of an anomaly detection method such as an autoencoder applied to an imaging data set 12 of a wafer 120. The input 14 of the method comprises an imaging data set 12 of a wafer 120, which includes one or more images of the wafer 120. Based on the imaging data set 12, a machine learning model 16 is trained. This machine learning model 16 can be based on a die-to-database principle or on a die-to-die principle. The autoencoder model is trained on the imaging data set 12 on a die-to-die principle. The autoencoder learns a compressed internal representation of "clean" or "defect-free" data, which is abundantly available. As a result, the machine learning model 16 is able to perfectly reconstruct defect-free image samples. During inspection, contaminated input images are not faithfully reconstructed. Spatial regions with large reconstruction errors indicate outliers with respect to the training data, also known as anomalies 15. Based on the autoencoder model, a comparison data set in the form of a reconstruction 18 of the input 14 is calculated. Since defects 23 in the imaging data set 12 are not reconstructed by the autoencoder, the difference between the reconstruction 18 (comparison data set) and the input 14 includes defects 23 in the imaging data set 12. Not all such differences are due to defects 23, since imaging artifacts such as noise, semiconductor structure variations, or lithography imperfections also cause differences in the difference image. Thus, defects 23 only form a subset of anomalies 15. Anomalies 15 that are not defects 23 but are detected by some anomaly detection method are referred to as nuisances. The difference image shows anomalies 15 in the imaging data set 12. Thus, the difference image is the anomaly detection image 20.

[0106] Based on a defect-free input image 22, a reconstructed image 24 is obtained which differs only due to noise as shown in the difference image 26. The input image 22' containing the defect 23 is reconstructed in the reconstructed image 24', except for the area containing the defect 23 which is only partially reconstructed. The difference image 26' therefore contains deviations from 0 at the defect locations. The defects can be localized by applying a threshold to the difference image 26'.

[0107] Since an autoencoder is a machine learning model based on unsupervised learning, it derives information only from the input data without human input. However, expert knowledge is still required to supervise the model training by defining the hyperparameter values ​​(e.g., bottleneck size) of the hyperparameters that define the machine learning model. To reduce the required user interaction and improve the reconstruction results of the autoencoder, a computer-implemented method 10 for the detection of anomalies 15 according to the first or second embodiment of the present invention as shown in FIG. 3 includes the following steps: in a data selection step 19, selecting an imaging data set 12 of a wafer 120; in a training data generation step 21, generating training data from the imaging data set 12 and repeating the following steps: in a hyperparameter value selection step 25, based on a sampling strategy, selecting a hyperparameter value from a related hyperparameter value set or selecting multiple hyperparameters from multiple related hyperparameter value sets; A selection step, where the hyper parameter values ​​correspond to at least one hyper parameter defining a machine learning model 16 for detection of anomalies 15; a training step 27, a step of training the machine learning model 16 defined by the hyper parameters 48 based on a subset of the generated training data, and a model evaluation step 29, a step of evaluating the trained machine learning model 16 by calculating an associated objective function value of the objective function; a model selection step 31, a step of selecting one of the trained machine learning models based on the associated objective function value and applying it to the imaging dataset 12 of the wafer 120 to detect anomalies 15.

[0108] The objective function ensures that the hyper-parameter or hyper-parameters are selected such that the machine learning model defined by the hyper-parameter or hyper-parameters correctly reconstructs the imaging dataset 12 except for regions containing defects 23. To this end, the objective function may include a model evaluation measure in the form of an Lp-norm loss function (p≧1), which ensures correct reconstruction of the anomaly-free imaging dataset 12 by penalizing deviations. For example, the L2-norm loss function, which is a mean squared error loss function, is selected by default as the objective function for most hyper-parameter optimization methods.

[0109] However, when the objective function of the hyperparameter optimization includes only the Lp-norm based model evaluation measure, the autoencoder learns to also reconstruct the defects 23 contained in the imaging dataset 12. Thus, the defects 23 contained in the input images are at least partially reconstructed and the difference image shows only small or no deviation from 0 at the defect locations, as shown in Fig. 4.

[0110] In Fig. 4, the results of the anomaly detection method are shown, which are obtained by a general hyperparameter optimization technique. An input image 30 is reconstructed by an autoencoder machine learning method 16, which is p It was obtained by standard hyperparameter optimization methods based on an objective function consisting of norm measures only. This results in a comparison data set in the form of a reconstruction image 32 which contains all the defects 23, so that the anomaly detection image 28 in the form of a difference image does not indicate as anomalies 15 any of the defects 23 present in the imaging data set 12.

[0111] 5 is a flow chart of an anomaly detection method 10' according to an example of the first or second embodiment of the present invention. The method comprises a hyperparameter optimization unit 43 performing a training data generation step 21 for generating training data from the imaging dataset 12, a hyperparameter value selection step 25 for selecting hyperparameter values, a training step 27 for training a machine learning model 16 based on the selected hyperparameter values, a model evaluation step 29 for evaluating the trained machine learning models by an objective function 46, and a model selection step 31 for selecting one of the trained machine learning models 16 based on an associated objective function value.

[0112] The machine learning model for anomaly detection trained according to the first or second embodiment of the present invention is based on the die-to-die principle or the die-to-database principle. In either case, training the machine learning model 16, e.g. an autoencoder or a principal component analysis, requires abundant and preferably defect-free data 34.

[0113] The hyperparameter optimization unit 43 requires an objective function 46 that is used to evaluate each of the trained machine learning models 16. The objective function 46 includes one or more model evaluation measures. In one example of the first or second embodiment of the present invention, the objective function includes at least two different model evaluation measures 46 to prevent the trained machine learning models from reconstructing the anomalies 15 as well as the defects 23.

[0114] L p Apart from the norm loss function, the preferred model evaluation metric is the discriminative loss function L CE, which also requires a small number of expert annotations 36. The expert annotations 36 are given, for example, as pixel-wise annotations or as bounding boxes. These expert annotations 36 can be facilitated using the expert knowledge 38, for example based on a critical distance, critical dimension, or pitch size related to the smallest size of structures on the wafer. In this way, the expert only needs to click on the center of the defect 23 in the anomaly detection image or imaging data set 12, and a large area around the click point corresponding to the critical distance, critical dimension, or pitch size is automatically assigned as the defect 23.

[0115] The hyperparameter optimization unit 43 includes a sampler 42 and a pruner 44. The sampler 42 is used to execute the sampling strategy by selecting hyperparameter values ​​for the hyperparameters 48 from a set of associated hyperparameter values. In one example, expert knowledge 38 is applied by introducing hyperparameter ranges 40 for the hyperparameters. From these ranges, the hyperparameter values ​​are selected by the sampler 42. Additionally or alternatively, the expert knowledge 38 can be applied by introducing probability distributions indicating the likelihood of each hyperparameter value to be selected by the sampling strategy based on imaging hardware settings or design knowledge, such as, for example, critical distances, critical dimensions and / or pitch sizes. Each hyperparameter value corresponds to at least one hyperparameter 48 that defines a machine learning model for the detection of anomalies 15. The hyperparameters 48 may refer, for example, to the architecture of the neural network underlying the machine learning model (e.g., the number and size of layers, or the size of the bottleneck of an autoencoder), or the learning rate. An optional pruner 44 decides whether training of the machine learning model 16 should continue by selecting another hyperparameter value according to the sampling strategy, or whether training should be interrupted, for example by an early stopping criterion. In case of interruption, new hyperparameter values ​​can be selected randomly, if possible, with respect to a given hyperparameter range and / or probability distribution for hyperparameter values. Training is then started again according to the sampling strategy.

[0116] The machine learning model 16 is trained on a sub-sampled data set 50 sampled from the full imaging data set 52. The size of the sub-sampled data set 50 grows with the number of iterations performed by the hyperparameter optimization unit 43. In this way, a cold start on a small sub-sampled data set 50 can be achieved with little effort and time for the user. In addition, new training data (e.g., training data including new types of defects) can be easily incorporated into the training process, thus enabling online learning and optimization. In this way, the training process can be adapted quickly and easily. After selecting the trained machine learning model that achieves the best objective function value, the optimized machine learning model 54 is finally trained on the full imaging data set 52 to obtain a final machine learning model 56 for anomaly detection.

[0117] 6 shows the results of an anomaly detection method in the first or second embodiment of the present invention, where at least two different model evaluation measures are combined in an objective function 46 to obtain improved results. In addition to the Lp norm (e.g., the L2 norm, which penalizes deviations of the reconstructed image 32' from the input image 30'), the objective function 46 includes a discriminative defect / non-defect loss function L based on a small number of pixel-wise annotations provided by an expert. CE The expert indicates the locations of defects 23 that should not be reconstructed by the autoencoder. The anomaly detection image 28' (difference image) can thus be compared to the binary labels indicated by the user by additionally using cross-entropy loss as an additional model evaluation measure in the objective function 46. The expert annotations 36 are not required to encompass all types of defects 23 occurring in the imaging dataset 12, and the number of annotations can be very small, e.g., less than 10%, preferably less than 1%, of the training data samples. The annotations can be provided via multiple formats (e.g., bounding boxes, click points, auxiliary process information, etc.), so that the discrimination loss (L CE) needs to be adapted accordingly. For example, the bounding box annotation is scaled by an overlap measure, i.e., the intersection-over-union loss L IOU is related to.

[0118] In one example of the first or second embodiment of the present invention, the objective function 46 also includes a model metric that includes an Occam's razor penalty on complexity to avoid overfitting. The model metric is the total number of floating-point operations in one forward pass of the model (L FLOP ) Alternatively, the number and / or size of layers of the trained neural network and / or the number of connections between neurons, and / or other suitable hyperparameter values, can be used as a model evaluation metric to measure the complexity of the machine learning model. This complexity-based model evaluation metric is optional.

[0119] Thus, in one example of the first or second embodiment of the present invention, the objective function is a set of three model evaluation measures, e.g. f=w1L2+w2L CE +w3L FLOP where the weights w1, w2, and w3 are chosen by an expert. Typically, the first two weights are similar in magnitude, while the third weight is at least ten times lower, e.g., w1=10, w2=1, and w3=0.01. One or some of the weights, e.g., weight w3, can be set to 0. Instead of the L2 norm, other L p We can also use the norm. CE is an arbitrary loss function that penalizes the deviation of the difference image for anomalies in the imaging dataset from some ground truth data, e.g., the intersection-over-union loss L IOUcan be substituted. In one example, the objective function also or alternatively includes a quality value as a model evaluation measure for evaluating the quality of the trained machine learning model 16, and the user interface 128 is configured to present information about the trained machine learning model 16 to the user and have the user indicate the quality value. Additional model evaluation measures may be added or used instead of one or more of the model evaluation measures mentioned above. In addition, all options of the backbone Optuna library may be added.

[0120] Figure 6 shows an input image 30' containing the same defects 23 as in Figure 4. In reconstructed image 32', defects 23 are not reconstructed because the objective function includes at least two model evaluation measures. Although the background image is correctly reconstructed, the reconstruction does not replicate any of the defects 23 (white circles) as desired. As a result, the difference image, as anomaly detection image 28', contains defects 23.

[0121] FIG. 7 illustrates a sampling strategy for a Tree-Structured Parzen Estimator (TPE) that can be used for sampling hyperparameter values ​​by sampler 42.

[0122] The Parzen density estimator is the density of the observations x=x1,...,x n is a nonparametric kernel density estimator used to estimate the probability density function (hyperparameter values) of a random variable given a set of:

number

[0123] The Parzen estimator is close in concept to Bayesian optimization, but in theory it is the opposite: while Bayesian optimization seeks to find p(y|x), where y is the objective function value and x is a hyperparameter value, Parzen estimator trees model p(x|y) and p(y).

[0124] With respect to Bayesian optimization, the first step in TPE is to start sampling the objective function 46 by random search to initialize the algorithm.

[0125] The observations are then split into two groups: those that perform best according to the objective function 46 (the good group 58, e.g., the top quartile), and y * The remainder defines (poor group 60).

[0126] The probability of being in each of these groups is y <y * If p(x|y)=l(x), y ≥ y * If x, then it is modeled as p(x|y) = g(x) (the Gaussian process models the posterior probability).

[0127] The two densities l and g are modeled using the Parzen density estimator, where g denotes the density 62 of the good group and l denotes the density 64 of the bad group.

[0128] p(y) defines the percentile split between two categories. <y * ) = δ (i.e., if g models the top quartile, δ = 0.75).

[0129] Using Bayes' rule (i.e., p(x,y)=p(y)p(x|y)), the definition of expected improvement (EI)66 can be shown to be equivalent to l(x) / g(x).

[0130] The sampling strategy then selects the next point as the one that maximizes l(x) / g(x), as shown in FIG.

[0131] FIG. 8 illustrates the pruning strategy of the asynchronous successive halvings algorithm used by the pruner 44 to decide whether to continue or stop training the model, in an example of the first or second embodiment of the present invention. The vertical axis shows the training loss function, and the horizontal axis shows the training time. The lower the objective function value, the better the performance of the model defined by the associated hyperparameter value or values. Only the hyperparameter value or values ​​that achieve the lowest objective function value are continuously improved, while the remaining set is discarded in early iterations.

[0132] Asynchronous successive halving algorithm (ASHA) The ASHA algorithm, shown in Figure 8, is a pruning strategy that combines random search with normative early stopping in an asynchronous manner.

[0133] The successive halving algorithm (SHA) is a well-known multi-armed bandit algorithm for prescriptive early stopping. The successive halving algorithm starts with all candidate configurations in the base rung and proceeds as follows: 1. Uniformly assign a budget (value on the horizontal axis) to the set of candidate hyperparameter configurations at a given stage; 2. Evaluate the performance of all candidate configurations; 3. Promote the top 1 / η ​​of the candidate configurations to the next stage, where η is the erasure rate selected by the user; 4. In the next stage, double the budget per configuration and repeat until only one configuration remains.

[0134] Higher η indicates a more aggressive elimination rate, where all configurations except the top 1 / η ​​are eliminated.

[0135] The SHA algorithm is difficult to parallelize because the algorithm takes a set of configurations as input and waits for all configurations in one stage to complete before promoting a configuration to the next stage.

[0136] To remove the bottleneck created by synchronous promotion, the asynchronous successive halving algorithm (ASHA) grows bottom-up and promotes configurations as much as possible, instead of starting with a broad set of configurations and narrowing them down.

[0137] ASHA starts by assigning a worker to add configurations to the bottom tier. When a worker finishes one job and requests a new one, it checks the tiers from top to bottom, checking if there is a configuration in the top 1 / η ​​of each tier that can be promoted to the next tier. If there isn't, the worker is assigned to add one configuration to the bottom tier, widening the width of the level so that more configurations can be promoted.

[0138] The output of the ASHA algorithm is shown in Figure 9, which summarizes the results of all trials. Here, 17 trials were performed for hyperparameter optimization. Among the initial trials, the worst trials 68, which achieved the worst objective function values, were dropped in the first iteration. In each subsequent iteration, the best model from the previous iteration was seeded and trained for longer. Finally, the best trials 70 were retrained on the entire original training dataset.

[0139] FIG. 10 shows a flowchart of a computer-implemented method 10″ for anomaly detection in an imaging dataset 12 of a wafer 120, the imaging dataset 12 including defects 23 belonging to multiple defect classes according to the third or fourth embodiment of the present invention, the method comprising the following steps: in an anomaly detection image generation step 61, generating an anomaly detection image 72 by applying an anomaly detection method to the imaging dataset 12; performing one or more iterations 73, at least one of which comprises the following steps: in a sampling step 63, sampling one or more of the distributions of values ​​of the anomaly detection image for each defect class of a subset of defect classes; or a plurality of samples; in a calibration step 65, calibrating the anomaly detection image 72 by at least one calibration method comprising the following steps: in a training step 67, training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly detection image; in an application step 69, applying the trained machine learning model to the anomaly detection image 72 to obtain a calibrated anomaly detection image; in a thresholding step 71, applying a threshold to the calibrated anomaly detection image to detect anomalies 15, thereby reducing nuisances and highlighting defects 23 in the anomaly detection image 72.

[0140] 11 shows a computer-implemented method 10''' for anomaly detection in an imaging data set 12 of a wafer 120 according to an example of the third or fourth embodiment of the present invention. The anomaly-detection image 72 is calibrated in a calibration step 74 in each iteration 88. In the calibration step 74, a calibration method is selected from a set of calibration methods.

[0141] The set of calibration methods comprises a first calibration method for training a machine learning model for anomaly localization based on one or more samples of the distribution of values ​​of the anomaly detection image for all defect classes of the subset of defect classes. The trained machine learning model is then applied to the anomaly detection image 72. Finally, a threshold is applied to the calibrated anomaly detection image to obtain anomalies 15 or defects 23.

[0142] The set of calibration methods also includes a second calibration method based on global or local thresholding and / or filtering operations. For this purpose, one or more lower and / or upper thresholds are selected for the anomaly detection image 72, and the anomaly detection image 72 is adapted based on these thresholds. The filtering operations can include, for example, morphological cleaning, cluster size filtering, size filtering, etc. The size filter indicates, for example, a minimum and a maximum size for each defect type, so that only anomalies or defects exhibiting a size within the indicated size range are detected. The size of the anomaly or defect can be measured by the number of connected pixels, the length of the anomaly or defect in a particular direction, or its diameter, etc. In the calibration method, thresholding and filtering operations can be combined. The user interface can be configured to allow the user to select the thresholds and / or filters. Finally, the thresholds are applied to the calibrated anomaly detection image to obtain the anomaly 15 or defect 23.

[0143] In an evaluation step 76 it is checked whether all (or a sufficiently large number) of the defects 23 have been detected. In case of an affirmative answer 77 the iteration 88 is ended in an end step 78. Otherwise, in case of a negative answer 79 the calibration method is adapted in an adapt calibration method step 80.

[0144] To adapt the first calibration method, annotations are added or adapted in annotation adjustment step 84 by expert annotation, e.g., by clicking on defects and applying expert knowledge such as critical dimensions and / or critical distances and / or pitch sizes related to minimum structure sizes. To adapt the second calibration method, thresholds and / or filters are added or adapted in threshold or filter adjustment step 82, e.g., by user input. The calibration method can be adapted based on additional information, e.g., a machine learning model for anomaly localization is retrained in training step 86. In each iteration 88, a different calibration method can be selected and applied to the calibrated anomaly detection images from the previous iteration.

[0145] In one example of the third or fourth embodiment of the present invention, the second calibration method includes the following steps: for each defect class of the subset of defect classes, calculating an intermediate calibrated anomaly detection image by adapting the values ​​of the anomaly detection image based on one or more selected thresholds of the values ​​of the anomaly detection image of the current defect class; applying one or more filters, e.g., a size filter, of the current defect class, whereby only anomalies 15 within a certain size range are preserved by the calibration method; and finally, generating a calibrated anomaly detection image by applying an operator to all the intermediate calibrated anomaly detection images, the operator being selected from the group including pixel-wise sum, pixel-wise average, pixel-wise minimum, pixel-wise maximum, pixel-wise scaling. In this way, the final calibrated anomaly detection image includes defects 23 from different defect classes, each of which was extracted from the anomaly detection image 72 based on a different set of thresholds and filters. Using the maximum operator preserves as many anomalies 15 as possible, whereas the minimum operator further reduces noise and nuisance.

[0146] The adaptation of the value a of the anomaly detection image by the first calibration method is performed by normalizing the value based on an upper threshold u and a lower threshold l, i.e.

number

[0147] Alternatively, the adaptation of the anomaly detection image values ​​may involve clipping the values ​​to the range [l, u]. a new =min(max(a,l),u)

[0148] In an example of the third or fourth embodiment of the present invention, the calibration method is formulated as a pixel-wise segmentation problem, as shown in FIG. 12. The input image 90 to the anomaly detection method contains two defect classes 23, called hollow diamonds and triangles. The output of the anomaly detection method is an anomaly detection image 92, which is a difference image between the input image 90 and a comparison dataset (here the output of an autoencoder that reconstructs the input image 90). The mottled background is due to noise and high reconstruction error near the defects 23. A simple threshold that ensures a high defect recall rate would result in a high nuisance rate. Therefore, the anomaly detection image 92 is calibrated with a machine learning model for anomaly localization. For this purpose, the user interface is configured to let the user provide expert annotations 96 for several samples of a subset of defect classes. To facilitate the annotation process, the user provides click points 94 that are automatically processed into pixel-level annotations 96 by utilizing available meta-information (e.g., critical distances and / or critical dimensions and / or pitch sizes). Based on this meta-information, the click points are extended to an area covering the smallest feature size of the wafer. In the partially annotated anomaly detection image 95, the annotated pixels are set to 1 and the non-annotated pixels are set to 0. It is sufficient if the user provides only a small number of click points (on the order of 5-10, less than 20) for a subset of defect classes. Even if the remaining defects 23 are not part of the training data, they can still be segmented by the machine learning model for anomaly segmentation due to the assumption that the annotated classes cover the appearance range of all defect classes. Based on the annotations, the machine learning model for anomaly segmentation is trained. In this way, the user effort is minimized and a cold start is possible, since new defects 23 can also be detected based on limited training data, and the user effort required for the annotation process is also minimized.

[0149] According to an example of the third or fourth embodiment of the present invention, during training, the machine learning model for anomaly segmentation considers as input partially annotated anomaly detection images, such as difference images. As output, the model labels each pixel as a target anomaly or a nuisance. Since not all input pixels are annotated by the user, the model learning is based on the following semi-supervised loss function:

[0150]

number

[0151]

number

number

number

[0152] FIG. 13 shows a qualitative analysis of the method according to the third or fourth embodiment of the invention. In the top row, the reconstructed image 102 differs from the input image 104 in the areas corresponding to the defects 23. In the bottom row, the uncalibrated anomaly detection image 106 is shown before applying the calibration method. This uncalibrated image 106 shows a high recall (all defects 23 are highlighted) but a low precision (a lot of specular highlighting corresponding to noise), i.e. a lot of false positives. Moreover, the strength of highlighting differs for each defect class. In the right column, the calibrated anomaly detection image 108 is shown after applying the calibration method. The calibrated anomaly detection image 108 shows both high recall and precision, with all defect classes showing a uniform intensity of 1.0. These can be easily and automatically detected based on a single uniform threshold of 0.5. Thus, applying a single threshold to the calibrated anomaly detection image 108 is sufficient for anomaly detection, since the application of the calibration method preserves a high recall, significantly improves precision and guarantees uniform intensity values ​​for the defects 23.

[0153] The benefit of applying the calibration method according to the third or fourth embodiment of the present invention is quantified in Fig. 14 which shows the confusion matrix before 110 and after calibration 112, i.e. before and after segmenting the anomaly detection image 108 and applying a threshold of 0.5. In this way, a high recall (0.93 vs. 1.0) is maintained and a large reduction in false positives (from 0.61 to 0.25) is achieved, resulting in a significant improvement in precision.

[0154] 15 illustrates a schematic of a system 114 that can be used to control the quality of wafers 120 produced in a semiconductor foundry. The system 114 includes an imaging device 116 and a processing device 118. The imaging device 116 is coupled to the processing device 118. The imaging device 116 is configured to acquire an imaging data set 12 of the wafer 120. The wafer 120 can include semiconductor structures, e.g. transistors such as field effect transistors, memory cells, etc. Exemplary embodiments of the imaging device 116 are a SEM or a multi-beam SEM, a helium ion microscope (HIM) or a cross-beam device including a FIB and a SEM, or any charged particle imaging device.

[0155] The imaging device 116 can provide the imaging dataset 12 to the processing device 118. The processing device 118 includes a processor, for example implemented as a CPU 122 or a GPU. The processor can receive the imaging dataset 12 via an interface 124. The processor can load program code from a memory 126. The processor can execute the program code. Upon executing the program code, the processor performs techniques as described herein, such as hyperparameter optimization, training an anomaly detection method, performing an anomaly detection method to detect one or more anomalies 15 in the imaging dataset 12 of the wafer 120, calibrating the anomaly detection image based on a sample of the value distribution of the anomaly detection image, etc. For example, upon loading the program code from the memory 126, the processor can execute the computer-implemented method shown in FIG. 3, FIG. 6 or FIG. 10, respectively. Optionally, the processing device can include a user interface 128 for inputting user input, such as a click point, a bounding box, or a characteristic of the distribution of values ​​of the anomaly detection image.

[0156] FIG. 16 illustrates in a schematic manner a system 114′ that can be used to control the production of wafers 120 in a semiconductor manufacturing plant. The system 114′ includes the same components as shown in FIG. 14, and what has been said above applies for each of the components in this figure. In addition, the system 114′ includes a means 130 for producing wafers 120 controlled by at least one wafer manufacturing process parameter. For this purpose, the imaging data set 12 is provided to a processing device 118 by means of an imaging device 116. A processor of the processing device 118 is configured to execute one of the disclosed methods that includes controlling at least one wafer manufacturing process parameter based on one or more measured properties of the detected anomalies 15 in the imaging data set 12 of the wafer 120. For example, a detected anomaly 15 due to a bridge defect indicates insufficient etching, so the etching amount is increased, a detected anomaly 15 due to an open line indicates excessive etching, so the etching amount is decreased, a consistently occurring anomaly 15 indicates a defect in the mask, so the mask needs to be checked, and an anomaly 15 due to a missing structure suggests that the material deposition is not ideal, so the material deposition is modified.

[0157] The following clauses include preferred embodiments of the present invention:

[0158] 1a. A computer-implemented method for anomaly detection, comprising: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - repeating the steps of: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for detection of anomalies; ii. training a machine learning model defined by the hyperparameters based on the generated subset of training data; iii. Evaluating the trained machine learning model by calculating associated objective function values ​​of the objective function. - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; 4. A computer-implemented method comprising:

[0159] 1b. A computer-implemented method for detection of anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes; - generating training data from the imaging dataset; - repeating the steps of: i. selecting a plurality of hyperparameter values ​​from a set of associated hyperparameter values ​​based on a sampling strategy, each hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for detection of anomalies; ii. training a machine learning model for anomaly detection based on the generated subset of training data; iii. evaluating the trained machine learning model according to an objective function that includes at least two different model evaluation measures; - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies, thereby providing an anomaly-detected image; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of a subset of defect classes; ii. calibrating an anomaly detection image based on the sample of the distribution by a calibration method selected from a group of calibration methods, thereby reducing nuisances and highlighting defects in the anomaly detection image, wherein at least one of the selected calibration methods includes the steps of: c. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly-detection images for all defect classes of the subset of defect classes; d. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a single threshold to the calibrated anomaly detection image to detect anomalies; 4. A computer-implemented method comprising:

[0160] 1c. A computer-implemented method for detection of anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method comprising: - providing an anomaly-detected image generated by applying an anomaly detection model to an imaging dataset; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of a subset of defect classes; ii. calibrating an anomaly detection image based on the sample of the distribution by a calibration method selected from a group of calibration methods, thereby reducing nuisances and highlighting defects in the anomaly detection image, wherein at least one of the selected calibration methods includes the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly detection images for all defect classes of the subset of defect classes; b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a single threshold to the calibrated anomaly detection image to detect anomalies; 4. A computer-implemented method comprising:

[0161] 1. A computer-implemented method for detection of anomalies in an imaging dataset of a wafer, comprising: - generating training data from the imaging dataset; - repeating the steps of: i. selecting a plurality of hyperparameter values ​​from a set of associated hyperparameter values ​​based on a sampling strategy, each hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for detection of anomalies; ii. training a machine learning model for anomaly detection based on the generated subset of training data; iii. evaluating the trained machine learning model according to an objective function that includes at least two different model evaluation measures; - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; 4. A computer-implemented method comprising:

[0162] 2. The method of any one of the preceding clauses, wherein a machine learning model is trained when presented with a subset of an imaging dataset as input to compute an anomaly-free reconstruction of the subset, and anomalies in the subset are detected based on a comparison between the subset and the reconstructed subset.

[0163] 3. The method of claim 2, wherein the machine learning model includes an autoencoder.

[0164] 4. The method of any one of the preceding clauses, wherein at least one hyperparameter is related to the design of the machine learning model.

[0165] 5. The method of any one of the preceding clauses, wherein one of the model evaluation measures comprises a loss function used during training of the machine learning model.

[0166] 6. The method of any one of the preceding clauses, wherein one of the model evaluation measures comprises an Lp-norm loss function, where p≧1.

[0167] 7. The method of any one of the preceding clauses, wherein the training data includes expert annotations of anomalies for a subset of the imaging dataset, and one of the model evaluation measures includes a discriminative loss function that evaluates the difference between the expert-annotated anomalies and the detected anomalies.

[0168] 8. The method of claim 7, wherein the expert annotations account for less than 10%, preferably less than 1%, of the training data.

[0169] 9. The method of claim 7 or 8, wherein the expert annotations include only a subset of the anomalies present in the imaging dataset.

[0170] 10. The method of any one of the preceding clauses, wherein one of the model evaluation measures includes a measure of the complexity of the machine learning model.

[0171] 11. The method of claim 10, wherein the measure of complexity of the machine learning model includes a total number of floating-point operations in one forward propagation of the machine learning model.

[0172] 12. The method of any one of the preceding clauses, wherein the objective function comprises a weighted sum of at least two model evaluation measures.

[0173] 13. The method of clause 12, wherein the training data includes expert annotations of anomalies for a subset of the imaging dataset, and the objective function includes a weighted sum of an Lp-norm loss function (p≧1), a discriminative loss function that evaluates the difference between the expert-annotated anomalies and the detected anomalies, and a measure of the complexity of the machine learning model.

[0174] 14. The method of any one of the preceding clauses, wherein the sampling strategy for selecting a plurality of hyperparameter values ​​includes taking into account hyperparameter values ​​and corresponding values ​​of the objective function from one or more prior iterations.

[0175] 15. The method of claim 14, wherein the sampling strategy selects the hyperparameter values ​​by optimizing one criterion selected from the group including expected improvement, maximum probability of improvement, and upper confidence interval.

[0176] 16. The method of any one of the preceding clauses, wherein the sampling strategy includes early stopping criteria.

[0177] 17. The method of any one of the preceding clauses, wherein the sampling strategy comprises randomly selecting hyperparameter values ​​from said relevant set of hyperparameter values.

[0178] 18. The method of any one of the preceding clauses, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy.

[0179] 18a. The method according to the preceding clause, wherein the probability distribution is modelled based on knowledge based on previous usage, preferably based on design knowledge such as imaging hardware settings or critical distances.

[0180] 19. The method of any one of the preceding clauses, wherein the sampling strategies are different in at least two replicates.

[0181] 20. The method of any one of the preceding clauses, wherein the size of the generated subset of training data varies based on a sampling strategy.

[0182] 21. The method of clause 20, wherein the size of the generated subset of training data increases with the number of iterations.

[0183] 22. The method of any one of the preceding clauses, wherein prior knowledge based on a particular application is used to select initial hyperparameter values.

[0184] 23. The sampling strategy is - in a first iteration: selecting initial hyperparameter values ​​from a set of associated hyperparameter values; - in a subsequent iteration: selecting, based on the hyperparameter values ​​and corresponding values ​​of the objective function from one or more previous iterations, a hyperparameter value from the associated set of hyperparameter values ​​that maximizes a measure of expected improvement; - if an early stopping criterion is met in an iteration: selecting initial hyperparameter values ​​from a set of associated hyperparameter values; 2. The method according to any one of the preceding clauses, comprising:

[0185] 24. The method of any one of the preceding clauses, wherein the iterations are performed in parallel by multiple threads.

[0186] 25. The method of any one of the preceding clauses, wherein the selected machine learning model is trained on a full imaging dataset of the wafer before being applied to the imaging dataset of the wafer to detect anomalies.

[0187] 26. The method according to any one of the preceding clauses, in which the trained machine learning model that achieves the lowest objective function value is selected.

[0188] 27. The method according to any one of the preceding clauses, wherein the trained machine learning model that achieves the highest objective function value is selected.

[0189] 28. The method according to any one of the preceding clauses, wherein the sampling strategy criteria implements at least one component selected from the group consisting of an exploratory scheme and an exploitative scheme.

[0190] 28a. A method according to any one of the preceding clauses, wherein the objective function includes a quality value for evaluating a quality of the trained machine learning model, and the user interface is configured to present information about the trained machine learning model to a user and to enable the user to indicate the quality value.

[0191] 29. A computer-implemented method for detection of anomalies in an imaging dataset of a wafer, the imaging dataset including defects belonging to a plurality of defect classes, the method comprising: - providing an anomaly-detected image generated by applying an anomaly detection model to an imaging dataset; - performing one or more iterations, the iterations including the steps of: i. providing one or more characteristics of a distribution of values ​​of the anomaly detection image for each defect class of the subset of defect classes; ii. calibrating the anomaly detection image based on characteristics of said distribution with a calibration method selected from a set of calibration methods, thereby reducing nuisances and highlighting defects in the anomaly detection image. 4. A computer-implemented method comprising:

[0192] 30. The method of claim 29, 1a, 1b or 1c, wherein the anomaly detection image is a difference image between the imaged data set and the comparison data set.

[0193] 31. The method according to clause 30, wherein the comparative data set is based on the die-to-die principle or the die-to-database principle.

[0194] 32. The method of claim 30 or 31, wherein the comparison dataset is generated by a machine learning model.

[0195] 33. The method of clause 32, wherein the comparison dataset includes a reconstructed representation of the imaging dataset generated by training an autoencoder on the imaging dataset or a subset thereof and applying the trained autoencoder to the imaging dataset to obtain a reconstructed representation.

[0196] 34. The method of claim 29, 1a, 1b or 1c, wherein the anomaly detection image is generated by a machine learning model.

[0197] 35. The method of clause 34, wherein anomaly detection and calibration of the anomaly detection image are jointly learned by a machine learning model that is applied to the imaging dataset to directly obtain a calibrated anomaly detection image.

[0198] 36. The method of any one of clauses 29-35, 1a, 1b or 1c, wherein the one or more characteristics comprises a sample from said distribution.

[0199] 37. The method of any one of clauses 29-36, 1a, 1b or 1c, wherein the one or more characteristics comprises the top quartile and / or the bottom quartile of said distribution.

[0200] 38. The method of any one of clauses 29-37, 1a, 1b or 1c, wherein the one or more characteristics include moments of the distribution.

[0201] 39. The method of any one of clauses 29-36, 1a, 1b or 1c, wherein the one or more characteristics include a minimum and / or a maximum of a domain of said distribution.

[0202] 40. The method of any one of clauses 36-39, further providing a filter for each defect class of the subset of defect classes.

[0203] 41. The method of claim 40, wherein the filter comprises a size filter.

[0204] 42. The method according to clause 40 or 41, wherein the set of calibration methods comprises a calibration method for calibrating the anomaly detection image by applying the following steps: - calculating, for each defect class of the subset of defect classes, an intermediate calibrated image obtained by: i. adapting the anomaly detection image values ​​based on one or more characteristics of a distribution of anomaly detection image values ​​for the current defect class; ii. applying one or more filters for the current defect class; - generating a calibrated anomaly detection image by applying an operator to all intermediate calibrated images, the operator being selected from the group comprising: pixel-wise sum, pixel-wise average, pixel-wise minimum, pixel-wise maximum, pixel-wise scaling.

[0205] 43. The method of claim 42, wherein the values ​​of the anomaly detection image are adapted by normalization.

[0206] 44. The method of claim 42, wherein the value of the anomaly detection image is adapted by clipping.

[0207] 45. The method according to clauses 29-44, 1a, 1b or 1c, wherein the set of calibration methods comprises a calibration method for calibrating the anomaly detection image by applying the following steps: - training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more characteristics of the distribution of values ​​of the anomaly detection image for all defect classes of the subset of defect classes; - applying the trained machine learning model to the anomaly-detection images to obtain calibrated anomaly-detection images.

[0208] 46. ​​The method of claim 45, wherein the machine learning model is trained to optimize a loss function for anomaly localization based on foreground samples and background samples.

[0209] 47. The method of claim 46, wherein the loss function is a weighted cross-entropy loss function.

[0210] 48. The method of claim 46, wherein the loss function is a Kullback-Leibler divergence loss function, or an L1 loss function or an L2 loss function.

[0211] 49. The method of any one of clauses 46 to 48, wherein the loss function is a semi-supervised loss function.

[0212] 50. The method of any one of clauses 45 to 49, wherein a machine learning model for anomaly localization is trained on partially labeled anomaly detection images derived from properties of the distribution.

[0213] 51. A method according to any one of clauses 45 to 50, wherein the one or more characteristics include a sample from said distribution being used as a foreground sample or a background sample.

[0214] 52. The method of claim 51, wherein the samples are used as foreground samples and background samples are selected from the remaining pixels of the anomaly detection image.

[0215] 53. The method of claim 52, wherein the remaining pixels of the anomaly-detected image are selected as background samples if their anomaly-detected image values ​​are below a threshold.

[0216] 54. The method according to clause 52 or 53, wherein each background sample is weighted by a weighting function w of its anomaly detection image value a, in particular by a negative exponential weighting function w(a)=exp(-a).

[0217] 55. The method of any one of clauses 45 to 54, wherein the machine learning model has been pre-trained with training data from a similar application.

[0218] 56. The method of any one of clauses 45 to 55, wherein the machine learning model addresses class imbalance by using focal loss functions and / or defect rate priors.

[0219] 57. The method of any one of clauses 29-56, 1a, 1b or 1c, further comprising presenting the anomaly detection image to a user via a user interface, the user interface configured to allow the user to input information regarding each defect class of the subset of defect classes from which one or more characteristics of the distribution of values ​​of the anomaly detection image for each defect class are derived.

[0220] 58. The method of clause 57, wherein the user interface is configured to allow a user to indicate in the anomaly detection image the location of a small number of defects of each class of a subset of defect classes, and one or more characteristics of the distribution of anomaly detection image values ​​for each of these classes are prepared by sampling anomaly detection image values ​​from the defects indicated by the user for that class.

[0221] 59. The method of claim 58, wherein a user indicates the location of a defect by selecting a single pixel, preferably a pixel in a central region, of the defect.

[0222] 60. The method of claim 59, wherein the defect annotation is automatically generated by selecting an area, preferably a circle, surrounding the selected single pixel.

[0223] 61. The method of clause 60, wherein anomaly detection image values ​​are sampled from an area surrounding the selected pixel.

[0224] 62. The method of clause 60 or 61, wherein the regions are selected based on application specific knowledge, in particular from the field of wafer manufacturing.

[0225] 63. The method of claim 62, wherein the size of the region is selected based on a critical dimension.

[0226] 64. The method of claim 62 or 63, wherein the size of the region is selected based on the pitch size.

[0227] 65. The method of any one of clauses 29-64, 1a, 1b or 1c, wherein the calibrated anomaly detection image is post-processed to further reduce nuisance.

[0228] 66. The anomaly detection method of any one of clauses 29-65, 1a, 1b or 1c, wherein one or more characteristics of the distribution of values ​​of the anomaly detection image for each defect class of the subset of defect classes are prepared based on similar applications.

[0229] 67. The method of any one of clauses 29-66, 1a, 1b or 1c, further generating an uncertainty estimate for each anomaly in the calibrated anomaly detection image.

[0230] 68. The method according to any one of clauses 29-67, 1a, 1b or 1c, further comprising a final step after performing one or more iterations. - thresholding the calibrated difference image with a single threshold to obtain a binary anomaly detection image.

[0231] 69. The method of clauses 29-68, 1a, 1b or 1c, wherein the subset of defect classes includes less than 50% of the defect classes, preferably less than 30% of the defect classes, more preferably less than 20% of the defect classes, and most preferably less than 10% of the defect classes.

[0232] 70. The method of any one of clauses 29-69, 1a, 1b or 1c, wherein multiple repetitions are performed and the calibration method is different in at least two of the multiple repetitions.

[0233] 70a. The method of any one of the preceding clauses, wherein the imaging dataset includes one or more multisensory images.

[0234] 70b. ​​The method of any one of the preceding clauses, wherein the imaging dataset comprises a two-dimensional image, a three-dimensional image, a slice-by-slice three-dimensional image, or a multisensory fusion image.

[0235] 71. A method according to any one of the preceding clauses, wherein the imaging data set is a multi-beam SEM image.

[0236] 72. A method according to any one of the preceding clauses, wherein the imaging data set is a focused ion beam SEM image.

[0237] 73. The method of any one of the preceding clauses, further comprising measuring one or more properties of the detected anomaly.

[0238] 74. The method of clause 73, further comprising controlling at least one wafer manufacturing process parameter based on the one or more measured properties.

[0239] 75. The method of clause 73, further comprising assessing a quality of the wafer based on the one or more measured properties and at least one quality assessment rule.

[0240] 76. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods described in any one of clauses 1 to 75.

[0241] 77. A system for controlling the quality of wafers produced in a semiconductor manufacturing facility, comprising: - an imaging device adapted to provide an imaging data set of said wafer; - one or more processing devices; - one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods of clause 75.

[0242] 78. A system for controlling the production of wafers in a semiconductor manufacturing facility, comprising: - means for producing wafers controlled by at least one manufacturing process parameter; - an imaging device adapted to provide an imaging data set of the wafer; - one or more processing devices; - one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods of clause 74; Including, the system.

[0243] 79. A computer-implemented method for anomaly detection, comprising: - selecting an imaging data set of a wafer; - generating training data from the imaging dataset; - repeating the steps of: i. selecting a hyperparameter value from a set of associated hyperparameter values ​​based on a sampling strategy, the hyperparameter value corresponding to at least one hyperparameter defining a machine learning model for detection of anomalies; ii. training a machine learning model defined by the hyperparameters based on the generated subset of training data; iii. Evaluating the trained machine learning model by calculating associated objective function values ​​of the objective function. - selecting one of the trained machine learning models based on an associated objective function value and applying it to the wafer imaging data set to detect anomalies; 4. A computer-implemented method comprising:

[0244] 80. The method of claim 79, wherein a machine learning model is trained when presented with a subset of an imaging dataset as input to compute an anomaly-free reconstruction of the subset, and anomalies in the subset are detected based on a comparison between the subset and the reconstructed subset.

[0245] 81. The method of claim 80, wherein the machine learning model includes an autoencoder.

[0246] 82. The method of any one of clauses 79 to 81, wherein at least one hyperparameter is related to the design of the machine learning model.

[0247] 83. The method of any one of clauses 79 to 82, wherein the objective function includes at least two different model evaluation measures.

[0248] 84. The method of claim 83, wherein one of the model evaluation measures includes a loss function used during training of the machine learning model.

[0249] 85. The method of claim 83 or 84, wherein one of the model evaluation measures includes an Lp norm loss function (p≧1).

[0250] 86. A method according to any one of clauses 83 to 85, wherein the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and one of the model evaluation measures comprises a discriminative loss function that assesses the difference between the expert annotations of anomalies and the detected anomalies.

[0251] 87. The method of claim 86, wherein the expert annotations include only a subset of the anomalies present in the imaging dataset.

[0252] 88. The method of any one of clauses 83 to 87, wherein one of the model evaluation measures includes a measure of the complexity of the machine learning model.

[0253] 89. The method of claim 88, wherein the measure of complexity of the machine learning model takes into account the total number of floating-point operations in one forward propagation of the machine learning model.

[0254] 90. A method according to any one of clauses 79 to 89, wherein the training data comprises expert annotations of anomalies for a subset of the imaging dataset, and the objective function comprises a weighted sum of an Lp-norm loss function (p ≧ 1), a discriminative loss function that evaluates the difference between the expert annotations of anomalies and the detected anomalies, and a measure of the complexity of the machine learning model.

[0255] 91. A method according to any one of clauses 79 to 90, wherein the objective function includes a quality value for evaluating the quality of the trained machine learning model, and the user interface is configured to present information about the trained machine learning model to a user and to have the user indicate the quality value.

[0256] 92. The method of any one of clauses 79 to 91, wherein the sampling strategy for selecting a plurality of hyperparameter values ​​includes taking into account hyperparameter values ​​and corresponding values ​​of the objective function from one or more prior iterations by optimizing one criterion selected from the group including expected improvement, maximum probability of improvement, and upper confidence interval.

[0257] 93. The method of any one of clauses 79 to 92, wherein the sampling strategy includes early stopping criteria.

[0258] 94. A method according to any one of clauses 79 to 93, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy.

[0259] 95. The method of claim 94, wherein the probability distribution is modeled based on previous usage based knowledge, preferably based on imaging hardware settings or design knowledge such as critical distances, critical dimensions, or pitch sizes.

[0260] 96. The method of any one of clauses 79 to 95, wherein the sampling strategy is different in at least two replicates.

[0261] 97. The method of any one of clauses 79 to 96, wherein the size of the generated subset of training data increases with the number of iterations.

[0262] 98. A computer-implemented method for detection of anomalies in an imaging data set of a wafer, the imaging data set including defects belonging to a plurality of defect classes, the method comprising: - generating an anomaly-detected image by applying an anomaly detection method to the imaging dataset; - performing one or more iterations, at least one of which includes the steps of: i. providing one or more samples of a distribution of values ​​of the anomaly-detection image for each defect class of a subset of defect classes; ii. calibrating the anomaly detection image by at least one calibration method comprising the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on one or more samples of the distribution of values ​​of the anomaly detection image; b. applying the trained machine learning model to the anomaly detection image to obtain a calibrated anomaly detection image; - applying a threshold to the calibrated anomaly detection image to detect anomalies, thereby reducing nuisances and highlighting defects in the anomaly detection image. 4. A computer-implemented method comprising:

[0263] 99. The method according to clause 98, wherein the anomaly detection image is a difference image between an imaging data set and a comparison data set, and the comparison data set is based on a die-to-die principle or a die-to-database principle.

[0264] 100. The method of clause 99, wherein the comparison dataset includes a reconstructed representation of the imaging dataset generated by training a machine learning autoencoder on the imaging dataset or a subset thereof and applying the trained autoencoder to the imaging dataset to obtain a reconstructed representation.

[0265] 101. The method of any one of clauses 98 to 100, wherein a machine learning model for anomaly localization is trained to optimize a loss function based on foreground samples and background samples.

[0266] 102. The method of clause 101, wherein the loss function is a semi-supervised loss function.

[0267] 103. The method of clause 101 or 102, wherein a sample from the distribution of anomaly detection image values ​​for each defect class of a subset of defect classes is used as a foreground sample or a background sample.

[0268] 104. The method of clause 103, wherein the samples are used as foreground samples and background samples are selected from the remaining pixels of the anomaly detection image.

[0269] 105. The method of claim 104, wherein the remaining pixels of the anomaly-detected image are selected as background samples if the anomaly-detected image value is below a threshold.

[0270] 106. The method according to clause 104 or 105, wherein each background sample is weighted by a weighting function w of its anomaly detection image value a, in particular by a negative exponential weighting function w(a)=exp(-a).

[0271] 107. A method according to any one of clauses 98 to 106, wherein a user interface is configured to allow a user to indicate the location of a small number of defects of each class of a subset of defect classes in the anomaly detection image, the user indicating the location of the defect by selecting a single pixel, preferably a pixel in a central region, of the defect, an annotation of the defect is automatically generated by selecting a region, preferably a circle, surrounding the selected single pixel, and a value of the anomaly detection image is sampled from the region surrounding the selected pixel.

[0272] 108. The method of claim 107, wherein the size of the region is selected based on a critical dimension and / or a critical distance and / or a pitch size.

[0273] 109. A method according to any one of clauses 79 to 108, wherein the imaging dataset includes one or more multisensory images.

[0274] 110. A method according to any one of clauses 79 to 109, wherein the imaging data set is a multi-beam SEM image.

[0275] 111. The method of any one of clauses 79-110, wherein the imaging data set is a focused ion beam SEM image.

[0276] 112. The method of any one of clauses 79 to 111, further comprising measuring one or more properties of the detected anomaly.

[0277] 113. The method of clause 112, further comprising controlling at least one wafer manufacturing process parameter based on the one or more measured properties.

[0278] 114. The method of clause 112, further comprising assessing a quality of the wafer based on the one or more measured properties and at least one quality assessment rule.

[0279] 115. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods described in any one of clauses 79 to 114.

[0280] 116. A system for controlling the quality of wafers produced in a semiconductor manufacturing facility, comprising: - an imaging device adapted to provide an imaging data set of said wafer; - one or more processing devices; - One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the methods of clause 114.

[0281] 117. A system for controlling production of wafers in a semiconductor manufacturing facility, comprising: - means for producing wafers controlled by at least one manufacturing process parameter; - an imaging device adapted to provide an imaging data set of the wafer; - one or more processing devices; - one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices to perform operations including the method of clause 113; Including, the system.

[0282] In summary, the following preferred features of the invention are inter alia: the invention relates to a computer-implemented method 10, 10' for detection of anomalies 15 comprising the steps of selecting an imaging data set 12 of a wafer 120 and hyperparameter values ​​defining a machine learning model 16 for anomaly detection, training and evaluating the machine learning model 16 by calculating an objective function value, and selecting one of the trained machine learning models and applying it to detect the anomaly 15. The invention also relates to a computer-implemented method 10'' for detection of anomalies 15 in an imaging data set 12 of a wafer 120 comprising the steps of providing a sample of a distribution of values ​​of the anomaly-detection images for each defect class, calibrating the anomaly-detection images 20, 26, 26' by training a machine learning model for anomaly localization, and applying a threshold to the calibrated anomaly-detection images 98, 108 to detect the anomaly 15. [Explanation of symbols]

[0283] 10,10',10'',10'''' Computer-Implemented Method 11 Defective cell structure 12 Imaging Datasets 14 Input 15 Abnormality 16 Machine Learning Models 18 Reconstruction 19 Data Selection Steps 20 Abnormal detection image 21 Training data generation step 22,22' Input image 23 Defects 24,24' Reconstructed image 25 Hyperparameter value selection step 26,26' Abnormal detection image 27 Training Steps 28,28' Abnormal detection image 29 Model Evaluation Steps 30,30' Input image 31 Model Selection Step 32,32' Reconstructed image 34 Flawless Data 36 Expert annotation 38 Expert Knowledge 40 Hyperparameter Range 42 Sampler 43 Hyperparameter Optimization Unit 44 Pruner 46 Objective Function 48 Hyperparameters 50 subsampled data sets 52 Complete Imaging Dataset 54 Optimization Model 56 Final Model 58 Good group 60 Poor Group 61 Anomaly detection image generation step 62 Good flock density 63 Sampling Steps 64 Poor Herd Density 65 Calibration Steps 66 Expected Improvements 67 Training Steps 68 Worst Attempt 69 Application Steps 70 Best Attempts 71 Thresholding Step 72 Abnormal detection image 73 iterations 74 Calibration Steps 76 Evaluation Steps 77 Affirmative Answer 78 End Step 79 Negative Answers 80 Calibration Method Fitting Steps 82 Threshold or filter adjustment steps 84 Annotation Adjustment Steps 86 Training Steps 88 iterations 90 Input image 92 Abnormal detection image 94 Click Points 95 Partially annotated anomaly detection images 96 Annotations 98 Calibrated anomaly detection image 100 Overlay 102 Reconstructed image 104 Input Image 106 Uncalibrated Anomaly Detection Images 108 Calibrated anomaly detection image 110 Uncalibrated Confusion Matrix 112 Calibrated confusion matrix 114,114' System 116 Imaging Devices 118 Processing Device 120 wafers 122 CPU 124 Interface 126 Memory 128 User Interface 130 means

Claims

1. A computer-implemented method (10, 10') for the detection of anomalies (15), comprising: - selecting an imaging data set (12) of a wafer (120); - generating training data from said imaging data set (12); repeating the following steps: i. selecting a hyperparameter value from a set of relevant hyperparameter values ​​based on a sampling strategy, said hyperparameter value corresponding to at least one hyperparameter (48) defining a machine learning model (16) for the detection of anomalies (15); ii. training the machine learning model (16) defined by the hyperparameters (48) based on the generated subset of training data; iii. Evaluating the trained machine learning model (16) by calculating an associated objective function value of an objective function (46). selecting one of the trained machine learning models based on the associated objective function value and applying it to the imaging data set (12) of the wafer (120) to detect anomalies (15); A computer-implemented method (10, 10') comprising:

2. 2. The method of claim 1 , wherein the machine learning model is trained to compute a reconstruction (18) of the imaging dataset (12) free of anomalies (15) when presented with a subset of the imaging dataset (12) as input (14), and the anomalies (15) in the subset are detected based on a comparison between the subset and the reconstructed subset.

3. The method of claim 2 , wherein the machine learning model (16) comprises an autoencoder.

4. The method of any one of claims 1 to 3, wherein at least one hyperparameter (48) is a design hyperparameter related to the design of the machine learning model (16).

5. The method of any one of claims 1 to 4, wherein all of the at least one hyperparameters (48) are design hyperparameters related to the design of the machine learning model.

6. The method of any one of claims 1 to 5, wherein the objective function (46) comprises at least two different model evaluation measures.

7. The method of claim 6 , wherein at least one of the model evaluation measures measures a quality of the anomaly detection.

8. The method of claim 6 or 7, wherein one of the model evaluation measures comprises a loss function used during training of the machine learning model (16).

9. The method of claim 8 , wherein one of the model evaluation measures is a loss function used during training of the machine learning model (16).

10. The method of any one of claims 6 to 9, wherein one of the model evaluation measures comprises an Lp-norm loss function, with p>1, of the training data samples and corresponding target data samples.

11. 11. The method of claim 7, wherein the training data comprises expert annotations (36) of anomalies (15) for a subset of the imaging dataset (12), and one of the model evaluation measures comprises a discriminative loss function that evaluates the difference between the expert annotations (36) of the anomalies (15) and the detected anomalies (15).

12. The method of claim 11 , wherein the expert annotation (36) assigns labels to only a subset of types of anomalies (15) present in the imaging dataset (12).

13. The method of any one of claims 7 to 12, wherein one of the model evaluation measures comprises a measure of the complexity of the machine learning model (16).

14. 14. The method of claim 1, wherein the training data comprises expert annotations (36) of anomalies (15) for a subset of the imaging dataset (12), and the objective function (46) comprises a weighted sum of an Lp-norm loss function (p≧1) of the training data samples and corresponding target data samples and a discriminative loss function that evaluates differences between the expert annotations (36) of the anomalies (15).

15. The method of claim 14 , wherein the objective function (46) further comprises a measure of complexity of the machine learning model (16).

16. 16. The method of claim 13 or 15, wherein the measure of the complexity of the machine learning model (16) takes into account a total number of floating point operations in one forward propagation of the machine learning model (16).

17. 17. The method of claim 1, wherein the objective function (46) includes a quality value for evaluating a quality of the trained machine learning model (16), and a user interface (128) is configured to present information about the trained machine learning model (16) to a user and to have the user indicate the quality value.

18. 18. The method of any one of claims 1 to 17, wherein the sampling strategy for selecting the plurality of hyperparameter values ​​comprises taking into account hyperparameter values ​​from one or more prior iterations and corresponding values ​​of the objective function (46) by optimizing a criterion selected from the group comprising expected improvement, maximum probability of improvement, an upper confidence interval.

19. The method of any one of claims 1 to 18, wherein the sampling strategy includes an early stopping criterion.

20. A method according to any preceding claim, wherein at least one set of hyperparameter values ​​is associated with a probability distribution indicating the likelihood of each hyperparameter value to be selected by the sampling strategy.

21. The method of claim 20 , wherein the probability distribution is modeled based on previous application-based knowledge of critical distances, critical dimensions, or pitch sizes.

22. The method of any one of claims 1 to 21, wherein the sampling strategy is different in at least two replicates.

23. The method of any one of claims 1 to 22, wherein the size of the generated subset of training data increases with the number of iterations.

24. 1. A computer-implemented method (10'', 10''') for detection of anomalies (15) in an imaging dataset (12) of a wafer (120), the imaging dataset (12) including defects (23) belonging to a plurality of defect classes, the method comprising: - applying an anomaly detection method to the imaging data set (12) to generate an anomaly detected image (20, 26, 26', 28, 28', 72, 92); - performing one or more iterations (88) comprising the steps of: i. providing one or more samples of a distribution of anomaly-detection image values ​​for each defect class of the subset of defect classes; ii. calibrating said anomaly detection image (20, 26, 26', 28, 28', 72, 92) by at least one calibration method comprising the steps of: a. training a machine learning model for anomaly localization, preferably anomaly segmentation, based on the one or more samples of the distribution of values ​​of the anomaly-detection image; b. applying the trained machine learning model to the anomaly-detection images (20, 26, 26', 28, 28', 72, 92) to obtain calibrated anomaly-detection images (98, 108); - applying a threshold to said calibrated anomaly-detection images (98, 108) to detect anomalies (15) to reduce nuisance and highlight defects (23) in said anomaly-detection images (20, 26, 26', 28, 28', 72, 92); A computer-implemented method (10'', 10''') comprising:

25. 25. The method of claim 24, wherein the anomaly detection image (20, 26, 26', 28, 28', 72, 92) is a difference image between the imaging data set (12) and a comparison data set, the comparison data set being based on the die-to-die principle or the die-to-database principle.

26. 26. The method of claim 25, wherein the comparison dataset comprises a reconstructed representation of the imaging dataset (12) generated by training a machine learning autoencoder on the imaging dataset (12) or a subset thereof and applying the trained autoencoder to the imaging dataset (12) to obtain a reconstructed representation.

27. The method of any one of claims 24 to 26, wherein the machine learning model for anomaly localization is trained to optimize a loss function based on anomalous and non-anomalous samples.

28. 28. The method of claim 27, wherein the loss function is a semi-supervised loss function.

29. 29. The method of claim 27 or 28, wherein the sample from the distribution of anomaly detection image values ​​for each defect class of the subset of defect classes is used as an anomaly sample.

30. 30. The method of claim 29, wherein the non-anomalous samples are selected from the remaining pixels of the anomaly detected image (20, 26, 26', 28, 28', 72, 92).

31. 31. The method of claim 30, wherein the remaining pixels of the anomaly detection image (20, 26, 26', 28, 28', 72, 92) are selected as non-anomalous samples if their anomaly detection image value is below a threshold.

32. Each abnormal sample is represented by the loss function, i In particular, the negative exponential weighting function w(a i ) = exp(-a i 32. The method of claim 30 or 31, wherein the weighting is done by

33. The method of any one of claims 27 to 32, wherein the loss function is a semi-supervised weighted cross-entropy loss function.

34. The loss function is [0010] where a is the abnormality detection image (20, 26, 26', 28, 28', 72, 92), y true Let y be a pixel-level annotation from the set {unannotated=0, anomaly=1}. pred 34. The method of claim 33, wherein: x denotes the label predicted by the machine learning model, w denotes a weighting coefficient, and i refers to a pixel index in the anomaly detection image (20, 26, 26', 28, 28', 72, 92).

35. The method of any one of claims 24 to 34, wherein a single threshold is applied to the calibrated anomaly detection image (98, 108) to detect anomalies (15).

36. 36. The method of claim 24, wherein a user interface (128) is configured to allow a user to indicate in the anomaly detection image (20, 26, 26', 28, 28', 72, 92) a location of a small number of defects (23) of each class of the subset of defect classes, the user indicating a location of the defect (23) by selecting a single pixel, preferably a pixel in a central region, of the defect (23), and an annotation (96) of the defect is automatically generated by selecting an area, preferably a circle, surrounding the selected single pixel, and a value of the anomaly detection image is sampled from the area surrounding the selected pixel.

37. 37. The method of claim 36, wherein the size of the regions is selected based on a critical dimension and / or a critical distance and / or a pitch size.

38. The method of any one of the preceding claims, wherein the imaging dataset (12) comprises one or more multisensory images.

39. The method of any one of the preceding claims, wherein the imaging data set (12) is a multi-beam SEM image.

40. The method of any one of the preceding claims, wherein the imaging data set (12) is a focused ion beam SEM image.

41. The method of any one of claims 1 to 40, further comprising measuring one or more properties of the detected anomaly (15).

42. 42. The method of claim 41, further comprising controlling at least one wafer manufacturing process parameter based on the one or more measured properties.

43. 43. The method of claim 42, further comprising assessing a quality of the wafer (120) based on the one or more measured properties and at least one quality assessment rule.

44. One or more machine-readable hardware storage devices containing instructions executable by one or more processing devices (118) to perform operations including the method of any one of claims 1-43.

45. A computer program product comprising instructions which, when executed by a computer, cause said computer to carry out the method according to any one of claims 1 to 43.

46. A system (114) for controlling the quality of wafers (120) produced in a semiconductor manufacturing facility, comprising: an imaging device (116) adapted to provide an image data set (12) of said wafer (120); one or more processing devices (118); one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices (118) to perform operations including the method of claim 43; A system (114).

47. A system (114') for controlling production of wafers (120) in a semiconductor manufacturing facility, comprising: means (130) for producing wafers (120) controlled by at least one manufacturing process parameter; an imaging device (116) adapted to provide an image data set (12) of said wafer (120); one or more processing devices (118); one or more machine-readable hardware storage devices containing instructions executable by one or more processing devices (118) to perform operations including the method of claim 42; The system (114') comprises:

Images