A collaborative learning model for semiconductor applications.

Abstract

Classifying wafers using collaborative learning. An initial wafer classification is determined by a rule-based model. A predicted wafer classification is determined by a machine learning model. Multiple users can manually review the classifications for confirmation or correction, or to add user classifications. All classifications are input to a machine learning model to continuously update its detection and classification scheme.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from U.S. Provisional Patent Application No. 62 / 914,901, entitled “Collaborative Learning for Semiconductor Applications,” filed October 14, 2019, which is incorporated by reference in its entirety.

[0002] This application relates to wafer classification in semiconductor manufacturing, and more particularly to a collaborative learning scheme for improving the classification process. [Background technology]

[0003] A typical semiconductor wafer manufacturing process may span many months and involve hundreds or even thousands of steps before the wafer becomes the final integrated circuit product produced by the process and is ready for packaging and shipment to the customer. Classification of wafers after manufacturing is important to assess the yield performance of the wafer manufacturing.

[0004] In one current approach, customers use various computer-generated outputs to determine wafer quality. For example, a graphic user interface (GUI) may be supported by a data template, such as the Exensio® analysis platform for semiconductor foundries commercially available from PDF Solutions, Inc. The template is configured to generate a GUI that contains wafer information, including results of wafer inspection, and displays it for user review. A typical wafer information display includes at least wafer identification information, wafer classification information, and a wafer map image.

[0005] In an example of a template for processing and presenting wafer information for a customer to review one or more wafer lots currently in the manufacturing process, a large number of rules (e.g., 200 or more) are typically generated. In particular, the recognition of clusters in the wafer map is the main purpose of the large number of rules. In one example, if there are clusters not captured in the 200 or more rules and deemed necessary, additional rules are generated to capture new cluster signatures and new rules are added to the existing rule set for predictions regarding future wafers. Additionally, upon review of the results and clustering information based on the rules, the customer can modify the wafer quality label.

[0006] In general, wafer classification may rely on various methods as input. For example, classification of baseline state, or excursion, or known spatial problem, or other general classification state may be deterministic based on various calculated rules and statistics, such as (i) automatic signature classification (ASC), where wafers are classified based on zone bin yield patterns according to multiple zone definitions and composite bin calculations, (ii) clustering, where wafers are classified based on clusters of die bin values, (iii) frequency selective surface (FSS), where pre-existing and / or user-defined rules are used to identify patterns, (iv) yield information, which is a statistical measure of the current wafer yield, such as statistical bin limits (SBL) and statistical yield limits (SYL).

[0007] The final wafer classification (or "unified classification" or "unified label") may be in the form of an output string that summarizes relevant information from the calculated classifications. However, determining the wafer classification using previous methods is not always sufficient, and the customer must manually review and potentially correct or update the wafer classification. This typically requires the customer to go through several iterations on each of the chip lines individually and manually adjust the rules and parameters. To improve the accuracy of the wafer classification, it is desirable to use manual review in combination with machine-based methods. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of a simplified graphic user interface (GUI). [Diagram 2] FIG. 1 is a block diagram illustrating one embodiment of a collaborative learning (CL) model for wafer classification. [Diagram 3] FIG. 1 is a block diagram showing an overview of a CL model for wafer classification. [Figure 4] FIG. 1 is a flow diagram illustrating one embodiment of a setup procedure for a CL model for wafer classification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Collaborative learning (CL) describes an implementation in the field of active learning (AL), which is more generally a branch of machine learning (ML). For example, AL is used by Internet websites to incorporate input from users (thumbs up, thumbs down) to determine product offerings or related market transactions. This approach can be incorporated during and after semiconductor manufacturing to improve current analysis-based approaches by correcting misclassification of die or wafer defect modes.

[0010] 1 is one embodiment, presented for illustrative purposes only, of a simplified graphic user interface (GUI) 100 that may be used to implement a collaborative learning environment for sorting wafers. GUI 100 is a processor-based tool that provides a visual display of information in a formatted manner, and, as is commonly known, has widgets of various designs to enable at least user interaction with the displayed information and provide control functions.

[0011] The processor may be desktop-based, i.e., stand-alone, or may be part of a network system, but when heavy information loads are processed and interactively displayed, the processor performance (CPU, RAM, etc.) must be state-of-the-art to maximize effectiveness. In a semiconductor fab environment, the Extensio® analytics platform is a useful option for building GUI templates. In one embodiment, the coding of the underlying processing routines may be performed using Spotfire® analytics software version 7.11 or later, which corresponds to the Python object-oriented programming language primarily used to code the machine learning models described below.

[0012] In the example of FIG. 1, GUI 100 includes two main windows or panels: a first window 110 for wafer information and a second window 140 for wafer maps, such as maps 141, 142.

[0013] In the first window 110, a template or spreadsheet 120 is presented that includes multiple columns 122, each displaying classification information for an in-process wafer. The columns provide the identity of the wafer being reviewed, as well as the associated classification data. Thus, column 124 identifies the wafer lot, column 125 identifies the particular wafer, column 126 identifies the current classification for the wafers in that column as determined by a first rule-based (RB) model, generally by heuristic and deterministic methods, column 127 identifies the classification as determined by a collaborative learning (CL) model, and column 128 identifies any user input modifications to the final classification.

[0014] One or more wafer maps, e.g., maps 141, 142, are displayed for the selected row in second window 140. For example, rows 122A and 122B are highlighted to indicate that they are selected, and thus corresponding sets of wafer maps 141A and 142A for the wafers identified in rows 122A and 122B, respectively, are currently displayed in second window 140.

[0015] One or more sets of user controls 160 are provided in a conventional manner as needed to navigate the display and select one or more items for review or modification, drill down into underlying data, etc. For example, panel 160 with selectable submenu options is implemented in this example as a pop-up window that is enabled when one or more rows are selected for review. Buttons, menus, and other widgets may be enabled in known manner to provide functionality related to the user controls and user interface, as described in more detail below.

[0016] Of course, the GUI may be formatted in many different ways, with more information items being presented or quickly accessible via the main GUI or sub-menus. For example, template 120 of FIG. 1 may be presented in the initial interface as a detailed spreadsheet with more wafer information columns that are considered key variables for a particular customer, and links to wafer maps and / or other wafer-related information.

[0017] The problem statement is straightforward: the model must classify the wafer, but must also provide a confidence level regarding the classification, even when the classification is "unknown" or "undetermined." As a conceptual example of collaborative learning applied to wafer classification, wafer classification can be determined by multiple methods, potentially resulting in different classifications for the same wafer problem. However, discrepancies between methods can be resolved by analytical review, including rule-based analysis, machine learning predictions, and manual review by a human user(s). The model for making such decisions is continually updated by learning from reviewing classification discrepancies and, as necessary, updating the various detection and classification schemes commensurate with the learnings.

[0018] 2 is a simplified block diagram of an example collaborative learning model 200 for wafer classification. In a first module 202, an initial classification is performed by one or more rule-based (RB) models, e.g., using ASC and / or other deterministic methods as described above. Such methods are known to provide appropriate classification for the majority (-95%) of wafers. Results from these one or more RB models can be provided to a GUI 206 for display and / or selection, as shown in column 126 of FIG. 1.

[0019] One or more machine learning (ML) models are used in module 204 to predict a classification based on the initial classification by module 202, such as schedule, interest features, etc., along with user feedback as described below. Results from the ML models may be provided directly to GUI 206 for individual display and / or selection. However, in one embodiment, the ML models take inputs including user feedback to create a classification based on a "collaboration" of various classification inputs with RB model analysis, ML model predictions, and multiple user feedback / corrections, resulting in a collaborative learning (CL) model determining the classification, as shown in column 127 of FIG. 1.

[0020] As shown, the results from the RB model 202 and the ML model 204 may be displayed in a GUI 206 for user review, where the user may select one or more wafers or lots to review and consider, and in some cases drill down into the data to try and understand unexplained anomalies or excursions before deciding on a final classification. The user may provide feedback to confirm the classification or input a different classification, as shown in column 128 of FIG. 2. However, the user is not limited to the classification provided by the RB model. For example, the user may create a new class that combines multiple RB classes into one or splits a single RB class into multiple RB classes. The GUI 206 may also provide the user with the ability to add comments, for example, to explain updates to the wafer classification. All classifications (rule-based, ML predicted, and user reclassification) and associated wafer information are stored in storage 210, which is accessible to a database 212 or portions thereof.

[0021] The stored classification information may be retrieved by a user in GUI 206, for example, to use for comparison in manual review of classification decisions. The classification information stored via database 212 is also used to train the ML models in module 211 such that the ML prediction models in module 204 are periodically updated.

[0022] Human interaction with the RB and ML models in the active learning mode allows users to continually improve and validate the classifications and build user confidence in the system. Also, as the dataset grows and more user reclassifications are provided, the collaborative learning environment described herein continues to improve the classification scheme. For example, existing rules can be modified and new rules can be added to the RB portion of the environment, and the ML model continues to improve the accuracy of its algorithms based on the inputs it receives, both in training and prediction, by being given sufficient examples of reviewer corrections and by evaluating different opinions by different human reviewers.

[0023] The combination of classification and active learning may be binary or multi-class. For example, the desired output is typically a classification label. In one use case, this may simply be implemented as a single column with the output strings of the associated clustering template. The initial labeling is typically based on the output by deterministic rules as described above. Some common labels may include bins, minutiae patterns, etc., but the customer dataset may, and often does, specify what the desired output is that can be predicted using the input dataset.

[0024] The input data set may consist of ASC data (which is typically very large), or other zone-based summaries, clustering, FSS, and yield per wafer, and possibly even larger dibin data. However, if ASC data is used for prediction, it must, at least initially, be recalculated on the fly from the raw dibin data during model training. The final data set may be determined based on model review and evaluation.

[0025] The following are reasonable assumptions made in creating a CL model: (1) there is enough data to train the model without user feedback, i.e., there are more sensor findings / test measurements than features for the majority of the dies in the wafer; (2) there are enough representative samples for each of the classes, e.g., conventional wisdom dictates that at a minimum there should be more than 30-50 findings per class or root cause, but this is not always valid. If there are insufficient findings available for classification and upsampling / downsampling is not enough to compensate, a classification of "unknown" is returned (meaning the wafer is abnormal but cannot be classified into a known category); (3) there is a sufficient amount of relabeled data; (4) there is good consistency between human labelers (a known problem in active learning); (5) the training data is a good representation of the wafer for prediction; (6) the required cases, i.e., the target features, are covered in the training set; (7) the output is class-specific data; (8) the input features are stable, i.e., there are no missing or new features.

[0026] A simple overview of a collaborative learning environment for wafer classification is shown in Figure 3. For example, typical process inputs 310 may include (1) a bin map, i.e., wafer (x,y) coordinates of chips and bins for chips. As an example, the bin map may be transformed using convolutions in a convolutional neural network; (2) a zone summary, i.e., for each zone (center, outer, top right, etc.), calculate the total number of wafers for each bin, which is transformed using a z-transform; and (3) other typical transformations, such as wavelets, 2D FFTs, etc., transformed on the 2D wafer bin map.

[0027] The target values ​​320 for the features of interest will naturally vary for different devices and processes, and ultimately rely heavily on the aggregation of data from the RB model and user input to determine and update the correct set of classification labels. User input 330, including confirmations, updates, and comments, allows the classification model to learn and grow. If there are conflicts in inputs from multiple users, the system always attempts to resolve the conflict, taking into account the timing of the inputs and other possible information, such as user group (administrators, subject matter experts, etc.).

[0028] The algorithmic element 350 may be defined by any typical classifier with or without hyper-parameter tuning, including, for example, K-nearest neighbors, robust logic regression with regularization, Naive Bayes, multi-layer perception and other neural networks, linear and non-linear SVMs, ensembles of decision trees, such as extra-trees, random forests, gradient boosting machines, xgboost, etc. These classifiers may be used collectively depending on the computation and accuracy requirements. Alternatively, the best performing algorithm may be selected based on metrics such as accuracy, F1 score, AUC, precision recall AUC, etc.

[0029] Results 360 continue to learn and update the classification scheme. As mentioned above, a GUI may be generated to implement a collaborative learning platform for wafer classification and should include functionality for a broad set of objectives.

[0030] The first set of functional aspects implemented in the GUI relates to reviewing templates and workflows. As mentioned above, templates may be user-interactive stand-alone templates or network-based with sufficient system and data protection. The user may be able to specify the scope of data to review in a data search window.

[0031] Users should be provided with the ability to review the original rule-based results, including but not limited to zone clusters, zone pattern classifications, statistical limits of system yield bins, integration rules, MRB templates, wafer maps, wafer information such as wafer ID, lot ID, and MRB-based (rule-based) predictions.

[0032] The model-based prediction decisions and model-based confidence scoring must be available for review, but if the model crashes or other errors occur during processing, an error message must be displayed.

[0033] The final wafer classification can be determined by an integrated labeling review. One option is to set the model-based prediction as the default state (once established). If the user has already reviewed the wafer classification, the results of the review are set as the default state for the final wafer classification.

[0034] The user should be given the ability to provide feedback via the classification GUI. For example, the user may be able to simply toggle between the final wafer-based integrated labels, e.g., by a drop-down list. A common set of available choices for the drop-down list includes selecting a string label output from the MRB. The user may be limited to selecting a label from an existing list rather than creating a custom label, but the existing list should include selectable choices, e.g., "unknown." In some cases, the user may also provide a new label. In this case, the GUI may provide similar classes to reduce the chance of creating redundant classes. The user should also have the ability to add comments, e.g., a comment section may include text explaining the user changes.

[0035] The model should provide the ability to not only record changes, but also to tie these changes to a particular user, e.g., by timestamp, username, unique key, etc. The model should also be able to create subsets of the data pulled into the template, primarily based on time. Data should preferably be restricted to data available in the template and / or similar data types that can be queried from the database.

[0036] The user must have the ability to extract input data from previous steps in a template or workflow, e.g., ASC data or other. If necessary, an intermediate solution extracts the input data via a portal programmed, e.g., with Python or R coding. This step may serve as a truth check, e.g., regarding consistency with previous classification decisions.

[0037] Models should have the ability to display summary model information, such as: (i) model name, (ii) model creation date / time, (iii) model location / link, (iv) error if the model was unable to make a prediction (e.g., due to missing features), (v) product specific information, and (vi) training performance and statistics.

[0038] The user should have the ability to invoke the model prediction and / or model training routines as part of a template or workflow. For example, a simple button or widget called "Run Manual Prediction" or the like may be configured in the GUI and / or a control may be provided to run predictions at user-defined intervals.

[0039] The next set of functional aspects implemented in the GUI are related to training and prediction performance. In one embodiment, this functionality may be provided in a separate panel or window. For example, the user must be able to create a classification model on demand. This refers to a basic model example constructed using, for example, Python or R coding resulting from model training. The user then also has the ability to call the method prediction from the model object.

[0040] A model object may be output, including all pre-processing information required to convert the appropriate features into a model object, and the classification model itself, for example for a new wafer. The model may be written to disk and to a server library. Alternatively, a path string may be returned to the template or workflow for use in future predictions. It is the default for the template to initially save and display the predictions in a default or user specified location. An authorization scheme may be implemented, for example, to write the model object to disk, along with returning updated classification labels and classification tables to the database and GUI.

[0041] Output files may be generated in a spreadsheet-friendly CSV format and may include additional columns such as model predictions and final labels with user input / relabeling. The database has the ability to store models, predictions, user modifications, etc. for easy retrieval by the user from a GUI template.

[0042] A typical process may result in the construction of, say, 5000 models, but only a small number of models, possibly only 10-20, that are truly key to improving the classification scheme.

[0043] If individual bin information is required, even for small chips, this can be a fairly heavy information load through Spotfire. For example, if a single wafer has 1000 chips, a 1000 wafer data set (reasonable for a manufacturing process) would result in 10 6If individual bins are not used, the resulting data may be significantly smaller, with a fraction of the remaining inputs per wafer.

[0044] Other features may be considered and integrated into the example templates depending entirely on customer demand. For example, collaborative classification methods may be automated into the process workflow. For displayed results, wafers with the lowest model confidence may be displayed at the top for user review. Drill-down capabilities may be implemented for each wafer to review, for example, WEH, metrology, defects, indicators, PCM, etc.

[0045] In particular, root cause analysis is and should be an important part of classification review and updating. For example, in FIG. 2, known root cause information may be stored in root cause storage 214 and also associated or linked to specific excursions or defects in database 212. The root cause information may be made available to the user in GUI 206 as part of the user's manual classification review, e.g., as drill-down information; similarly, user feedback may be provided to and incorporated into root cause learning.

[0046] Regarding interface specifications, in general, a collaborative learning template can pull data from two sources: (i) a database containing labels tuned through training and updating ML models, which have been updated by previous workflows, and (ii) a text file containing previously relabeled results.

[0047] The re-labeling result preferably includes user-defined information including: (i) wafer ID, (ii) test program, (iii) test version, (iv) tester ID, (v) insertion date, (vi) update date, (vii) user ID, and (viii) user comments. Other types of labels, such as process module, tool ID, and chamber ID, may be useful, for example, for drill-down analysis.

[0048] For each wafer, the collaborative learning template may display information about each wafer, including: (i) wafer ID, (ii) signature type (ASC or cluster drawn from the previous MRB template), (iii) signature name (from the previous MRB template), (iv) test version, (v) tester ID, (vi) insertion date, (vii) update date, (viii) user ID, and (ix) user comments.

[0049] In one embodiment, the interface may be configured to allow the user to quickly toggle between key information from pull-down menus or the like, rather than displaying all information for each wafer directly in the primary window, such as: (i) Lot ID, (ii) Test Program, and (iii) Test Version.

[0050] A user can generally manually enter comments for each wafer, and the interface may be configured with drill-down capabilities for each wafer, such as (i) WEH, metrology, defects, indicators, PCM, etc., and (ii) nearest neighbor evidence of which neighboring wafers are similar. As the user interacts with the template, a button or similar widget, such as "Apply Changes," may be selected to append or overwrite data (e.g., based on wafer ID) in a relabeled results file, which may be stored in a database with other wafer data.

[0051] The example Collaborative Learning (CL) template can be installed and configured with the following post-setup capabilities for use during manufacturing: For incoming wafer(s) labeled as anomaly, this capability can include internally comparing the new wafer's history and data with similarly acquired data of existing wafers. Since the comparison must be specific to the product stream, the interface must include the ability to determine the exact relevant data to search based on product ID. For example, wafers need to be labeled with the most likely test programs and test versions, or lack thereof, associated with the same or similar bin patterns.

[0052] Of course, the basic wafer information is displayed and preferably the user is provided with drill down capabilities to analyze the anomaly neighborhood of the wafer, including access to die maps, clusters, zone statistics, comments provided by other users, etc. The user should be able to override and update key relevant process modules and process steps and save the updates for use in comparison on further wafer(s).

[0053] During the CL template setup phase, process engineers should be provided with the ability to (i) load an existing set of wafer data via a workflow, (ii) automatically or manually identify boundary conditions / parameter cutoffs for this product stream, (iii) review and check the results for consistency (sanity check), and (iv) save the wafer data and boundary conditions for later retrieval and reading.

[0054] 4, with more detailed explanation below, is a simplified diagram of a setup configuration workflow 400. In step 402, preferably by an automated method, an existing or previous set of wafer data is loaded as input to the model.

[0055] The inputs preferably include, for each wafer (with wafer ID), (i) bin data by die, andx and Die y (ii) ASC output including zonal bin deltas from zonal medians for each bin type (number), and general pattern labels (by classification) and detailed patterns (by classification); and (iii) the presence of each statistically significant cluster is reported using the following information: (a) cluster median x Die y (b) the bounding box of the cluster, and (c) all the Dies in the cluster. x Die y The clustering output includes a clustering output that may be labeled with the process module, tool ID, and chamber ID. Other inputs may include the process history of each wafer from the Wafer Equipment History (WEH), including process module, tool ID, and chamber ID. If the process is dynamic, load the process history and tool for a given wafer ID and product. If the process is static, i.e., has a dedicated process path, load the process history and tool for the product.

[0056] One particular input desired for collaborative learning is the clustering output. In step 404, parameter cutoffs for clustering are identified for this product stream, either automatically via a Monte Carlo routine or the like, or manually by a user. For example, the following basic parameters are defined:

[0057] d c = a fixed distance close enough to the selected wafers to be considered identical, with only differences due to random noise, d f = fixed distance that is considered far enough away to denote no similarity in the bin clusters and training set. Any overlap should be considered merely coincidence.

[0058] k = the number of neighbors considered by the KNN algorithm.

[0059] D=Euclidean, Minkowski, Mahalanobis, L1, L ∞ , and metrics used to calculate distance, which may include similar solutions.

[0060] In step 406, there is an option for manual setup, but typically an automated setup is performed, starting in step 408, where a statistical simulation is performed to determine the limit. df and dc are functions of the wafer map and defect density. The algorithm is typically used to cluster the recognition, and can be modified to use Monte Carlo simulation with a variable amount of defects and fit a response surface to the simulation results to estimate these additional parameters.

[0061] In step 410, hyperparameter tuning is performed to ensure that the results do not overfit the data. For example, n-fold cross-validation can be used with labels to identify the best hyperparameters to fit the selected list of modes in the dataset.

[0062] If manual setup is selected in step 406, an expert user, such as a skilled process engineer, may manually specify variables as necessary in step 412, and hyperparameter tuning is performed in step 410A as per step 410.

[0063] Step 414 provides a way for the user to review the results and perform a sanity check. By way of example, the user may review the distance metrics d for various techniques, such as Zonal KNN, or Clustering KNN, or Spatial KNN, etc. c and d f , as well as the distribution from the Monte Carlo simulation. The user can also review all the bad wafers, correction labels, and recommended labels for each step.

[0064] Finally, in step 416, the wafer data and boundary conditions, including the necessary parameters, combinations, and KNN data, may be stored for later retrieval and review for prediction. In this manner, the model for collaborative learning may be used to resolve differences in wafer classification across various methodologies. Once resolved, the various rule-based and machine learning-based models may be updated and continually retrained with reclassification data to improve the effectiveness of the classification scheme.

Claims

1. 1. A method for sorting wafers in a semiconductor manufacturing process, comprising: receiving first wafer information representative of a first wafer at a selected step of the semiconductor manufacturing process; determining an initial classification of the first wafer according to a rule-based model based on the first wafer information; determining a predicted classification of the first wafer according to a machine learning model configured to predictively determine a wafer classification based on the initial classification and user input; providing a display of the first wafer information to the user, the display including the initial classification and the predicted classification, the display also having a plurality of user interactive elements including a first user interactive element for selecting and updating the initial classification or the predicted classification and a second user interactive element for inputting a user classification; receiving the user input from a first user interactive element or a second user interactive element of the display to establish a final classification; saving the initial classification, the predicted classification, and the final classification for the first wafer in storage for use in classifying a next wafer; A method comprising:

2. retrieving from storage the initial classification, the predicted classification, and the final classification for the first wafer; training the machine learning model based on the initial classification, the predicted classification, and the final classification for a plurality of wafers; The method of claim 1 further comprising:

3. receiving the initial classification or the predicted classification or both as the final classification as the user input; The method of claim 1 further comprising:

4. receiving as the user input a modification of the initial classification or the predicted classification as the final classification; The method of claim 1 further comprising:

5. receiving a user classification as the final classification as the user input; The method of claim 1 further comprising:

6. Enabling a formatted user display that lists the initial classification of the rule-based model, the predicted classification of the machine learning model, and the user input. The method of claim 1 further comprising:

7. Enabling a panel having selectable submenu options for drilling down into multiple details of the first wafer information upon user selection of the first wafer via a simplified graphic user interface (GUI). The method of claim 1 further comprising:

Images