Collaborative Learning Model for Semiconductor Applications

Abstract

Classify wafers using collaborative learning. Determine an initial wafer classification through a rule-based model. Determine a predicted wafer classification through a machine learning model. Multiple users can manually view the classification to confirm or modify it, or add a user classification. Input all the classifications into the machine learning model to continuously update its detection and classification scheme.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 914,901, filed October 14, 2019, entitled "Collaborative Learning for Semiconductor Applications", which is hereby incorporated by reference in its entirety. Technical Field

[0003] This application relates to wafer classification in semiconductor manufacturing processes, and more particularly, to collaborative learning schemes for improving classification processes. Background Art

[0004] Typical processes for manufacturing semiconductor wafers undergo hundreds or even thousands of steps over a period of several months, after which the wafers are transformed into final integrated circuit products produced by the process and are ready for packaging and shipment to customers. Classification of wafers after manufacturing is very important for evaluating wafer manufacturing yield performance.

[0005] In one current scenario, customers utilize various computer-generated outputs to determine wafer quality. For example, a graphical user interface (GUI) can be supported by a data template, such as an analysis platform for semiconductor foundries sold by PDF Solutions, Inc. The template is configured such that the GUI is generated to contain and display wafer information for the user to view, the wafer information including wafer inspection results. A typical wafer information display will include at least wafer identification information, wafer classification information, and an image of the wafer map.

[0006] Typically, a large number of rules (e.g., 200+) are created in a template instance for processing and presenting wafer information for customers to view one or more batches of wafers currently in a manufacturing process step. Specifically, identifying clusters in the wafer map is the main objective of many rules. In one example, if any clusters are not captured by the 200+ rules and it is deemed necessary, additional rules are created to capture the new cluster signature, and the new rules are added to the existing rule set for predicting future wafers. Additionally, after viewing the rule-based results and clustering information, the customer can modify the wafer quality label.

[0007] Automatic Signature Classification (ASC) Generally, wafer classification can depend on various methods as inputs. For example, classification of a baseline state, or an offset, or a known spatial problem or other typical classification states can be determined based on various calculation rules and statistical data, such as: (i) Automatic Signature Classification (ASC) : Classifying wafers based on a partition bin yield pattern according to multiple partition definitions and composite bin calculations; (ii)Clustering : Classify wafers based on die bin values; (iii) Frequency Selective Surface (FSS) : Use pre - existing and / or user - defined rules to identify patterns; (iv) Yield Information : Statistical metrics of the current wafer yield, such as Statistical Bin Limit (SBL) and Statistical Yield Limit (SYL).

[0008] The final wafer classification (or "merged classification" or "merged label") can take the form of an output string that encapsulates relevant information in the calculated classification. However, determining wafer classification using previous methods is not always sufficient because the customer has to manually review and sometimes correct or update the wafer classification. This typically requires the customer to manually adjust rules and parameters for each chip line separately in multiple iterations. There is a desire to combine the manual review with a machine - based scheme to improve the accuracy of wafer classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram of a simplified Graphical User Interface (GUI).

[0010] Figure 2 is a block diagram showing one implementation of a Collaborative Learning (CL) model for wafer classification.

[0011] Figure 3 is a block diagram showing a conceptual overview of the CL model for wafer classification.

[0012] Figure 4 is a flowchart showing one implementation of the setup procedure of the CL model for wafer classification. DETAILED DESCRIPTION

[0013] Collaborative Learning (CL) describes an implementation in the field of Active Learning (AL), which is a more general field of Machine Learning (ML). For example, Internet websites use AL to integrate user input (thumbs up, thumbs down) to decide on product offerings or related marketing. This method can be used in semiconductor manufacturing and post - manufacturing to enhance current analysis - based methods by correcting misclassifications of die or wafer failure modes.

[0014] Figure 1 is one implementation of a simplified Graphical User Interface (GUI) 100 that can be used to implement a collaborative learning environment for wafer classification, presented for illustrative purposes only. GUI 100 is a processor - based tool that provides a visual display of information in a formatted manner, with widgets of various designs for enabling user interaction with at least the displayed information and for providing control functions, all of which are well - known.

[0015] The processor can be desktop-based, i.e., stand-alone, or part of a network system; however, considering the large amount of information to be processed and interactively displayed, the processor capabilities (CPU, RAM, etc.) should be state-of-the-art at present to maximize efficiency. In a semiconductor foundry environment, The analysis platform is a useful option for building GUI templates. In one embodiment, an analysis software version 7.11 or higher that is compatible with the Python object-oriented programming language can be used to complete the coding of the underlying processing routines, mainly for coding the machine language model described below.

[0016] In Figure 1 the example of, GUI 100 includes two main windows or panels: a first window 110 for wafer information and a second window 140 for wafer maps, such as FIGS. 141, 142.

[0017] In the first window 110, a template or spreadsheet 120 is presented, which includes a plurality of rows 122, and each row displays classification information of the wafers in the process. The columns provide the identification of the wafers being viewed and the associated classification data. Thus, column 124 identifies the wafer lot, while column 125 identifies a specific wafer; column 126 identifies the current classification of the wafer in the row, and the classification is typically determined by a model based on a first rule (RB) through heuristic and deterministic methods; column 127 identifies the classification determined by a collaborative learning (CL) model, and column 128 identifies the modification (if any) of the user input to the final classification.

[0018] In the second window 140, one or more wafer maps are displayed for the selected rows, such as FIGS. 141, 142. For example, rows 122A and 122B are highlighted to indicate that they have been selected, and thus a set of corresponding wafer maps 141A and 142A of the wafers identified in rows 122A and 122B, respectively, are simultaneously displayed in the second window 140.

[0019] A set or multiple sets of user controls 160 are provided in a conventional manner as needed to navigate in the display, select one or more items for viewing, modification, or drilling down into the underlying data, etc. For example, in this example, the panel 160 is implemented as a pop-up window with selectable sub-menu options, and the pop-up window is enabled when one or more wafer rows are selected for viewing. Buttons, menus, and other widgets can be enabled in a well-known manner to provide functions for user control and user interaction, as described in more detail below.

[0020] Of course, the GUI can be formatted in many different ways and have more information items presented or quickly accessible through the main GUI or sub-menus. For example, Figure 1The template 120 in [the document] can be presented as a detailed spreadsheet in the main interface, and the spreadsheet has more column wafer information that is considered to be key variables of a specific customer, as well as links to wafer maps and / or other wafer-related information.

[0021] The problem statement is simple and clear - the model needs to classify wafers, but should also provide a confidence level for the classification, even if the classification is "unknown" or "uncertain". As a conceptual example of collaborative learning applied to wafer classification, wafer classification can be determined by multiple methods, which may lead to different classifications for the same wafer problem. However, the differences between the methods can be resolved through analytical reviews, including rule-based analysis, machine learning predictions, and manual reviews by human users. Utilize the learning obtained from reviewing classification differences and continuously update the model that makes such determinations by updating various detection and classification schemes as necessary based on the learning.

[0022] Figure 2 is a simplified block diagram of an example of a collaborative learning model 200 for wafer classification. In the first module 202, initial classification is performed by one or more rule-based (RB) models, such as using ASC and / or other deterministic methods as described above. It is known that such methods can provide correct classifications for most wafers (about 95%). The results from these one or more RB models can be provided to the GUI 206 for display and / or selection, as Figure 1 shown in column 126 of [the document].

[0023] One or more machine learning (ML) models are used in module 204 to predict classifications, for example, based on the initial classification from module 202, schedules, features of interest, etc., and user feedback as described below. The results from the ML models can be directly provided to the GUI 206 for separate display and / or selection. However, in one implementation, the ML models incorporate inputs including user feedback to create a classification based on the "collaboration" of different classification inputs from RB model analysis, ML model predictions, and multiple user feedback / corrections, and thus the collaborative learning (CL) model determines the classification, as Figure 1 shown in column 127 of [the document].

[0024] As indicated, the results from the RB model 202 and the ML model 204 can be displayed in the GUI 206 for the user to view, and the user can select one or more wafers or batches for viewing and analysis, and may drill down into the data to try to better understand any unexplained anomalies or offsets before making a final classification. The user can provide feedback to confirm the classification or enter a different classification, as Figure 2As shown in column 128. It should be noted that the user is not limited to the classifications provided by the RB model. For example, the user can create new classes that combine multiple RB classes or divide a single RB class into multiple RB classes. The GUI 206 can also provide the user with the ability to add annotations, for example, to explain updates to the wafer classification. All classifications (rule-based, ML predictions, and user reclassifications) and related wafer information are saved to a storage device 210 that can be accessed by or is part of the database 212.

[0025] The user can retrieve the stored classification information in the GUI 206, for example, for comparison when manually reviewing classification determinations. The classification information stored via the database 212 is also used to train the ML model in the module 211 so as to periodically update the ML prediction model in the module 204.

[0026] The human-machine interaction with the RB model and the ML model in the active learning mode allows the user to continuously improve and validate the classification, thus building the user's confidence in the system. In addition, as the dataset grows and more user reclassifications are provided, the collaborative learning environment described here will continue to improve the classification scheme. For example, existing rules can be modified or new rules can be added in the RB part of the environment; and the ML model will continue to improve the algorithm in terms of training and prediction based on the received input, because different opinions of different human reviewers can be evaluated even if there are enough reviewers to modify the instances.

[0027] The combination of classification and active learning can be binary or multi-class. For example, the required output is usually a classification label. In one use case, this can be simply implemented as a single column with an output string in the relevant cluster template. As described above, the initial tagging is usually based on the output of deterministic rules. Some common labels may include bins, detail patterns, etc., but the customer dataset may and usually does specify the desired output that can be predicted using the input dataset.

[0028] The input dataset can consist of ASC data (usually very large) or other partition-based generalizations, clustering, FSS, and per-wafer yield, as well as possibly die bin data (even larger). However, if ASC data is used in the prediction, then at least initially, the ASC data should be dynamically recalculated from the original die bin data during model training. The final dataset can be determined based on model viewing and evaluation.

[0029] The following are reasonable assumptions made when creating a CL model: (1) There is sufficient data to train the model without user feedback, i.e., the sensor observations and test measurements of most die within a wafer are more than the features; (2) There are sufficient representative samples for each class; for example, a classic rule of thumb is that there should be at least more than 30 to 50 observations for each class or root cause; however, this is not always true. If there are not enough observations available for classification and upsampling / downsampling is not sufficient to compensate, "unknown" is returned as the classification (which means the wafer is abnormal but cannot be classified into a known class); (3) There is a sufficient number of relabeled data; (4) There is good agreement among human labelers (a known problem in active learning); (5) The training data is a good representation of the wafers for prediction; (6) The required cases, i.e., the target features, are covered in the training set; (7) The output is classification data; and (8) The input features are stable, i.e., there are no missing or new features.

[0030] Figure 3 A simple conceptual overview of a collaborative learning environment for wafer classification is shown. For example, typical process inputs 310 can include: (1) BinMap, i.e., the wafer (x, y) coordinates of the chips and the bins of the chips. For example, convolution in a convolutional neural network can be used to transform the BinMap; (2) Partition summary, i.e., for each partition (center, outer, upper right, etc.), the wafer count of each bin that has been transformed using the z-transform is calculated; and (3) Other typical transforms transformed on the 2D Wafer BinMap, such as Wavelet, 2D FFT.

[0031] The target values 320 of the features of interest will of course be different for different devices and processes and will ultimately depend heavily on the aggregation of data from the RB model and user input to determine and update the correct set of classification labels. User input 330, including confirmation, update, and comments, allows the classification model to learn and evolve. Whenever there are conflicting inputs from multiple users, the system will attempt to resolve the conflict by considering the timing of the inputs and other possible information such as the user group (manager, subject matter expert, etc.).

[0032] The algorithm element 350 can be defined by any typical classifier with or without hyperparameter tuning, including for example K-nearest neighbor, robust logistic regression with regularization, naive Bayes, multi-layer perceptron and other neural networks, linear and non-linear SVM, decision tree ensembles such as ExtraTree, random forest, gradient boosting machine, xgboost, etc. These classifiers can be used together depending on the computational and accuracy requirements. Alternatively, the algorithm with the best performance can be selected based on metrics such as accuracy, F1-score, AUC, precision-recall AUC, etc.

[0033] The result 360 is a continuously learning and updated classification scheme. As mentioned before, a GUI can be generated to implement a collaborative learning platform for wafer classification, and the GUI should include functions for a wide range of objectives.

[0034] The first set of functional aspects to be implemented in the GUI relates to the viewing of templates and workflows. As mentioned above, the template can be a stand-alone template for user interaction or a web-based template with sufficient system and data protection. The user can specify the data range to be viewed in the data retrieval window.

[0035] The user should be provided with the possibility to view the following: results based on original rules, including but not limited to: partition clustering, partition pattern classification, system yield bin statistical limits, merge rules; MRB templates; wafer maps; wafer information, such as wafer ID, lot ID; and MRB (rule-based) predictions.

[0036] Model-based prediction decisions and model-based confidence scores should also be viewable; however, if the model crashes or encounters other errors during processing, an error message should be displayed.

[0037] The final wafer classification can be determined by viewing the unified label. One option is to set the model-based prediction (once established) as the default. If the user has viewed the wafer classification, the viewed result is set as the default for the final wafer classification.

[0038] The user must be able to provide feedback through the classification GUI. For example, the user can simply switch the final wafer-based merge label via a drop-down list. A typical set of available options for the drop-down list includes selecting string labels output from the MRB. The user may be restricted to selecting labels from the existing list rather than creating custom labels, but the existing list should include selectable options such as "unknown" or similar. For some cases, the user is also allowed to provide new labels. In this case, the GUI can provide similar classes to reduce the chance of creating redundant classes. The user should also be able to add comments. For example, the comment section can include text explaining the user's changes.

[0039] The model should be able to record changes but also be able to connect these changes to a specific user, for example, via a timestamp, username, unique key, etc. The model should also be able to create a subset of the data pulled into the template mainly based on time. It may be necessary to limit the data to the data available within the template and / or similar data types queryable from the database.

[0040] Users should be able to extract input data from previous templates or steps in the workflow, such as ASC data or other data. If necessary, an intermediate solution is to extract the input data programmatically through a programming portal, for example, using Python or R coding. For example, this step can be used as a reality check for consistency with previous classification decisions.

[0041] The model should be able to display summary model information, such as: (i) the model name; (ii) the date / time the model was created; (iii) the model location / link; (iv) errors, if the model is unable to make predictions (due to missing features, etc.); (v) product-specific information; and (vi) training performance and statistics.

[0042] As part of a template or workflow, users should be able to call model prediction routines and / or model training routines. For example, a simple button or widget named "Run Manual Prediction" or a similarly named one can be configured within the GUI, and / or controls can be provided to run predictions at user-defined intervals.

[0043] The next set of functions to be implemented in the GUI relate to training and prediction capabilities. In one implementation, this functionality can be provided in a separate panel or window. For example, users should be able to create a classification model upon request. This refers to a basic model instance, for example, generated from model training, configured using Python or R coding. Then, the user should also be able to call the method predict from the model object.

[0044] The model object can be output, including all the preprocessing information required to transform the features suitable for the model object, as well as the classification model itself for predicting new wafers. The model can be written to disk and the server library. As an alternative, a path string can be returned to the template or workflow for future predictions. By default, the template will initially save and display the default location for predictions or the user-specified location. For example, a permissions scheme can be implemented to write the model object to disk and return the updated classification labels and tables to the database and the GUI.

[0045] The output file can be generated in a CSV format suitable for spreadsheets and may include, for example, additional columns for model predictions, as well as the final labels with user input / relabeling. The database will be able to save models, predictions, user modifications, etc., for easy retrieval by the user from the GUI template.

[0046] For example, a typical process may result in building 5000 models, but only a small fraction (say, 10 to 20) of the models will truly be key to improving the classification scheme.

[0047] If individual bin information is required, this can result in a rather large information payload via Spotfire even for small chips. For example, if there are approximately 1000 chips per wafer, a dataset of 1000 wafers (reasonable for a production process) will result in 10 6 observations. If individual bins are not used, the data generated can be much smaller, and some of the remaining inputs are wafer-based.

[0048] Other features can be considered depending entirely on customer needs and integrated into the template instance. For example, collaborative classification methods can be automated into the process workflow. For the displayed results, the wafer has the lowest model confidence, and the model confidence can be shown at the top to ensure the user views it. In-depth functionality can be implemented for each wafer to, for example, view WEH, metrology, defects, indicators, PCM, etc.

[0049] Specifically, root cause analysis can and should be an important part of classification viewing and updating. For example, in Figure 2 , the known root cause information can be stored in the root cause storage device 214 and also associated or linked with a specific offset or defect in the database 212. In the GUI 206, the root cause information can also be provided to the user as part of the user's manual classification viewing (e.g., as in-depth information). Similarly, user feedback can be provided and incorporated into root cause learning.

[0050] Regarding the interface specification, generally speaking, the collaborative learning template can extract data from two sources: (i) the database, which has been updated through previous workflows, including adjusting labels through training and updating the ML model; and (ii) the text file, which contains the results of previous relabeling.

[0051] The relabeling results will preferably contain user-defined information, including: (i) wafer ID; (ii) test program; (iii) test version; (iv) tester ID; (v) insertion date; (vi) update date; (vii) user ID; and (viii) user comments. Other types of labels can be used for in-depth analysis, such as process module, tool ID, and chamber ID.

[0052] For each wafer, the collaborative learning template can display information for each wafer, including: (i) wafer ID; (ii) signature type (ASC or cluster, extracted from the previous MRB template); (iii) signature name (from the previous MRB template); (iv) test version; (v) tester ID; (vi) insertion date; (vii) update date; (viii) user ID; and (ix) user comments.

[0053] In one embodiment, the interface can be configured to allow the user to quickly switch key information from a drop-down menu or equivalent, rather than directly display all information in the main window, such as the following items for each wafer: (i) batch ID; (ii) test program; and (iii) test version.

[0054] Typically, the user is allowed to manually enter comments for each wafer, and the interface can be configured with in-depth functions for each wafer, such as: (i) WEH, metrology, defects, indicators, PCM, etc.; and (ii) evidence of the nearest neighbor scheme - which adjacent wafers are similar. When the user finishes interacting with the template, a button such as "Apply Changes" or a similar widget can be selected to append or overwrite the data in the relabeled result file (e.g., based on the wafer ID) and store it in the database along with other wafer data.

[0055] Instances of the collaborative learning (CL) template can be installed and configured with the following post-setup functions for use in production. For incoming wafers marked as anomalies, the functions can include internally comparing the history and data of the new wafer with similar retrieved data of existing wafers. The comparison must be product flow specific, so the interface needs to include the ability to determine the correct associated data to retrieve based on the product ID. For example, the wafer should be marked with the most likely test program and test version associated with the same or similar bin pattern, or without said test program and test version.

[0056] Of course, the basic wafer information will be displayed, and preferably, the user is provided with in-depth functions to analyze the anomalous neighbors of the wafer, including access to die maps, clusters, partition statistics, other user-provided comments, etc. The user should be allowed to override and update the main associated process modules and process steps and save the updated results for future wafer comparisons.

[0057] During the setup phase of the CL template, the process engineer should have the following functions: (i) load a set of existing wafer data through the workflow; (ii) automatically or manually identify the boundary conditions / parameter cutoffs for this product flow; (iii) view and check the consistency of the results (integrity check); and (iv) save the wafer data and boundary conditions for later retrieval and reading.

[0058] Figure 4 FIG. 400 is a simplified illustration of the workflow for setting specifications, which is described in more detail below. At step 402, a set of existing or previous wafer data is preferably loaded as input into the model by an automated method.

[0059] For each wafer (with a wafer ID), the input preferably includes: (i) bin data per die, and Die x and Die yCoordinates; (ii) ASC output, including the bin increment of the partition mid-zone for each bin type (numeric), and the general mode label (categorical) and detailed mode (categorical); and (iii) clustering output, where the presence of each statistically significant cluster can be labeled with the following information: (a) the center Die of the cluster x 、Die y location (numeric); (b) the bounding box of the cluster; and (c) all Dies within the cluster x 、Die y locations. Other inputs can include the process history of each wafer in the wafer equipment history (WEH), including process modules, tool IDs, and chamber IDs. If the process is dynamic, the process and tool history for a given wafer ID and product are loaded. If the process is static, i.e., has a dedicated process path, the process and tool history for the product are loaded.

[0060] One specific input required for collaborative learning is to cluster the output. At step 404, the clustering parameter cut-off value for this product flow is automatically or manually identified by the user through a Monte Carlo routine or a similar routine. For example, the following basic parameters are defined:

[0061] d c = a fixed distance close enough to the selected wafer to be considered the same, where any differences are attributed to random noise; d f = a fixed distance considered far enough to have no similarity in terms of bin clusters and the training set. Any overlap should be considered only random.

[0062] k = the number of neighbors considered by the KNN algorithm; and

[0063] D = the metric used to calculate the distance, which may include Euclidean, Minkowski, Mahalanobis, L1, L ∞ and similar solutions.

[0064] At step 406, there is an option for manual setting, but usually automatic setting starts from step 408, where a statistical simulation is run to find the limit values. d f and d c are functions of the wafer map and the defect density. Usually, one algorithm is used for cluster identification and the algorithm can be modified to use Monte Carlo simulations with different numbers of defects and fit a response surface to the simulation results to estimate these additional parameters.

[0065] In step 410, hyperparameter tuning is performed to ensure that the results do not overfit the data. For example, n-fold cross-validation can be used with the labels to identify the best hyperparameters for the selected model list in the fitted dataset.

[0066] If manual setting is selected in step 406, an expert user, such as an experienced process engineer, can manually specify variables as needed in step 412 and perform hyperparameter tuning in step 410A as in step 410.

[0067] Step 414 provides a way for the user to view the results and perform an integrity check on the results. For example, the user can view the distance metrics d c and d f , such as for partition KNN, or clustering KNN or spatial KNN and the distribution of Monte Carlo simulations. The user can also view all the anomalous wafers, corrected labels, and recommended labels for each step.

[0068] Finally, in step 416, the wafer data and boundary conditions can be saved for later retrieval and reading for prediction, including the required parameters, combinations, and KNN data. Thus, the collaborative learning model can be used to resolve differences between different methods in wafer classification. Once resolved, various rule-based and machine learning-based models can be updated and the models continuously retrained using the reclassified data to enhance the effectiveness of the classification scheme.

Claims

1. A method for classifying wafers in a semiconductor manufacturing process, comprising: Receiving first wafer information representing a first wafer at a selected step of the semiconductor manufacturing process; Based on the first wafer information, determining an initial classification of the first wafer according to a rule-based model; Determining a predicted classification of the first wafer according to a machine learning model, the machine learning model being configured to determine a predicted wafer classification based on the initial classification and user input; Providing a display to the user including the first wafer information with the initial classification and the predicted classification, the display further having a plurality of user interaction elements, including a first user interaction element for selecting and updating the initial classification or the predicted classification and a second user interaction element for inputting a user classification; Receiving the user input from the first user interaction element or the second user interaction element of the display to establish a final classification; And Saving the initial classification, the predicted classification, and the final classification of the first wafer to a storage device for classifying the next wafer.

2. The method according to claim 1, further comprising: Retrieving the initial classification, the predicted classification, and the final classification of the first wafer from the storage device; And Training the machine learning model based on the initial classification, the predicted classification, and the final classification of a plurality of wafers.

3. The method according to claim 1, wherein the user input further comprises: Receiving, as the user input, confirmation of the initial classification or the predicted classification or both as the final classification.

4. The method according to claim 1, wherein the user input further comprises: Receiving, as the user input, a modification of the initial classification or the predicted classification as the final classification.

5. The method according to claim 1, wherein the user input further comprises: Receiving, as the user input, a user classification as the final classification.

6. The method according to claim 1, further comprising: Enabling a formatted user display that lists the initial classification of the rule-based model, the predicted classification of the machine learning model, and the user input.

7. The method according to claim 6, further comprising: Enabling the formatted user display to drill down into multiple details of the first wafer information when the user selects the first wafer via a third user interaction element.

Images