Systems and Methods for Process Monitoring and Control

Abstract

To provide a system and a method for continuous deployment in advanced process control and monitoring.SOLUTION: A system comprises: a data processing module that receives and processes a plurality of data types and data sets from a plurality of different sources in order to generate training data; a training and optimization module that provides the training data to a machine learning pipeline in order to train and optimize a model; and an inference module that uses the model in order to generate one or more predictive metrics in substantially real-time. The one or more predictive metrics are associated with the inference module that is usable to characterize output of a process performed by a process apparatus.SELECTED DRAWING: Figure 1

Description

Background Art

[0001] Some virtual measurement systems and methods can model processes and characteristics related to the monitoring and control of manufacturing processes, such as the processing of wafers in the semiconductor industry. Statistically modeling processes and characteristics can, in part, use current or historical data, such as measurements from up-to-date sensor data or historical sensor data. Virtual measurement systems and methods can provide more accurate measurements, for example, to control a manufacturing process. Virtual measurement systems and methods can increase productivity, improve quality, or reduce maintenance costs when compared to, for example, the physical inspection of a manufacturing process that uses conventional measurements. For example, while a human operator using conventional measurements may only be able to sample a very small part of the units, virtual measurement systems and methods can potentially sample all or substantially all of the units (e.g., semiconductor wafers) in a manufacturing process. Virtual measurement systems and methods can, in part, use machine learning methods to predict process variables or target properties associated with a manufacturing process.

Summary of the Invention

Means for Solving the Problems

[0002] Disclosed herein are systems and methods for continuous deployment in advanced process control and monitoring. The systems and methods can improve the prediction performance of process variables or target properties.

[0003] In one aspect, what is disclosed is a system and method for process monitoring and control. The system is configured to receive and process multiple data types and data sets from multiple different sources, for example, to generate training data, a data processing module, and to provide the training data to a machine learning pipeline to train and optimize a model, a training and optimization module, and to use the model to generate one or more prediction metrics substantially in real time, an inference module, wherein the one or more prediction metrics can be used to characterize the output of a process performed by process equipment.

[0004] In some embodiments, the inference module is configured to receive process data and provide it to the model to generate one or more prediction metrics, where the process data is received from the process equipment substantially in real time when the process is being performed.

[0005] In some embodiments, the inference module is configured to provide one or more prediction metrics for process control or for process monitoring, improvement, or troubleshooting.

[0006] In some embodiments, the system further comprises a process control module configured to use one or more prediction metrics to detect drift, deviation, or departure in the process or process equipment.

[0007] In some embodiments, the process control module is configured to use one or more prediction metrics to correct or reduce drift, deviation, or departure in the process or process equipment.

[0008] In some embodiments, the process control module is configured to use one or more predictive metrics to improve process productivity through integration with inter-execution control.

[0009] In some embodiments, the model comprises a virtual metrology (VM) model.

[0010] In some embodiments, the system further comprises process equipment, and the process equipment comprises semiconductor process equipment.

[0011] In some embodiments, the output of the process comprises a deposited or processed structure.

[0012] In some embodiments, the deposited or processed structure comprises a film, a layer, or a substrate.

[0013] In some embodiments, one or more predictive metrics comprise one or more dimensions or properties of a film, a layer, or a substrate.

[0014] In some embodiments, the system is configured to be used or deployed in a manufacturing environment.

[0015] In some embodiments, the plurality of data types and data sets comprise (1) historical process data, (2) current process data, (3) historical measurement data of one or more predictive metrics, (4) current measurement data of one or more predictive metrics, (5) operation data, and / or (6) equipment specification data.

[0016] In some embodiments, the data processing module is configured to verify historical process data and historical measurement data against operation data and equipment specification data.

[0017] In some embodiments, the plurality of sources comprises a database configured to store at least historical process data or historical measurement data.

[0018] In some embodiments, the plurality of sources comprises a database or log configured to store at least operation data or device specification data.

[0019] In some embodiments, the plurality of sources comprises process equipment.

[0020] In some embodiments, the plurality of sources comprises measurement equipment configured to collect current measurement data.

[0021] In some embodiments, the data processing module is configured to receive and process a plurality of data types or data sets by generating a component hierarchical structure of the process equipment.

[0022] In some embodiments, the component hierarchical structure comprises a nested structure of one or more components used within and / or in conjunction with (i) the process equipment and (ii) the process equipment.

[0023] In some embodiments, one or more components comprise one or more sub-devices including chambers, stations, and / or sensors.

[0024] In some embodiments, the data processing module is configured to receive and process a plurality of data types or data sets by generating a step operation hierarchical structure of a recipe related to the process.

[0025] In some embodiments, the recipe comprises a plurality of steps, and each step of the plurality of steps comprises a plurality of different sub-operations.

[0026] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by removing one or more data outliers.

[0027] In some embodiments, the data processing module is configured to preprocess and remove data outliers from process data before the process data is input into the model within the inference module.

[0028] In some embodiments, the training data is continuously updated with current process data and current measurement data.

[0029] In some embodiments, the machine learning pipeline comprises two or more components from multiple components consisting of (i) feature engineering, (ii) time recognition data normalization, and / or (iii) adaptive learning algorithms.

[0030] In some embodiments, the machine learning pipeline is configured to apply the training data through two or more components sequentially or simultaneously.

[0031] In some embodiments, feature engineering comprises extracting multiple features from raw trace data or sensor data within the training data.

[0032] In some embodiments, feature engineering comprises, at least in part, the use of an algorithm for selecting one or more features from the list of extracted features based on the local relationship between the input and output of the model.

[0033] In some embodiments, time recognition data normalization comprises decomposing time series data into one or more components including smoothed data, trend data, and / or detrended data.

[0034] In some embodiments, the time recognition data normalization is based on the model and the data type of the model.

[0035] In some embodiments, the adaptive learning algorithm is an adaptive online ensemble learning algorithm.

[0036] In some embodiments, the training and optimization module is configured to optimize the model using hyperparameter optimization, at least in part.

[0037] In some embodiments, the training and optimization module is configured to train the model using a given set of hyperparameters on the output from the machine learning pipeline.

[0038] In some embodiments, the training and optimization module is further configured to evaluate the performance of the model based on the validation data.

[0039] In some embodiments, the validation data is split from the training data for hyperparameter optimization.

[0040] In some embodiments, the training and optimization module is further configured to use a hyperparameter optimization algorithm to select a set of hyperparameters for the next iteration to increase or improve the performance of the model based on past performance.

[0041] In some embodiments, the training and optimization module is further configured to repeatedly iterate (i)-(iii) until the performance of the model meets an end criterion.

[0042] In another aspect, what is disclosed is a method for process monitoring and control. The method may include, for example, (a) receiving and processing a plurality of data types and data sets from a plurality of different sources to generate training data; (b) providing the training data to a machine learning pipeline to train and optimize a model; and (c) generating one or more prediction metrics substantially in real time, where the one or more prediction metrics can be used to characterize the output of a process performed by process equipment.

[0043] Additional aspects and advantages of the present disclosure will become readily apparent from the following detailed description, which illustrates only exemplary embodiments of the present disclosure. As will be recognized, the present disclosure is capable of other different embodiments, and some of the details thereof are capable of modification in various obvious respects without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive. The present invention provides, for example, the following items. (Item 1) A system for process monitoring and control, a data processing module configured to receive and process a plurality of data types and data sets from a plurality of different sources to generate training data; a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model; an inference module configured to use the model to generate one or more prediction metrics substantially in real time, where the one or more prediction metrics can be used to characterize the output of a process performed by process equipment A system comprising (Item 2) The inference module is configured to receive process data and provide it to the model in order to generate the one or more prediction metrics, and the process data is received from the process equipment substantially in real time when the process is carried out, for the system described in the above item. (Item 3) The inference module is configured to provide the one or more prediction metrics for process control or for process monitoring, improvement, or troubleshooting, for the system described in any one of the above items. (Item 4) The system further comprises a process control module configured to use the one or more prediction metrics to detect drift, deviation, or departure in the process or the process equipment, for the system described in any one of the above items. (Item 5) The process control module is configured to use the one or more prediction metrics to correct or reduce the drift, deviation, or departure in the process or the process equipment, for the system described in any one of the above items. (Item 6) The process control module is configured to use the one or more prediction metrics to improve process productivity through integration with inter-execution control, for the system described in any one of the above items. (Item 7) The model comprises a virtual measurement (VM) model, for the system described in any one of the above items. (Item 8) The system further comprises the process equipment, and the process equipment comprises semiconductor process equipment, for the system described in any one of the above items. (Item 9) The output of the process comprises a deposited or processed structure, for the system described in any one of the above items. (Item 10) The deposited or processed structure is the system according to any one of the above items, comprising a film, a layer, or a substrate. (Item 11) The one or more predictive metrics are the system according to any one of the above items, comprising one or more dimensions or properties of the film, the layer, or the substrate. (Item 12) The system is the system according to any one of the above items, configured to be used or deployed in a manufacturing environment. (Item 13) The plurality of data types and data sets are the system according to any one of the above items, comprising (1) historical process data, (2) current process data, (3) historical measurement data of the one or more predictive metrics, (4) current measurement data of the one or more predictive metrics, (5) operation data, and / or (6) equipment specification data. (Item 14) The data processing module is the system according to any one of the above items, configured to verify the historical process data and the historical measurement data against the operation data and the equipment specification data. (Item 15) The plurality of sources are the system according to any one of the above items, comprising a database configured to store at least the historical process data or the historical measurement data. (Item 16) The plurality of sources are the system according to any one of the above items, comprising a database or a log configured to store at least the operation data or the equipment specification data. (Item 17) The plurality of sources are the system according to any one of the above items, comprising the process equipment. (Item 18) The above-mentioned plurality of sources includes a measuring device configured to collect the above-mentioned current measurement data, and the system according to any one of the above items. (Item 19) The above data processing module is configured to receive and process the above plurality of data types or data sets by generating a component hierarchical structure of the above process equipment, and the system according to any one of the above items. (Item 20) The above component hierarchical structure includes a nested structure of one or more components used (i) in the above process equipment and (ii) within or in combination with the above process equipment, and the system according to any one of the above items. (Item 21) The above one or more components include one or more sub-devices including chambers, stations, and / or sensors, and the system according to any one of the above items. (Item 22) The above data processing module is configured to receive and process the above plurality of data types or data sets by generating a step operation hierarchical structure of a recipe related to the above process, and the system according to any one of the above items. (Item 23) The above recipe includes a plurality of steps, and each step of the above plurality of steps includes a plurality of different sub-operations, and the system according to any one of the above items. (Item 24) The above data processing module is configured to receive and process the above plurality of data types or data sets by removing one or more data outliers, and the system according to any one of the above items. (Item 25) The above data processing module is configured to preprocess and remove data outliers from the above process data before the above process data is input into the above model within the above inference module, and the system according to any one of the above items. (Item 26) The training data is the system according to any one of the above items, which is continuously updated with the current process data and the current measurement data. (Item 27) The machine learning pipeline is the system according to any one of the above items, which includes two or more components selected from a plurality of components consisting of (i) feature engineering, (ii) time recognition data normalization, and / or (iii) adaptive learning algorithms. (Item 28) The machine learning pipeline is the system according to any one of the above items, which is configured to apply the training data through the two or more components sequentially or simultaneously. (Item 29) The feature engineering is the system according to any one of the above items, which includes extracting a plurality of features from raw trace data or sensor data in the training data. (Item 30) The feature engineering is the system according to any one of the above items, which includes using an algorithm for selecting one or more features from the list of extracted features, at least partially based on the local relationship between the input and output of the model. (Item 31) The time recognition data normalization is the system according to any one of the above items, which includes decomposing time series data into one or more components including smoothed data, trend data, and / or detrended data. (Item 32) The time recognition data normalization is the system according to any one of the above items, which is based on the model and the data type of the model. (Item 33) The adaptive learning algorithm is the system according to any one of the above items, which is an adaptive online ensemble learning algorithm. (Item 34) The training and optimization module is configured to optimize the model using hyperparameter optimization, at least in part, for the system according to any one of the above items. (Item 35) The training and optimization module is configured to train the model using a given set of hyperparameters for the output from the machine learning pipeline, for the system according to any one of the above items. (Item 36) The training and optimization module is further configured to evaluate the performance of the model based on validation data, for the system according to any one of the above items. (Item 37) The validation data is split from the training data for the hyperparameter optimization, for the system according to any one of the above items. (Item 38) The training and optimization module is further configured to use a hyperparameter optimization algorithm to select a set of hyperparameters for the next iteration based on past performance to increase or improve the performance of the model, for the system according to any one of the above items. (Item 39) The training and optimization module is further configured to repeatedly iterate (i)-(iii) until the performance of the model meets an end criterion, for the system according to any one of the above items. (Item 40) A method for process monitoring and control, (a) Receiving and processing multiple data types and data sets from multiple different sources to generate training data; (b) Providing the training data to a machine learning pipeline to train and optimize a model; (c) generating one or more prediction metrics substantially in real time, the one or more prediction metrics being usable to characterize the output of a process implemented by process equipment A method comprising. (Abstract) Systems and methods for advanced process control and monitoring are described. The systems and methods include a data processing module configured to receive and process multiple data types and data sets from multiple different sources to generate training data, a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model, and an inference module configured to use the model to generate one or more prediction metrics substantially in real time, the one or more prediction metrics being usable to characterize the output of a process implemented by process equipment.

[0044] (Incorporation by reference) All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. Insofar as any incorporated publication, patent, or patent application conflicts with the disclosure contained herein, this specification is intended to supersede and / or be superior to any such conflicting material.

Brief Description of the Drawings

[0045] The novel features of the present disclosure are set forth in detail in the appended claims. A more complete understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that illustrates embodiments in which the principles of the present disclosure are utilized, and to the accompanying drawings.

[0046]

Figure 1

[0047]

Figure 2

[0048]

Figure 3

[0049]

Figure 4A

Figure 4B

[0050]

Figure 5

[0051]

Figure 6

[0052]

Figure 7

[0053]

Figure 8

[0054]

Figure 9

[0055]

Figure 10

[0056]

Figure 11A

Figure 11B

Figure 11C

Figure 11D

[0057]

Figure 12

[0058]

Figure 13A

Figure 13B

Figure 13C

[0059]

Figure 14A

Figure 14B

Figure 14C

Figure 14D

[0060]

Figure 15

Figure 16

[0061]

Figure 17

[0062]

Figure 18

[0063]

Figure 19

[0064] Detailed Description Various embodiments of the present disclosure are shown and described herein, but such embodiments are provided only as examples. Numerous variations, modifications, or alternatives may occur without departing from the present disclosure. It should be understood that various alternatives to the embodiments of the disclosure described herein may be employed.

[0065] Typical virtual metrology systems and methods are defective for at least several reasons. Some systems and methods may not be able to model data drift, data shift, or hierarchical data structures observed in manufacturing processes such as semiconductor manufacturing processes. Data drift generally can refer to a gradual change in process dynamics, for example, due to aging of equipment used in a manufacturing process. Data shift generally can refer to a sudden change in process dynamics, for example, due to external actions such as maintenance or calibration. Advanced process control methods associated with virtual metrology methods may require accurate prediction of process variables or target properties in a manufacturing process. However, some methods may not be able to model non-stationarity observed in real or actual data, for example, up-to-date or historical sensor data. Non-stationarity can occur, for example, in semiconductor manufacturing processes such as chemical vapor deposition (CVD), etching, diffusion, or other processes. For example, data drift can cause a steady or slow decrease in the accuracy of predictions associated with a virtual metrology model. Data shift can cause a sudden malfunction of a virtual metrology model. Some virtual metrology methods that use moving window-based methods can be inaccurate, for example, due to selecting a small subset of data. Some virtual metrology methods that use just-in-time learning methods cannot adapt to changes in the underlying relationships between process variables or target properties.

[0066] What is recognized herein is the need for better systems and methods for advanced process control and monitoring in manufacturing processes. The systems and methods disclosed herein can generate adaptive online time series predictions regarding process variables or target properties in a manufacturing process. Systems and methods for process monitoring and control

[0067] In one aspect, what is disclosed in this specification (Figure 1) is a system for process monitoring and control. The system is configured to receive and process multiple data types and data sets from multiple different sources, for example, to generate training data, a data processing module, and to provide the training data to a machine learning pipeline for training and optimizing a model, a training and optimization module, and to use the model to generate one or more prediction metrics substantially in real time, where the one or more prediction metrics can be used to characterize the output of a process implemented by process equipment, an inference module.

[0068] As shown in Figure 2, the system may be associated with a method for advanced process control and monitoring in a manufacturing process. The methods described herein can generally include methods related to adaptive online time series prediction of target properties or process variables. Target properties can include, for example, the properties or characteristics of a target product (e.g., the film thickness or refractive index of a wafer) determined through measurements from measurement tools or instruments. Process variables can include, for example, data generated from sensors associated with a manufacturing process or process equipment (e.g., temperature, power, or current in a chemical vapor deposition process or equipment). Adaptive online time series methods can generally include time-aware normalizer methods or adaptive online learner methods. As shown in Figure 3, the systems and methods described herein can be used in advanced process control and monitoring of a manufacturing process. The manufacturing process can include a semiconductor manufacturing process associated with the processing of semiconductor wafers. For example, the process can include a chemical vapor deposition process for semiconductor wafers. Use

[0069] In some embodiments, the inference module is configured to receive process data and provide it to a model to generate one or more prediction metrics, where the process data is received from process equipment substantially in real-time as the process is being performed. In some cases, the systems and methods disclosed herein may include virtual metrology systems and methods that include one or more inference modules. In some embodiments, the model comprises a virtual metrology (VM) model. The virtual metrology model can be configured to predict metrics or properties associated with a manufacturing process. The metric or property may be related to a process variable or a target property.

[0070] In some cases, a metric, property, or characteristic may be associated with a target property or a process variable in a product's manufacturing process. The target property can include properties associated with a product generated by equipment in a manufacturing process. For example, the target property may include the thickness or refractive index of a film or layer on a wafer or substrate in a chemical vapor deposition process. The process variable can include variables associated with equipment configured to perform a manufacturing process. For example, the process variable may include pressure, gas flow rate, power, temperature, current, and the like, associated with equipment configured to perform a chemical vapor deposition process. The prediction of the target property or process variable may be based, at least in part, on measurement data, sensor data, sensor specification data, equipment data, equipment specification data, process data, or process specification data. The inference module disclosed herein may generate a prediction of the target property or process variable without involving a physical measurement of the target property or process variable.

[0071] In some embodiments, the inference module is configured to provide one or more predictive metrics for process control or for process monitoring, improvement, or troubleshooting. Manufacturing processes, such as semiconductor manufacturing processes, may use advanced process control systems and methods to control and monitor target properties or process variables (e.g., metrics). Advanced process control and monitoring methods can include methods for monitoring, controlling, improving, or troubleshooting a process using a feedback loop. Advanced process control systems and methods can include virtual metrology systems and methods. The virtual metrology systems and methods described herein may determine or predict target properties or process variables (e.g., outputs) and may provide predicted target properties or process variables (e.g., inputs) in a feedback loop for advanced process control and monitoring.

[0072] In some embodiments, the system further comprises a process control module configured to use one or more predictive metrics to detect drift, offset, or deviation in a process or process equipment. Metrics (e.g., data) from a manufacturing process (e.g., a semiconductor process) can be significantly different from other data, for example, due to the distribution of continuously changing data. For example, the data can include time series data with non-stationary characteristics. The non-stationary characteristics can include data drift, data offset, hierarchical data structure, or deviation in a process or equipment. Data drift can cause a steady or slow decrease in the accuracy of a virtual measurement model. Data drift can occur, for example, due to aging in equipment or sensors. Data offset can cause a sudden malfunction in a virtual measurement model. Data offset can occur, for example, due to a change in equipment or sensor characteristics after maintenance or calibration. The data can have a hierarchical structure associated with a process, equipment, station, sensor, and the like. In some embodiments, the process control module is configured to use one or more predictive metrics to correct or mitigate drift, offset, or deviation in a process or process equipment. Alternatively, or in addition, the process control module can be configured to use one or more predictive metrics to correct or mitigate deviation in a process or equipment.

[0073] In some embodiments, the process control module is configured to use one or more predictive metrics to improve process productivity through integration with in-run control. In some cases, the systems and methods described herein may be integrated into a manufacturing processing line as an in-run system in real time. In some embodiments, the system is configured to be used or deployed in a manufacturing environment. An in-run system can generally include a method for modifying a recipe associated with a process in real time. A recipe can include a set of instructions for performing a process in a manufacturing process. A recipe may be associated with parameters. For example, a recipe and parameters for a chemical vapor deposition process may include deposition instructions with parameters that define power levels, such as specified current and voltage. Additionally, an in-run system can generally include a method for modifying process variables of equipment for chemical vapor deposition, such as pressure, gas flow rate, power, temperature, current, and the like, which are control parameters associated with the process in real time. In some cases, as shown in FIGS. 15-16, the step of modifying a control parameter, recipe, or model may be performed using a graphical user interface (GUI). For example, a user can create their own unique model incorporating their expertise or compare actual process outcomes with predicted process outcomes. The systems and methods described herein can include a method for generating real-time predictions about data drift, data shift, hierarchical data structures, or deviations in a process or equipment and for inferring or characterizing the output of a manufacturing process. The inferred or characterized output can be used to modify recipe parameters or control parameters in a real-time feedback loop to improve the predictive performance of process variables or target properties.

[0074] In some embodiments, the system further comprises process equipment, which comprises semiconductor process equipment. For example, as shown in FIGS. 4A-4B, the process equipment can include equipment associated with semiconductor wafer production, oxidation, photolithography, etching, deposition, ion implantation, metal wiring, electrical dicing, packaging, and the like. In some embodiments, the deposited or processed structure comprises a film, a layer, or a substrate. The systems and methods described herein may be configured to monitor or control a target property or a process variable. The target property may be associated with a wafer film, layer, or substrate. Alternatively, or in addition, the target property may include mechanical, electrical, or structural properties of the product. For example, the electrical properties of a wafer can include controlling and monitoring the conductivity of the wafer via a doping process. The mechanical properties of a wafer can include controlling and monitoring the strain of the wafer via a metalorganic chemical vapor deposition process. The structural properties can include controlling and monitoring the thickness or the refractive index of a film via a chemical vapor deposition process. In some embodiments, one or more predictive metrics comprise one or more dimensions or properties of a film, layer, or substrate. Data processing

[0075] In some embodiments, the plurality of data types and data sets may comprise (1) historical process data, (2) current process data, (3) historical measurement data of one or more predictive metrics, (4) current measurement data of one or more predictive metrics, (5) operational data, and / or (6) equipment specification data. As described elsewhere herein, the systems and methods may be associated with virtual metrology systems and methods. The virtual metrology systems and methods can generate a substantial amount of data. The data may include up-to-date (e.g., current, or real-time) data or historical data. The up-to-date data or historical data may be associated with sensor data, sensor specification data, process data, process specification data, measurement data, operational data, equipment data, or equipment specification data. For example, the data can comprise fault detection and classification (FDC) data, sensor data (e.g., temperature, pressure, power, current, gas volume, and the like), measurement data (e.g., film thickness, film refractive index, critical dimension, and the like), or operational data.

[0076] In some cases, the systems and methods disclosed herein can assist in the processing of manufacturing data in streaming mode or batch mode. Streaming mode generally may include data received in real-time during the course of a manufacturing process, e.g., up-to-date or real-time data. Batch mode generally may include data received non-real-time from a manufacturing process, e.g., historical data.

[0077] In some embodiments, the data processing module is configured to verify historical process data and historical measurement data against operational data and equipment specification data. Verification of historical process data against operational data or equipment data can be associated with a threshold value. In some cases, the historical process data may be verified when the threshold value is at least about 60%, 70%, 80%, 90%, or more of the data associated with the operational data or equipment data. Verification of historical measurement data against operational data or equipment data can be associated with a threshold value. In some cases, the historical measurement data may be verified when the threshold value is at least about 60%, 70%, 80%, 90%, or more of the data associated with the operational data or equipment data. The threshold value can be associated with a predetermined standard deviation. In some cases, verification of historical process data may be performed when the threshold value is within at least about 3 standard deviations (3σ), 2 standard deviations (2σ), or 1 standard deviation (1σ) of the operational data or equipment data. In some cases, verification of historical measurement data may be performed when the threshold value is within at least about 3 standard deviations (3σ), 2 standard deviations (2σ), or 1 standard deviation (1σ) of the operational data or equipment data.

[0078] In some embodiments, the plurality of sources comprises a database configured to store at least historical process data or historical measurement data. In some cases, the plurality of sources can include a database configured to store up-to-date or real-time process data. In some cases, the plurality of sources can include a database configured to store up-to-date or real-time measurement data. Some or all of the data can be stored within one or more databases. In some cases, the plurality of sources can include a database configured to store a subset of up-to-date or real-time process data. In some cases, the plurality of sources can include a database configured to store a subset of up-to-date or real-time measurement data. The subset of data can comprise at least about 25%, 50%, 75%, or more of the data associated with historical process data, historical measurement data, up-to-date process data, or up-to-date measurement data.

[0079] In some embodiments, the plurality of sources comprises a database or log configured to store at least operational data or equipment specification data. In some cases, the plurality of sources can include a database configured to store up-to-date or real-time or historical operational data. In some cases, the plurality of sources can include a database configured to store up-to-date or real-time or historical equipment specification data. Some or all of the data can be stored within one or more databases. In some cases, the plurality of sources can include a database configured to store a subset of up-to-date or real-time or historical operational data. In some cases, the plurality of sources can include a database configured to store a subset of up-to-date or real-time or historical equipment specification data. The subset of data can comprise at least about 25%, 50%, 75%, or more of the data associated with historical operational data, historical equipment specification data, up-to-date operational data, or up-to-date equipment specification data.

[0080] In some embodiments, the plurality of sources comprises process equipment. As described elsewhere herein, the manufacturing process may be associated with the process equipment, including one or more set points that can be set by an engineer / designer. For example, process equipment in a semiconductor manufacturing process can include equipment associated with wafer production, oxidation, photolithography, etching, deposition, ion implantation, metal wiring, electrical dicing, packaging, and the like. The user can set various set points for any of these pieces of equipment, such as temperature, angle, height, depth, width, size, amount of material, and many others.

[0081] In some embodiments, the plurality of sources comprise measurement equipment configured to collect current measurement data. As described elsewhere herein, the manufacturing process may be associated with the measurement equipment. The measurement equipment can include measurement tools or instruments configured to measure a target property. For example, the measurement tools or instruments may be used to measure target properties in semiconductor manufacturing processes such as film thickness, refractive index, critical dimensions, and the like, with respect to a wafer.

[0082] In some embodiments, the data processing module is configured to receive and process a plurality of data types or data sets by generating a component hierarchy of process equipment. The manufacturing process can be associated with a subsystem, e.g., process equipment having components. The manufacturing process can include at least one, two, three, four, five, or more types of process equipment. The process equipment can all be of the same type of process equipment. The process equipment can be of different types of process equipment. The components of the process equipment can include sensors configured to collect data associated with process variables.

[0083] In some embodiments, the component hierarchy structure comprises a nested structure of one or more components that are (i) process equipment and (ii) used within or in conjunction with the process equipment. For example, as shown in FIG. 5, a semiconductor manufacturing process may include a chemical vapor deposition process associated with process equipment, such as equipment 1, equipment 2, etc. The components of the process equipment can include chambers for performing the chemical vapor deposition process, such as chamber A, chamber B, etc. The chamber can include stations, such as station S1, station S2, etc., configured to measure sensor data such as temperature, pressure, power, current, gas flow rate, and equivalents. In some embodiments, one or more components comprise one or more sub-devices including chambers, stations, and / or sensors.

[0084] In some cases, the manufacturing process may comprise many different types of processes having different types of process variables or target properties. As described elsewhere herein, the systems and methods can be extended by aggregating data from sensors of more than one process equipment or from more than one sensor. Aggregating data from processes, process equipment, or sensors can improve the step of generating predictive metrics, such as predictions of target properties or process variables. For example, by aggregating and processing data from multiple equipment chambers for the same process step, a sparse data set can still be used to generate a reliable, robust, and scalable virtual metrology model by training the machine learning algorithms described herein.

[0085] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by generating a step operation hierarchy of a recipe for a process. For example, as shown in Table 1, a recipe for chemical vapor deposition in a semiconductor manufacturing process may include a recipe or method having operations and sub-operations. The operations can include loading, preparation, deposition, removal, or cleaning. Sub-operations related to deposition can include controlling a voltage or current to generate a predetermined power. In some embodiments, the recipe comprises a plurality of steps, and each step of the plurality of steps comprises a plurality of different sub-operations. The operations or sub-operations may be associated with a data type or data set related to a process variable or a target property. The process variables may include, for example, pressure, gas amount, power, temperature, and equivalents related to a chemical vapor deposition process. The target property may include, for example, the film thickness or refractive index of a semiconductor wafer processed through a chemical vapor deposition process. [Table 1]

[0086] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by removing one or more data outliers. The multiple data types or data sets may include data in raw or unmodified form, such as, for example, trace data. For example, FIG. 6 depicts trace data associated with process variables related to the operation or sub-operations of a chemical vapor deposition process. The process variables can include, for example, the temperature, power, or current of the chemical vapor deposition process. In some cases, the trace data may be processed to generate clean data. The step of generating clean data can include, for example, steps of handling irregular data or removing data outliers (e.g., within or outside a predetermined threshold) based on process specifications, equipment specifications, or sensor specifications. The data can be up-to-date data (e.g., current, or real-time data) or historical data. In some embodiments, the data processing module is configured to preprocess and remove data outliers from the process data before the process data is input into the model within the inference module.

[0087] In some embodiments, the training data is continuously updated with current process data and current measurement data. Features can be extracted from the trace data or clean data to generate training data, test data, or validation data. The training and optimization module may use the features to train and optimize a virtual measurement model via a machine learning pipeline. Model Training Pipeline

[0088] In some embodiments, the machine learning pipeline comprises two or more components from a plurality of components consisting of (i) feature engineering, (ii) time-aware data normalization, and / or (iii) adaptive learning algorithms. Feature engineering can generally include determining the main characteristics of time-series data having data drift, data shift, hierarchical data structure, or deviation of data in a process or process equipment, extracting features, and selecting features. Feature engineering may be associated with a correlation coefficient or variable shrinkage. Time-aware data normalization can generally include decomposing time-series data into a plurality of components and individually adjusting the components of the process. Time-aware data normalization may be associated with differentiation or moving average. An adaptive learning method or algorithm can generally include determining a changing relationship between input data (e.g., process variables such as features or sensor data) and output data (e.g., process variables or target properties). The adaptive learning method or algorithm may be associated with rolling regression or online regression. One or more combinations of feature engineering, time-aware data normalization, or adaptive learning algorithms can improve the prediction performance of target properties or process variables in advanced process control and monitoring. In some embodiments, the machine learning pipeline is configured to apply training data sequentially or simultaneously through two or more components.

[0089] As described elsewhere herein, advanced process control and monitoring may be associated with virtual metrology systems and methods. The virtual metrology systems and methods may be associated with machine learning systems and methods. The machine learning systems and methods can use data from a manufacturing process to train, test, and validate a virtual metrology model. The virtual metrology model can generate adaptive online time-series predictions for use in virtual metrology. The predictions may include predictions regarding process variables or target properties in a manufacturing process such as a semiconductor manufacturing process.

[0090] The virtual measurement models disclosed herein can address technical challenges specific to data associated with semiconductor manufacturing processes, such as data drift, data shift, hierarchical data structures, or data deviation in a process or process equipment. As described elsewhere herein, the systems and methods may compare actual data (e.g., raw or trace data from current or historical sensor data) to defined or certified process or equipment data, such as equipment specifications or process specifications. Actual data that does not match the defined or certified data outside of a predetermined threshold may be further processed prior to use in the construction of a machine learning model, such as a virtual measurement model. The virtual measurement models can enable a more efficient workflow for controlling and monitoring manufacturing processes.

[0091] The step of generating a machine learning model (e.g., a virtual measurement model) via a machine learning pipeline generally may include steps of receiving data, preprocessing data, selecting or engineering features from the data, training a model using the data or features, testing a model using the data or features, or validating a model using the data or features. The validated model can be deployed or integrated into a manufacturing process. The data or features can include data or features generated from multiple iterations of training, testing, or validation via the machine learning pipeline. The data or features can include data or features generated from multiple instances of a manufacturing process. The multiple instances can occur during different periods or during the same period. The data or features can be automatically tracked or used in real time. The data or features can be stored for auditing and used at a later time. In some embodiments, time-aware data normalization comprises decomposition of time series data into one or more components, including smoothed data, trend data, and / or detrended data.

[0092] Figures 7 and 8 depict operations that may be associated with feature engineering in a machine learning pipeline. Feature engineering can generally include operations related to feature creation, transformation, extraction, or selection. Feature engineering operations can comprise at least 1, 2, 3, 4, 5, or more operations. Feature engineering operations can comprise at most 5, 4, 3, 2, or 1 operation. In some cases, feature engineering operations can include operations such as a feature engineer (e.g., performing a combination of columns), a nan frac threshold (e.g., removing columns with values above a threshold that are not a number (NaN)), a simple imputer (e.g., performing mean imputation of missing values), a sliding window correlation selector, or polynomial features (e.g., multiplying columns together). In some cases, feature engineering operations can include operations such as a feature engineer, a nan frac threshold, a simple imputer, a function transformer such as a detrender, a differentiator, an exponentially weighted moving average, a standard scaler, or a sliding window correlation selector.

[0093] In some embodiments, feature engineering comprises extracting a plurality of features from raw trace data or sensor data in training data. Compared to methods that use global relationships, the sliding window method can better determine local relationships for data with non-stationary characteristics. Non-stationary characteristics can include, for example, data drift or data shift as described elsewhere in this specification. In some embodiments, feature engineering comprises, at least in part, using an algorithm to select one or more features from a list of extracted features based on local relationships between the inputs and outputs of the model. For example, as shown in FIG. 9, methods that use global relationships can inaccurately determine the relationship between input data and output data. Methods that use global relationships may predict the relationship between output (y) and input (x) as y = -0.6x + 132 over all time points, although there are three clearly different periods. The sliding window method can predict the changing relationship between output (y) and input (x) as y = -2x + 201, y = x - 103, and y = -4.1x + 932 over three clearly different periods of trace data. FIG. 10 conceptually depicts the sliding window method. The sliding window method can determine the running correlation between output data and input data by (i) sliding a window across the time series of data and (ii) averaging the correlations as the selection score. In some embodiments, time-aware data normalization is based on the model and the data type of the model. The correlation of the i-th feature can be determined using the following equation. [Chemical Formula]

[0094] Where i is the i-th variable of the input data (e.g., X data, feature, sensor data), j is the j-th variable of the output data (e.g., Y data, response, target, measurement), r is the correlation coefficient, T is the total number of input data, W is the window size, and J is the total number of variables of the output data.

[0095] In some embodiments, the adaptive learning algorithm is an adaptive online ensemble learning algorithm. A single machine learning model may not be able to accurately predict all process variables or target properties at a given confidence level. Thus, the methods disclosed herein can generate, maintain, or update two or more models for use in an ensemble model to predict process variables or target properties at a given confidence level. The models may predict process variables or target properties at a confidence level of at least about 60%, 70%, 80%, 90%, 95%, or better. In some cases, the models may predict process variables or target properties at a confidence level of at least about 95%. The systems and methods described herein can generate, maintain, or update multiple machine learning models for selection in different manufacturing processes to improve prediction performance. In some cases, the systems and methods described herein can verify the performance of machine learning models against multiple manufacturing processes or targets, for example, to generate the best functioning models. After automatically generating a benchmark test for a process variable or target property, the best functioning machine learning model can be deployed in a sensitive manner with little or no human intervention. Model Optimization

[0096] In some embodiments, the training and optimization module is configured to optimize the model using, at least in part, hyperparameter optimization. In some cases, the virtual metrology models disclosed herein can be trained using hyperparameter optimization with streaming data or batch data. Hyperparameter optimization (e.g., tuning of hyperparameters in a machine learning model) may be used for one, more than one, or all of the manufacturing processes. Hyperparameter optimization for all or substantially all of the manufacturing processes can improve overall prediction performance by searching for a shared set of hyperparameters that cross the processes, which can mitigate overfitting compared to optimizing for one process or some processes. In some cases, hyperparameter optimization can improve prediction performance by at least about 1%, 5%, 10%, 20%, or more when optimizing for all or substantially all of the processes compared to optimizing for some processes or one process. Some processes can be up to about 30%, 20%, 10%, or less of all of the processes at most.

[0097] In some embodiments, the training and optimization module is configured to (i) train a model using a given set of hyperparameters on the output from a machine learning pipeline. In some cases, the systems and methods disclosed herein may process data from many different sensors, different devices, or different processes and select the most relevant features. Training and optimization may include using hyperparameter optimization on one or more features for better prediction performance of a target property or process variable. For example, referring to FIGS. 7 or 8, the hyperparameters associated with a sliding window correlation selector may include multicollinearity and the like. The hyperparameters associated with polynomial features may include degree and the like. The hyperparameters associated with detrending may include half-life, change point, and the like. The hyperparameters associated with a scaler may include the type of scaler and the like.

[0098] In some embodiments, the training and optimization module is further configured to (ii) evaluate the performance of the model based on validation data. The systems and methods described herein can track the temporal changes in sensor data, device data, or process data to continuously update or optimize a virtual measurement model. Test data, training data, or validation data may change over time and thus may change the local or global relationship between input data and output data over time. The performance of the model can be continuously improved over time by training, testing, or validating the model using other training data, other test data, or other validation data that are different from previous training data, previous test data, or previous validation data.

[0099] In some embodiments, the validation data is split from the training data for hyperparameter optimization. Splitting the data for training, testing, or validation (e.g., split validation) can be performed using a given percentage. For example, the data may be split into 80% for training and 20% for testing. The data may be split into 80% for training, 10% for testing, and 10% for validation. Alternatively, or in addition, splitting the data can be performed using cross-validation, e.g., exhaustive cross-validation or non-exhaustive cross-validation such as k-fold validation.

[0100] In some embodiments, the training and optimization module is further configured to use a hyperparameter optimization algorithm to select a set of hyperparameters for the next iteration based on past performance so as to increase or improve the performance of the model. The systems and methods described herein may use methods associated with an algorithm for selecting hyperparameters for hyperparameter optimization. For example, a set of hyperparameters can be determined to improve the prediction performance over another, or previous, set of hyperparameters. The set of hyperparameters can be used for the next integration of training, testing, or validation of the model to increase or improve the prediction performance of the model.

[0101] In some embodiments, the training and optimization module is further configured to repeatedly repeat (i)-(iii) until the performance of the model meets an end criterion. In some cases, the end criterion can be a given metric based on the absolute value of the metric. Alternatively, or in addition, the end criterion can be a given metric based on the relative change of the metric. The end criterion can include no change in the metric over a given number of epochs, an absolute change in the metric, a decrease in the prediction performance observed over a given number of epochs, or an average change in the metric over a given number of epochs.

[0102] In another aspect, what is disclosed herein is a method for process monitoring and control. The method may include, for example, (a) receiving and processing multiple data types and data sets from multiple different sources to generate training data; (b) providing the training data to a machine learning pipeline to train and optimize a model; and (c) generating one or more prediction metrics substantially in real time, where the one or more prediction metrics can be used to characterize the output of a process implemented by process equipment. Machine learning method for prediction in process monitoring and control

[0103] Many machine learning (ML) methods implemented as algorithms are suitable as an approach for implementing the methods described herein. Such methods include, but are not limited to, supervised learning approaches, unsupervised learning approaches, semi-supervised approaches, or any combination thereof.

[0104] Machine learning algorithms may include, but are not limited to, neural networks (e.g., artificial neural networks (ANN), multi-layer perceptrons (MLP), long short-term memory (LSTM)), support vector machines, k-nearest neighbors, Gaussian mixture models, Gaussian processes, naive Bayes, decision trees, random forests, or gradient boosting trees. Linear machine learning algorithms may include, but are not limited to, linear regression, logistic regression, naive Bayes classifier, perceptron, or support vector machine (SVM), with or without a regularizer. Other machine learning algorithms for use in combination with the methods according to the present disclosure may include, but are not limited to, secondary classifiers, k-nearest neighbors, boosting, decision trees, random forests, neural networks, pattern recognition, Bayesian networks, or hidden Markov models. Other machine learning algorithms, including improvements or combinations of any of these, commonly used for machine learning may also be suitable for use in combination with the methods described herein. Any use of machine learning algorithms in a workflow may also be suitable for use in combination with the methods described herein. The workflow may include, for example, cross-validation, nested cross-validation, feature selection, row compression, data transformation, binning, normalization, standardization, and algorithm selection.

[0105] Machine learning algorithms can generally be trained by the following methodologies to build a machine learning model. In some cases, the generated model may determine or predict a target property of a product, such as the film thickness or refractive index of a wafer. The input data can include, for example, process variables such as X data, features, fault detection and classification (FDC) data, sensor data, and equivalents, as described elsewhere in this specification. The output data can include, for example, target properties such as Y data, responses, measurements, and equivalents, as described elsewhere in this specification. In some cases, the generated model may determine or predict a process variable. The input data can include, for example, process variables as described elsewhere in this specification. The output data can include, for example, process variables as described elsewhere in this specification.

[0106] 1. Collect a dataset for "training" and "testing" the machine learning algorithm. The dataset can include many features, such as features associated with sensor data, equipment, processes, and equivalents. The training dataset is used to "train" the machine learning algorithm. The test dataset is used to "test" the machine learning algorithm.

[0107] 2. Determine the "features" for the machine learning algorithm to be used for training and testing. The accuracy of the machine learning algorithm can depend on how the features are represented. For example, the feature values may be transformed using one-hot encoding, binning, standardization, or normalization. Also, not all features in the dataset may be used to train and test the machine learning algorithm. The selection of features can depend, for example, on the available computing resources and time or the importance of the features discovered during iterative testing and training. For example, features associated with sensor data or equipment specifications may be found to predict process variables or target properties.

[0108] 3. Select an appropriate machine learning algorithm. For example, a machine learning algorithm described elsewhere in this specification may be selected. The selected machine learning algorithm may depend, for example, on the available computing resources and time or whether the prediction is continuous or categorical in nature. The machine learning algorithm is used to build a machine learning model.

[0109] 4. Build a machine learning model. The machine learning algorithm is executed on the collected training dataset. The parameters of the machine learning algorithm may be adjusted by optimizing the performance with respect to the training dataset or via a cross-validation dataset. After parameter adjustment and learning, the performance of the machine learning algorithm may be verified on a dataset of naive samples that is separate from the training dataset and the test dataset. The constructed machine learning model can be accompanied by feature coefficients, importance measures, or weights assigned to individual features.

[0110] Once the machine learning model is determined ("trained") as described above, it can be used to generate predictions for process monitoring and control in a manufacturing process such as a semiconductor manufacturing process.

Examples

[0111] Although various examples of the present disclosure are shown and described herein, such examples are provided only as examples. Numerous variations, modifications, or alternatives may occur without departing from the present disclosure. It should be understood that various alternatives to the examples described herein may be employed. Example 1 - Adaptive Online Time-Series Prediction for Virtual Measurement in a Semiconductor Manufacturing Process Introduction

[0112] The systems and methods described herein (e.g., virtual metrology (VM) models) were used to statistically model wafer characteristics using sensor data and historical measurements to predict, control, and monitor semiconductor manufacturing processes, such as chemical vapor deposition (CVD) processes. The virtual metrology models increased productivity, improved quality, and reduced maintenance costs by reducing physical or human inspections associated with semiconductor manufacturing processes.

[0113] Compared to the systems and methods described herein, other methods (e.g., algorithmic methods or models) may not be able to work with data having data drift or data shift observed in semiconductor manufacturing processes. Data drift generally refers to a gradual change in process dynamics, for example, due to equipment aging. Data shift generally refers to a sudden change in process dynamics, for example, due to external actions such as maintenance, calibration, or layer changes.

[0114] To ensure the reliable use of virtual metrology models in advanced process control and monitoring, the predictions of the models should be highly accurate at all times of the process. However, some models may not be able to handle the non-stationarity observed in sensor data or measurement data generated by, for example, chemical vapor deposition, etching, diffusion, or other processes. Data drift can cause a steady or slow decrease in the accuracy of the model. Data shift can cause a sudden malfunction of the model. For example, Gaussian process solutions or neural network architectures may generate incomplete solutions or predictions mainly by focusing on data drift. For example, window-based models may generate unreliable predictions or fail when data shift occurs due to a fixed window size. For example, just-in-time learning models may not be able to adapt to changes in the relationship between input and output.

[0115] The systems and methods described herein use a novel time series prediction framework that incorporates a time recognition data normalization method and an adaptive learning method to generate an accurate virtual measurement model (see FIGS. 1 and 2). The time recognition normalization method can remove data drift from the data. The method may use various smoothing algorithms, such as exponential weighted moving average, to perform this transformation. The adaptive learning method can capture the time-varying relationship between the response (e.g., output) and the input. The systems and methods described herein can also generate an ensemble of predictions of several models, resulting in more accurate predictions in a data shift environment and reducing process variability on a real production line in a manufacturing process. Adaptive Online Time Series Prediction

[0116] The systems and methods described herein (e.g., an adaptive online time series method, algorithm, or approach) may integrate two complementary strategies or operations into a pipeline process to overcome data drift or data shift. The pipeline may include a time recognition normalization method and an adaptive online learning method. The method may normalize the input or output, for example, using a moving average method, remove data drift from the data, and generate transformed data such that the distribution of the input or output remains consistent over time. The method may provide the transformed data as input to an adaptive online learning algorithm and capture the changing relationships between variables by considering the variability of the process dynamics. Both operations can complement each other to solve different problems, such as chamber drift or concept drift. Chamber drift can generally occur when a change in the state of the chamber modifies the output of the sensor. Concept drift can generally modify the relationship between the sensor and the measured value. The time recognition normalization method herein can solve chamber drift to handle a drifting distribution. The adaptive online learning method herein can solve concept drift to handle the changing relationships. Time Recognition Normalization Method

[0117] The time-aware normalization method can reduce the effect of data drift on prediction accuracy. In a virtual measurement dataset associated with a semiconductor manufacturing process, features can drift slowly and steadily over time. Data drift can create a distortion between the input and output and reduce prediction accuracy over time. The time-aware normalization method can remove such changes and restore the relationship between features and targets. For example, FIGS. 11A-11D show how drifting inputs can obscure a linear relationship. In FIGS. 11A-11B, the feature drifts slowly away from its initial range while the response remains constant. As a result, the correlation between the input and output is null. Applying the time-aware normalization method to the feature can restore the linear relationship as illustrated in FIGS. 11C-11D.

[0118] The mean value, scale, sampling rate, slope, or any statistical value can change over time and affect the prediction performance of the virtual measurement model. Therefore, the time-aware normalization method can use different approaches. For example, differencing, smoothing, detrending, online estimation of various parameters, or time series decomposition methods may be required depending on the drifting data. The systems and methods described herein may, for example, use an exponentially weighted moving average and automatically transform both the input and output. By appropriately weighting each data point based on the observation time, the methods described herein can ensure that the input and output are consistent over time. After transforming the data through the time-aware normalization method, the transformed data can be provided as input data for an adaptive online learning method. Adaptive online learning method

[0119] The adaptive online learning method described in this specification can handle changes in the input-output relationship. The method can model those relationships as they change over time. The method may aggregate multiple online experts (e.g., regressors) by evaluating them based on their current and past performance. The method may emphasize the expert that functions best for any given time window by minimizing the adaptive regret. Adaptive regret generally measures, with hindsight, how well the algorithm functions compared to the optimal condition at all time intervals. At each point in time, the best regressor contributes the most to the prediction. The pseudo-code illustrated by the example of FIG. 12 can give an intuition regarding how individual experts contribute to the final prediction. At each time point t, the weight i of the expert varies based on its residual. A higher residual leads to a smaller weight relative to other experts. The experts may include, for example, linear models, Gaussian processes, decision tree regressors, or any online regressor. Selecting the correct regressor can be important for the performance of the adaptive online learning model, for example, from the perspective of speed or accuracy. To control the speed at which the ensemble adapts to changes, the decay parameter η can be adjusted based on the dynamics of the process.

[0120] FIGS. 13A-13C provide an example of an adaptive online learning method that models a simple univariate linear relationship. In FIG. 13A, y = a t x + b t where, for t ≦ 200, (a t , b t ) = (1, 5) and for t ≧ 200, (a t , b t) = (-1, 15). Assuming that the points after t = 200 are not observed during the test time, an offline algorithm trained on data from t ≤ 200 may provide inaccurate predictions. On the other hand, other online methods without adaptation may struggle to adapt after a sharp shift. Furthermore, the time-aware normalization method presented above can reduce the effect of the shifting intercept, but this may not be possible to handle the reversing coefficient, as illustrated in FIG. 13B. Since the model changes completely, an adaptive online learner model may be required. FIG. 13C compares the prediction performance of the linear regressor method, the online linear regressor method, and the adaptive online learning method of the present disclosure that uses an online linear regressor as the base expert. The method disclosed herein provides more accurate predictions than the linear regressor and the online linear regressor. The linear regressor does not adapt to model changes. The online linear regressor provides inappropriate predictions and does not reach the correct regime in time. Results

[0121] The systems and methods described herein were used to generate predictions regarding a chemical vapor deposition process using both synthetic data and actual data from a manufacturing process.

[0122] Synthetic data. The synthetic data included a dataset that reproduced the behavior of a carbon deposition process. The dataset shown in FIG. 14A included concatenated samples of five regression problems with 200 samples and 10 features. Each regression problem had different coefficients and intercepts. In this concatenated dataset, model changes occurred every 200 time steps to determine how fast each method adapted to data drift. Additionally, random data drift was introduced in each feature. FIG. 14B shows two features evolving over time. In this synthetic dataset, the features drift and the responses shift. This synthetic dataset can emulate data drift caused, for example, by maintenance, recipe changes, or calibration operations, and data drift resulting from, for example, equipment aging.

[0123] The systems and methods disclosed herein were compared to other virtual sensing methods that use the rolling ordinary least squares (OLS) method and the online linear method (also known as recursive least squares). The rolling OLS method can be parameterized by window size and stride. In each window, a linear model can be fitted. This model can make predictions until the window strides, a new model is estimated, and used for prediction. In the online linear method, the coefficients and intercept can always be updated when the response is observed. By default, these methods may not have a specific adaptation mechanism.

[0124] Each model was trained on a training dataset and tuned on an evaluation dataset. Performance was measured on a separate test dataset. The training and evaluation datasets covered the first 600 points. The test dataset contained the last 400 points. Structurally, the test dataset contains data drift and data shift that were not observed during training. Table 2 summarizes the coefficient of determination (R 2 ) and root mean square error (RMSE) for three methods with respect to the test dataset. The adaptive online time series method described herein improved the regression metrics by 24% and 48% in terms of R 2 and RMSE, respectively, compared to the rolling OLS method.

Table 2

[0125] Figures 14C - 14D compare the performance of the rolling OLS method, the online linear method, and the method disclosed herein. The adaptive online time series method described herein adapts to offsets much faster compared to the rolling OLS method. As shown in Figure 14C, at time stamps 600 and 800, the method described herein requires approximately 25 points to adapt to the data offset, while the rolling OLS method requires approximately 50 points. The online linear method cannot adapt to rapid data offsets. Figure 14D compares the absolute residuals of the three methods. The method disclosed herein limits the magnitude of the error immediately after the data offset. The mean absolute residual can reach up to a height of 2 for the rolling OLS method and the online linear method, but they remain near or below 1 for the method described herein. After approximately 50 - 60 points after the data offset, the rolling OLS method and the method described herein recover the underlying model and function similarly. The online linear method cannot adapt quickly enough and never reaches proper performance.

[0126] The adaptive online time series methods described herein are superior in performance to the rolling OLS method due to their adaptation speed. Window methods can be limited by their window size. When the window overlaps two inconsistent ranges of data, the performance can be affected. The aggregation of regressors fitted over different time ranges can reduce the impact of data offsets. The adaptive models described herein consistently train and update the experts, so that the naive related experts are ready as soon as an offset occurs.

[0127] Actual data. The systems and methods described herein (e.g., adaptive online time series methods) have also been deployed and integrated into the advanced process control and monitoring systems of large semiconductor manufacturers. The model covered four thin film depositions on a memory chip process using 72 deposition chambers. The memory chip process included a dynamic random access memory (DRAM) and a "not and" (NAND) process. By running the system in daily advanced process control operations over several months, the systems and methods described herein achieved at least a reduction of about 45.2% (an average reduction of about 21.5%) in the variation of film thickness and refractive index. Computing system

[0128] Referring to FIG. 17, a block diagram depicting an exemplary machine is shown that includes a computer system 1700 (e.g., a processing or computing system) in which a set of instructions can be executed to cause the device to implement or perform any one or more than one of the aspects and / or methodologies for static code scheduling of the present disclosure. The components of FIG. 17 are merely examples and do not limit the scope of use or functionality of any hardware, software, embedded logic component, or combination of two or more such components for implementing a particular embodiment.

[0129] Computer system 1700 may include one or more processors 1701, memory 1703, and storage device 1708 that communicate with each other and other components via bus 1740. Bus 1740 may also couple display 1732, one or more input devices 1733 (which may include, for example, a keypad, keyboard, mouse, stylus, etc.), one or more output devices 1734, one or more storage devices 1735, and various tangible storage media 1736. All of these elements may interface with bus 1740 directly or via one or more interfaces or adapters. For example, the various tangible storage media 1736 may interface with bus 1740 via storage media interface 1726. Computer system 1700 may have any suitable physical form, including, but not limited to, one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile phones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0130] Computer system 1700 includes one or more processors 1701 (e.g., a central processing unit (CPU) or a general-purpose graphics processing unit (GPGPU)) that execute functions. Processor 1701 optionally contains a cache memory unit 1702 for temporary local storage of instructions, data, or computer addresses. Processor 1701 is configured to assist in the execution of computer-readable instructions. Computer system 1700 may provide functionality for the components depicted in FIG. 17 as a result of processor 1701 executing non-transitory processor-executable instructions embodied in one or more tangible computer-readable storage media such as memory 1703, storage device 1708, storage device 1735, and / or storage medium 1736. The computer-readable medium may store software implementing a particular embodiment, and processor 1701 may execute the software. Memory 1703 may read software from one or more other computer-readable media (such as mass storage devices 1735, 1736, etc.) or from one or more other sources through a suitable interface such as network interface 1720. The software may cause processor 1701 to execute one or more processes described or illustrated herein, or one or more steps of one or more processes. Executing such a process or step may include defining a data structure stored in memory 1703 and modifying the data structure as directed by the software.

[0131] Memory 1703 may include various components (e.g., machine-readable media), including, but not limited to, random access memory components (e.g., RAM 1704) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM (registered trademark)), phase change random access memory (PRAM), etc.), read-only memory components (e.g., ROM 1705), and any combination thereof. ROM 1705 may act to communicate data and instructions unidirectionally to processor 1701, and RAM 1704 may act to communicate data and instructions bidirectionally with processor 1701. ROM 1705 and RAM 1704 may include any suitable tangible computer-readable media as described below. In one embodiment, a basic input / output system 1706 (BIOS) including basic routines that help transfer information between elements within computer system 1700 during startup and the like may be stored in memory 1703.

[0132] Fixed storage device 1708 is optionally connected bidirectionally to processor 1701 through storage device control unit 1707. Fixed storage device 1708 provides additional data storage capacity and may include any suitable tangible computer-readable media as described herein. Storage device 1708 may be used to store an operating system 1709, executable files 1710, data 1711, applications 1712 (application programs), and the like. Storage device 1708 may also include an optical disk drive, a solid state memory device (e.g., a flash-based system), or any combination of the above. Information within storage device 1708 may, if appropriate, be incorporated into memory 1703 as virtual memory.

[0133] In one embodiment, the memory device 1735 may be removably interfaced with the computer system 1700 via the memory device interface 1725 (e.g., via an external port connector (not shown)). In particular, the memory device 1735 and the associated machine-readable medium may provide non-volatile and / or volatile memory for machine-readable instructions, data structures, program modules, and / or other data for the computer system 1700. In one embodiment, software may reside, in whole or in part, within the machine-readable medium on the memory device 1735. In another embodiment, the software may reside, in whole or in part, within the processor 1701.

[0134] The bus 1740 connects a variety of subsystems. As used herein, the reference to a bus may, where appropriate, include one or more digital signal lines serving a common function. The bus 1740 may be any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Extended ISA (EISA) bus, Micro Channel Architecture (MCA) bus, Video Electronics Standards Association Local Bus (VLB), Peripheral Component Interconnect (PCI) bus, PCI Express (PCI-X) bus, Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, Serial Advanced Technology Attachment (SATA) bus, and any combination thereof.

[0135] Computer system 1700 may also include an input device 1733. In one embodiment, a user of computer system 1700 may enter commands and / or other information into computer system 1700 via input device 1733. Examples of input device 1733 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices (e.g., mice or touch pads), touch pads, touch screens, multi-touch screens, joysticks, styli, game pads, audio input devices (e.g., microphones, voice response systems, etc.), optical scanners, video or still image capture devices (e.g., cameras), and any combination thereof. In some embodiments, the input device is Kinect (登録商標) , Leap Motion (登録商標) , or the like. Input device 1733 may interface with bus 1740 via any of a variety of input interfaces 1723 (e.g., input interface 1723), including, but not limited to, serial, parallel, game port, USB, FIREWIRE®, THUNDERBOLT®, or any combination of the above.

[0136] In certain embodiments, when computer system 1700 is connected to network 1730, computer system 1700 may communicate with other devices, specifically, mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, that are connected to network 1730. Communication to and from computer system 1700 may be transmitted through network interface 1720. For example, network interface 1720 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1730, and computer system 1700 may store the incoming communications in memory 1703 for processing. Similarly, computer system 1700 may store outgoing communications (such as requests or responses to other devices) in memory 1703 that are in the form of one or more packets and are communicated from network interface 1720 to network 1730. Processor 1701 may access these communication packets stored in memory 1703 for processing.

[0137] Examples of network interface 1720 include, but are not limited to, network interface cards, modems, and any combination thereof. Examples of network 1730 or network segment 1730 include, but are not limited to, distributed computing systems, cloud computing systems, wide area networks (WANs) (such as the Internet, enterprise networks), local area networks (LANs) (such as networks associated with offices, buildings, campuses, or other relatively small geographical spaces), telephone networks, direct connections between two computing devices, peer-to-peer networks, and any combination thereof. Networks such as network 1730 may employ wired and / or wireless communication modes. Generally, any network topology may be used.

[0138] Information and data can be displayed through display 1732. Examples of display 1732 include, but are not limited to, cathode ray tube (CRT), liquid crystal display (LCD), thin film transistor liquid crystal display (TFT-LCD), organic liquid crystal display (OLED) such as passive matrix OLED (PMOLED) or active matrix OLED (AMOLED) display, plasma display, and any combination thereof. Display 1732 can interface with processor 1701, memory 1703, fixed storage device 1708, and other devices such as input device 1733 via bus 1740. Display 1732 is coupled to bus 1740 via video interface 1722, and the transport of data between display 1732 and bus 1740 can be controlled via graphic control 1721. In some embodiments, the display is a video projector. In some embodiments, the display is a head-mounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting example, HTC Vive (登録商標) , Oculus Rift (登録商標) , Samsung Gear VR (登録商標) , Microsoft HoloLens (登録商標) , Razer OSVR (登録商標) , FOVE VR (登録商標) , Zeiss VR One (登録商標) , Avegant Glyph (登録商標) , Freefly VR (登録商標) headsets, and equivalents. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0139] In addition to the display 1732, computer system 1700 may include one or more other peripheral output devices 1734 including, but not limited to, audio speakers, printers, memory devices, and any combination thereof. Such peripheral output devices may be connected to bus 1740 via output interface 1724. Examples of output interface 1724 include, but are not limited to, serial port, parallel connection, USB port, FIREWIRE® port, THUNDERBOLT® port, and any combination thereof.

[0140] In addition to, or alternatively, computer system 1700 may provide functionality as a result of logic that is wired in a circuit, otherwise embodied, or stored in a memory associated with a processor that operates instead of or in conjunction with software to perform one or more processes or one or more steps of one or more processes described or illustrated herein. References to software in this disclosure may include logic, references to logic may include software, and references to a computer-readable medium may include, where appropriate, a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both. The disclosure encompasses any suitable combination of hardware, software, or both.

[0141] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. For clarity of illustration of the interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described generally in terms of their functionality above.

[0142] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0143] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processors, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0144] According to the description in this specification, suitable computing devices include, as non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, subnotebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Selective televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the systems described in this specification. Suitable tablet computers include, in various embodiments, those with a booklet, slate, and convertible configuration.

[0145] In some embodiments, the computing device includes an operating system configured to execute executable instructions. The operating system is, for example, software including programs and data, which manages the device's hardware and provides services related to the execution of applications. Suitable server operating systems include, as non-limiting examples, FreeBSD (登録商標) , OpenBSD (登録商標) , NetBSD (登録商標) , Linux (登録商標) , Apple (登録商標) Mac OS X Server (登録商標) , Oracle Solaris (登録商標) , Windows Server (登録商標) , and Novell NetWare (登録商標) Suitable personal computer operating systems include, as non-limiting examples, Microsoft Windows (登録商標) , Apple Mac (登録商標) OS X, UNIX (registered trademark), and GNU / Linux (登録商標)including operating systems such as UNIX (registered trademark). In some embodiments, the operating system is provided by cloud computing. Suitable mobile smartphone operating systems include, by way of non-limiting example, Nokia Symbian (登録商標) OS, Apple (登録商標) iOS, Research In Motion BlackBerry (登録商標) OS, Google (登録商標) Android (登録商標) , Microsoft (登録商標) Windows Phone (登録商標) OS, Microsoft (登録商標) Windows Mobile (registered trademark) OS, Linux (登録商標) , and Palm (登録商標) WebOS. Suitable media streaming device operating systems include, by way of non-limiting example, Apple TV (登録商標) , Roku (登録商標) , Boxee (登録商標) , Google TV (登録商標) , Google Chromecast (登録商標) , Amazon Fire (登録商標) , and Samsung (登録商標) HomeSync (登録商標) . Suitable video game console operating systems include, by way of non-limiting example, Sony (登録商標) PS3 (登録商標) , Sony (登録商標) PS4 (登録商標) , Microsoft (登録商標) Xbox 360 (登録商標) , Microsoft Xbox One (登録商標) , Nintendo Wii (登録商標) , Nintendo Wii U (登録商標) , and Ouya (登録商標) . Suitable virtual reality headset systems include, by way of non-limiting example, Meta Oculus (登録商標) . Non-transitory computer-readable storage medium

[0146] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer-readable storage media encoded with a program comprising instructions executable by an operating system of a networked computing device, optionally. In further embodiments, the computer-readable storage media is a tangible component of the computing device. In still further embodiments, the computer-readable storage media is optionally removable from the computing device. In some embodiments, the computer-readable storage media includes, by way of non-limiting example, CD-ROMs, DVDs, flash memory devices, solid state memories, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the program and instructions are encoded on the media permanently, substantially permanently, semi-permanently, or non-transitorily. Computer program

[0147] In some embodiments, the platforms, systems, media, and methods disclosed herein include at least one computer program or the use thereof. The computer program includes a sequence of instructions executable by one or more processors of a CPU of a computing device and described to perform a specified task. The computer-readable instructions may be implemented as program modules such as functions, objects, application programming interfaces (APIs), computing data structures, and the like that perform a particular task or implement a particular abstract data type. In light of the disclosure provided herein, the computer program may be described in various versions of various languages.

[0148] The functionality of the computer-readable instructions may be combined or distributed as desired in various environments. In some embodiments, the computer program includes one sequence of instructions. In some embodiments, the computer program includes multiple sequences of instructions. In some embodiments, the computer program is provided from one location. In other embodiments, the computer program is provided from multiple locations. In various embodiments, the computer program includes one or more software modules. In various embodiments, the computer program includes, in whole or in part, one or more web applications, one or more mobile applications, one or more stand-alone applications, one or more web browser plugins, extensions, add-ins, or add-ons, or combinations thereof. Web application

[0149] In some embodiments, the computer program includes a web application. In light of the disclosure provided herein, a web application utilizes, in various embodiments, one or more software frameworks and one or more database systems. In some embodiments, the web application is created on a software framework such as Microsoft (登録商標) .NET or Ruby on Rails (登録商標) (RoR). In some embodiments, the web application utilizes one or more database systems including, by way of non-limiting example, relational, non-relational, object-oriented, associative, and XML database systems. In further embodiments, suitable relational database systems include, by way of non-limiting example, Microsoft (登録商標) Structured Query Language (SQL) Server, mySQL TM , and Oracle (登録商標)It includes. Web applications are described in one or more versions of one or more languages in various embodiments. A web application may be described in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, a web application is described to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or Extensible Markup Language (XML). In some embodiments, a web application is described to some extent in a presentation definition language such as Cascading Style Sheets (CSS). In some embodiments, a web application is described to some extent in a client-side scripting language such as Asynchronous Javascript and XML (登録商標) (AJAX), Flash Actionscript, Javascript (registered trademark), or Silverlight (登録商標) In some embodiments, a web application is described to some extent in a client-side scripting language such as Asynchronous Javascript and XML, Flash Actionscript, Javascript (registered trademark), or Silverlight. In some embodiments, a web application is described to some extent in a server-side coding language such as Active Server Pages (登録商標) (ASP), ColdFusion (登録商標) , Perl (登録商標) , Java (registered trademark), JavaServer Pages (登録商標) (JSP), Hypertext Preprocessor (登録商標) (PHP), Python (登録商標) , Ruby (登録商標) , Tcl (登録商標) , Smalltalk (登録商標) , WebDNA (登録商標) , or Groovy (登録商標) In some embodiments, a web application is described to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion, Perl, Java (registered trademark), JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python, Ruby, Tcl, Smalltalk, WebDNA, or Groovy. In some embodiments, a web application is described to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, a web application is described to some extent in IBM Lotus Domino (登録商標)Integrate enterprise server products such as. In some embodiments, the web application includes a media player element. In various further embodiments, the media player element utilizes one or more than one of many suitable multimedia technologies including, by way of non-limiting example, Adobe (登録商標) Flash (登録商標) 、HTML 5、Apple (登録商標) QuickTime (登録商標) 、Microsoft Silverlight (登録商標) 、Java (registered trademark), and Unity (登録商標) among others.

[0150] Referring to FIG. 18, in certain embodiments, the application provisioning system includes one or more databases 1800 accessed by a database management system (DBMS) 1810. Suitable DBMSs include Firebird (登録商標) 、MySQL (登録商標) 、NoSQL (登録商標) 、PostgreSQL (登録商標) 、SQLite (登録商標) 、Oracle Database (登録商標) 、Microsoft SQL Server (登録商標) 、IBM DB2 (登録商標) 、IBM Informix (登録商標) 、SAP Sybase (登録商標) 、SAP Sybase (登録商標) 、Teradata (登録商標) 、PostGIS (登録商標) 、Apache (登録商標) Hive, Apache (登録商標) Impala, time series databases, graph databases, key-value storage devices, and equivalents. In this embodiment, the application provisioning system further includes one or more application servers 1820 (Java (registered trademark) servers,.NET (登録商標) servers, PHP (登録商標) servers, and equivalents etc.) and one or more web servers 1830 (Apache (登録商標) 、IIS(登録商標) and GWS (登録商標) and equivalents, etc.). The web server optionally publishes one or more web services via an application programming interface (API) 1840. Via a network such as the Internet, the system provides a browser-based and / or mobile native user interface. In some cases, the DBMS may be a relational DBMS.

[0151] Referring to FIG. 19, in certain embodiments, the application provisioning system alternatively has a distributed cloud-based architecture 1900, including autoscaling web server resources 1910 and application server resources 1920 that are elastically load-balanced, and a database 1930 that is replicated synchronously. Mobile application

[0152] In some embodiments, the computer program includes a mobile application provided to a mobile computing device. In some embodiments, the mobile application is provided to the mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to the mobile computing device via the computer network described herein.

[0153] In view of the disclosure provided herein, mobile applications are created by techniques using hardware, languages, and development environments. Mobile applications are described in several languages. Suitable programming languages include, by way of non-limiting example, C, C++, C#, Objective-C, Java®, JavaScript®, Pascal (登録商標) Object Pascal (登録商標) Python TM Ruby (登録商標) VB.NET (登録商標) WML(登録商標) including XHTML / HTML with or without CSS, or combinations thereof.

[0154] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non-limiting example, AirplaySDK (登録商標) alcheMo (登録商標) Appcelerator (登録商標) Celsius (登録商標) Bedrock (登録商標) Flash Lite (登録商標) .NET Compact Framework (登録商標) Rhomobile (登録商標) and WorkLight Mobile Platform (登録商標) including. By way of non-limiting example, Lazarus (登録商標) MobiFlex (登録商標) MoSync (登録商標) and Phonegap (登録商標) Other development environments are also available at no cost. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting example, iPhone (登録商標) and iPad (登録商標) (iOS) SDK, Android (登録商標) SDK, BlackBerry (登録商標) SDK, BREW SDK, Palm (登録商標) OS SDK, Symbian (登録商標) SDK, webOS (登録商標) SDK, and Windows (登録商標) Mobile SDK.

[0155] By way of non-limiting example, Apple (登録商標) App Store, Google (登録商標) Play, Chrome (登録商標) WebStore, BlackBerry (登録商標) App World, App Store for Palm devices (登録商標), App Catalog for webOS (登録商標) , Windows for Mobile (登録商標) Marketplace, Nokia (登録商標) Ovi Store for Devices, Samsung (登録商標) Apps, and Nintendo (登録商標) Several commercial sources, including the DSi Shop, are available for the distribution of mobile applications. Standalone applications

[0156] In some embodiments, the computer program includes standalone applications that are not add-ons to existing processes, e.g., not plug-ins, but rather programs that are executed as independent computer processes. Standalone applications are often compiled. A compiler is a computer program that converts source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting example, C, C++, Objective-C (登録商標) , COBOL (登録商標) , Delphi (登録商標) , Eiffel (登録商標) , Java (registered trademark), Lisp (登録商標) , Python (登録商標) , Visual Basic (登録商標) , and VB.NET (登録商標) , or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, the computer program includes one or more executable compiled applications. Additionally, microservices related to Python (登録商標) and JavaScript (registered trademark) may be used. Web browser plug-ins

[0157] In some embodiments, the computer program includes a web browser plugin (e.g., a web extension, etc.). In computing, a plugin is one or more software components that add specific functionality to a larger software application. Software application makers support plugins to enable third-party developers to create the ability to extend the application, support the easy addition of new features, and reduce the size of the application. Once supported, the plugin enables customization of the functionality of the software application. For example, plugins are commonly used in web browsers to play videos, generate interactivity, scan for viruses, and display certain file types. Some web browser plugins include Adobe Flash Player (登録商標) , Microsoft Silverlight (登録商標) , and Apple QuickTime (登録商標) . In some embodiments, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some embodiments, the toolbar comprises one or more explorer bars, toolbands, or deskbands.

[0158] In view of the disclosure provided herein, by way of non-limiting example, several plugin frameworks are available that enable the development of plugins in various programming languages, including C++, Delphi (登録商標) , Java®, PHP (登録商標) , Python (登録商標) , and VB.NET (登録商標) , or combinations thereof.

[0159] A web browser (also known as an Internet browser) is a software application designed for use in conjunction with a network-connected computing device to read, present, and traverse information resources on the World Wide Web. Suitable web browsers include, by way of non-limiting example, Microsoft Internet Explorer (登録商標) , Mozilla Firefox (登録商標) , Google Chrome (登録商標) , Apple Safari (登録商標) , Opera Software Opera (登録商標) , and KDE Konqueror (登録商標) . In some embodiments, the web browser is a mobile web browser. A mobile web browser (also known as a microbrowser, mini-browser, and wireless browser) is designed for use on a mobile computing device, including, by way of non-limiting example, a handheld computer, tablet computer, netbook computer, subnotebook computer, smartphone, music player, personal digital assistant (PDA), and handheld video game system. Suitable mobile web browsers include, by way of non-limiting example, Google Android (登録商標) Browser, RIM BlackBerry (登録商標) Browser, Apple Safari (登録商標) , Palm Blazer (登録商標) , Palm WebOS (登録商標) Browser, Mozilla Firefox for Mobile (登録商標) , Microsoft Internet Explorer Mobile (登録商標) , Amazon Kindle Basic Web (登録商標) , Nokia Browser (登録商標) , Opera Software Opera Mobile (登録商標) , and Sony PSP (登録商標) Browser. Software module

[0160] In some embodiments, the platforms, systems, media, and methods disclosed herein include software, servers, and / or database modules, or the use thereof. In view of the disclosure provided herein, software modules are created by techniques using machines, software, and languages. The software modules disclosed herein are implemented in a number of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, or a combination thereof. In further various embodiments, a software module comprises multiple files, multiple sections of code, multiple programming objects, multiple programming structures, or a combination thereof. In various embodiments, one or more software modules comprise, by way of non-limiting example, web applications, mobile applications, and stand-alone applications. In some embodiments, a software module is within one computer program or application. In other embodiments, a software module is within more than one computer program or application. In some embodiments, a software module is hosted on one machine. In other embodiments, a software module is hosted on more than one machine. In further embodiments, a software module is hosted on a distributed computing platform such as a cloud computing platform. In some embodiments, a software module is hosted on one or more machines in one location. In other embodiments, a software module is hosted on one or more machines in more than one location. Database

[0161] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more databases (DBs) or the use thereof. In view of the disclosure provided herein, many databases are suitable for storing and retrieving data. In various embodiments, suitable databases include, by way of non-limiting example, relational databases, non-relational databases, object-oriented databases, object databases, entity-relationship model databases, associative databases, XML databases, time-series databases, graph databases, and the like. Further non-limiting examples include SQL, PostgreSQL (登録商標) , MySQL (登録商標) , Oracle (登録商標) , DB2 (登録商標) , and Sybase. In some embodiments, the database is Internet-based. In further embodiments, the database is web-based. In still further embodiments, the database is cloud-computing-based. In certain embodiments, the database is a distributed database. In other embodiments, the database is based on one or more local computer storage devices. Terms and Definitions

[0162] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0163] As used herein, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Any reference to "or" herein is intended to encompass "and / or" unless otherwise stated.

[0164] As used herein, the term "about" refers to an amount that, in some cases, is described as being approximately the amount.

[0165] As used herein, the term "about" refers to an amount that is within 10%, 5%, or 1% (including increments therein) of the recited amount.

[0166] As used herein, the term "about" in relation to a percentage refers to an amount that is 10%, 5%, or 1% (including increments therein) more or less than the recited percentage.

[0167] As used herein, the phrases "at least one", "one or more", and "and / or" are non-limiting expressions that are both conjunctive and disjunctive in operation. For example, the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", and "A, B, and / or C" each mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0168] Preferred embodiments of the present disclosure are shown and described herein, but such embodiments are provided only as examples. The present disclosure is not intended to be limited by the specific examples provided within this specification. The present disclosure is described with reference to the foregoing specification, but the description and illustration of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, modifications, and substitutions can occur without departing from the present disclosure. Further, it should be understood that all aspects of the present disclosure are not limited to the specific descriptions, configurations, or relative proportions described herein, which depend on various conditions and variables. It should be understood that various alternatives of the embodiments of the disclosure described herein can be employed in practicing the present disclosure. Accordingly, it is contemplated that the present disclosure also encompasses any such alternatives, modifications, variations, or equivalents. The following claims define the scope of the present disclosure, and it is intended that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

1. 1. A system for process monitoring and control, comprising: a data processing module configured to receive and process a plurality of data sets from a plurality of different sources to generate training data; a training and optimization module configured to provide the training data to a machine learning (ML) pipeline to train and optimize a virtual metrology (VM) model, the ML pipeline comprising at least an adaptive online time series model configured to capture changing relationships in the multiple datasets to detect drift, shifts, or deviations; an inference module configured to use the VM model to generate, in substantially real time, one or more predictive metrics based at least on process data received by the inference module and provided to the VM model, the one or more predictive metrics usable to characterize an output of a process implemented by a controllable process device using at least one control parameter; and a process control module configured to use the one or more predictive metrics to at least: (i) detect the deviation in the changing relationship, the deviation being an abrupt change in the changing relationship; and (ii) correct or mitigate the deviation by modifying the at least one control parameter of the process equipment. wherein the plurality of data sets includes at least one of: (1) historical process data; (2) current process data; (3) historical measurement data for the one or more predictive metrics; (4) current measurement data for the one or more predictive metrics; (5) operational data for the process equipment; or (6) equipment specification data.

2. The system of claim 1, wherein the process data is received from the process equipment substantially in real time as the process is performed.

3. 2. The system of claim 1, wherein the inference module is configured to provide the one or more predictive metrics to correct or mitigate the deviation through at least one of process control, process monitoring, refinement, or troubleshooting.

4. The system of claim 1 , wherein the process control module is configured to use the one or more predictive metrics to improve process productivity through integration with run-to-run control.

5. 10. The system of claim 1, further comprising: said process equipment, said process equipment comprising semiconductor process equipment.

6. The system of claim 1 , wherein an output of the process comprises a deposited or fabricated structure.

7. The system of claim 6 , wherein the deposited or processed structure comprises a film, a layer, or a substrate.

8. The system of claim 7 , wherein the one or more predictive metrics comprise one or more dimensions or properties of the film, the layer, or the substrate.

9. The system of claim 1 , wherein the system is configured to be used or deployed in a manufacturing environment.

10. The system of claim 1 , wherein the data processing module is further configured to validate the historical process data and the historical measurement data against the operational data and the equipment specification data.

11. The system of claim 1 , wherein the multiple sources comprise a database configured to store at least the historical process data or the historical measurement data.

12. The system of claim 1 , wherein the multiple sources comprise a database or a log configured to store at least the operational data or the equipment specification data.

13. The system of claim 1 , wherein the plurality of sources comprises the process equipment.

14. The system of claim 1 , wherein the plurality of sources comprises a measurement device configured to collect the current measurement data.

15. The system of claim 1 , wherein the data processing module is further configured to receive and process the plurality of data sets by generating a component hierarchy for the process equipment.

16. 16. The system of claim 15, wherein the component hierarchy comprises a nesting structure of (i) the process equipment and (ii) one or more components used within or in conjunction with the process equipment.

17. The system of claim 16 , wherein the one or more components comprise one or more sub-equipment including a chamber, a station, and / or a sensor.

18. The system of claim 1 , wherein the data processing module is further configured to receive and process the plurality of data sets by generating a recipe step operation hierarchy for the process.

19. 20. The system of claim 18, wherein the recipe comprises a plurality of steps, each step of the plurality of steps comprising a plurality of different sub-operations.

20. The system of claim 1 , wherein the data processing module is further configured to receive and process the plurality of data sets by removing one or more data outliers.

21. 3. The system of claim 2, wherein the data processing module is further configured to pre-process and remove data outliers from the process data before the process data is input to the VM model within the inference module.

22. The system of claim 1 , wherein the training data is continuously updated with the current process data and the current measurement data.

23. 10. The system of claim 1, wherein the machine learning pipeline further comprises one or more of: (i) feature engineering; and (ii) time-aware data normalization.

24. 24. The system of claim 23, wherein the machine learning pipeline is configured to apply the training data through the one or more components sequentially or simultaneously.

25. 24. The system of claim 23, wherein the feature engineering comprises extraction of features from raw trace data or sensor data in the training data.

26. 24. The system of claim 23, wherein the feature engineering comprises use of an algorithm to select one or more features from a list of extracted features based, at least in part, on local relationships between inputs and outputs of the VM model.

27. 24. The system of claim 23, wherein the time-aware data normalization comprises a time-aware data normalization model for decomposition of time series data into one or more components including smoothed data, trended data, and / or detrended data.

28. 28. The system of claim 27, wherein an output of the time-aware data normalization model is provided to the adaptive online time series model to capture the changing relationships in the multiple data sets.

29. 24. The system of claim 23, wherein the adaptive online time series model is configured as an adaptive online ensemble learning algorithm.

30. 10. The system of claim 1, wherein the training and optimization module is further configured to optimize the VM model, at least in part, using hyperparameter optimization.

31. 31. The system of claim 23, wherein the training and optimization module is further configured to: (i) train the VM model with a given set of hyper-parameters on output from the machine learning pipeline.

32. 32. The system of claim 31, wherein the training and optimization module is further configured to: (ii) evaluate performance of the VM model based on validation data.

33. 33. The system of claim 32, wherein the validation data is split from the training data for the hyper-parameter optimization.

34. 33. The system of claim 32, wherein the training and optimization module is further configured to: (iii) use a hyperparameter optimization algorithm to select a set of hyperparameters for a next iteration based on past performance so as to increase or improve performance of the VM model.

35. 35. The system of claim 34, wherein the training and optimization module is further configured to iteratively repeat (i)-(iii) until the performance of the VM model meets a termination criterion.

36. 1. A method for process monitoring and control, comprising: (a) receiving and processing a plurality of datasets from a plurality of different sources to generate training data; (b) providing the training data to a machine learning (ML) pipeline to train and optimize a virtual metrology (VM) model, the ML pipeline comprising at least an adaptive online time series model to capture changing relationships in the multiple datasets to detect drift, shifts, or deviations; and (c) the VM model generating, in substantially real time, one or more predictive metrics based at least on process data received and provided to the VM model, the one or more predictive metrics usable to characterize an output of a process implemented by a controllable process device using at least one control parameter; and (d) using the one or more predictive metrics to: (i) detect at least the deviation in the changing relationship, the deviation being an abrupt change in the changing relationship; and (ii) correct or mitigate the deviation by modifying the at least one control parameter of the process equipment. wherein the plurality of data sets includes at least one of: (1) historical process data; (2) current process data; (3) historical measurement data for the one or more predictive metrics; (4) current measurement data for the one or more predictive metrics; (5) operational data for the process equipment; or (6) equipment specification data.

Images