System and method for process monitoring and control

Abstract

To provide a system and a method for advanced process control and monitoring.SOLUTION: A system and a method may be associated with a data processing module configured to receive and process multiple data types and data sets from multiple different sources to generate training data, a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model, and an inference module configured to use the model to generate one or more predictive metrics in substantially real time, in which the one or more predictive metrics is usable to evaluate characteristics of an output of a process performed by a process device.SELECTED DRAWING: None

Description

[Background technology]

[0001] Some virtual metrology systems and methods can model processes and characteristics associated with monitoring and controlling manufacturing processes, e.g., wafer processing in the semiconductor industry. Statistically modeling processes and characteristics may use, in part, measurements from current or historical data, e.g., current or historical sensor data. Virtual metrology systems and methods may provide more accurate measurements, for example, for controlling manufacturing processes. Virtual metrology systems and methods may increase productivity, improve quality, or reduce maintenance costs, for example, when compared to physical inspection of manufacturing processes using traditional metrology. For example, virtual metrology systems and methods may be able to sample all or substantially all units (e.g., semiconductor wafers) in a manufacturing process, whereas a human operator using traditional metrology may only be able to sample a small fraction of units. Virtual metrology systems and methods may use, in part, machine learning methods to predict process variables or target properties associated with a manufacturing process. Summary of the Invention [Means for solving the problem]

[0002] Disclosed herein are systems and methods for continuous evolution in advanced process control and monitoring that can improve predictive performance of a process variable or target property.

[0003] In one aspect, disclosed are systems and methods for process monitoring and control. The system may include, for example, a data processing module configured to receive and process multiple data types and data sets from multiple different sources to generate training data, a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model, and an inference module configured to use the model to generate one or more predictive metrics in substantially real time, the one or more predictive metrics usable to characterize an output of a process performed by the process equipment.

[0004] In some embodiments, the inference module is configured to receive and provide process data to the model to generate one or more predictive metrics, the process data being received from the process equipment in substantially real time as the process is performed.

[0005] In some embodiments, the inference module is configured to provide one or more predictive metrics for process control or for process monitoring, improvement, or troubleshooting.

[0006] In some embodiments, the system further comprises a process control module configured to use one or more predictive metrics to detect drift, deviation, or excursion in a process or process equipment.

[0007] In some embodiments, the process control module is configured to use one or more predictive metrics to correct or mitigate drift, deviation, or excursion in a process or process equipment.

[0008] In some embodiments, the process control module is configured to use one or more predictive metrics to improve process productivity through integration with inter-run control.

[0009] In some embodiments, the model comprises a virtual measurement (VM) model.

[0010] In some embodiments, the system further comprises process equipment, wherein the process equipment comprises semiconductor process equipment.

[0011] In some embodiments, the output of the process comprises a deposited or fabricated structure.

[0012] In some embodiments, the deposited or processed structure comprises a film, layer, or substrate.

[0013] In some embodiments, the one or more predictive metrics comprise one or more dimensions or properties of the film, layer, or substrate.

[0014] In some embodiments, the system is configured to be used or deployed in a manufacturing environment.

[0015] In some embodiments, the multiple data types and data sets comprise: (1) historical process data, (2) current process data, (3) historical measurement data of one or more predictive metrics, (4) current measurement data of one or more predictive metrics, (5) operational data, and / or (6) equipment specification data.

[0016] In some embodiments, the data processing module is configured to validate the historical process data and historical measurement data against the operational data and equipment specification data.

[0017] In some embodiments, the plurality of sources comprises a database configured to store at least the historical process data or the historical measurement data.

[0018] In some embodiments, the plurality of sources comprises a database or a log configured to store at least operational data or equipment specification data.

[0019] In some embodiments, the multiple sources comprise process equipment.

[0020] In some embodiments, the plurality of sources comprises measurement equipment configured to collect current measurement data.

[0021] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by generating a component hierarchy of the process equipment.

[0022] In some embodiments, the component hierarchy comprises a nested structure of (i) process equipment and (ii) one or more components used within or in conjunction with the process equipment.

[0023] In some embodiments, the one or more components comprise one or more sub-instruments including a chamber, a station, and / or a sensor.

[0024] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by generating a recipe step operation hierarchy for the process.

[0025] In some embodiments, a recipe comprises multiple steps, each step of the multiple steps comprising multiple different sub-operations.

[0026] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by removing one or more data outliers.

[0027] In some embodiments, the data processing module is configured to pre-process and remove data outliers from the process data before the process data is input into a model in the inference module.

[0028] In some embodiments, the training data is continuously updated with current process data and current measurement data.

[0029] In some embodiments, the machine learning pipeline comprises two or more components from a plurality of components consisting of: (i) feature engineering, (ii) time-aware data normalization, and / or (iii) adaptive learning algorithms.

[0030] In some embodiments, the machine learning pipeline is configured to apply training data through two or more components, either sequentially or simultaneously.

[0031] In some embodiments, feature engineering comprises the extraction of multiple features from raw trace data or sensor data in the training data.

[0032] In some embodiments, feature engineering comprises the use of an algorithm to select one or more features from a list of extracted features based, at least in part, on local relationships between the inputs and outputs of the model.

[0033] In some embodiments, time-aware data normalization comprises decomposition of time series data into one or more components, including smoothed data, trended data, and / or detrended data.

[0034] In some embodiments, the time-aware data normalization is based on a model and the data type of the model.

[0035] In some embodiments, the adaptive learning algorithm is an adaptive online ensemble learning algorithm.

[0036] In some embodiments, the training and optimization module is configured to optimize the model, at least in part, using hyperparameter optimization.

[0037] In some embodiments, the training and optimization module is configured to (i) train a model with a given set of hyperparameters on output from the machine learning pipeline.

[0038] In some embodiments, the training and optimization module is further configured to (ii) evaluate the performance of the model based on the validation data.

[0039] In some embodiments, the validation data is split from the training data for hyperparameter optimization.

[0040] In some embodiments, the training and optimization module is further configured to (iii) use a hyperparameter optimization algorithm to select a set of hyperparameters for the next iteration based on past performance so as to increase or improve the performance of the model.

[0041] In some embodiments, the training and optimization module is further configured to iteratively repeat (i)-(iii) until the model's performance meets a termination criterion.

[0042] In another aspect, disclosed is a method for process monitoring and control. The method may include, for example, (a) receiving and processing multiple data types and data sets from multiple different sources to generate training data, (b) providing the training data to a machine learning pipeline to train and optimize a model, and (c) generating one or more predictive metrics in substantially real time, the one or more predictive metrics usable to characterize an output of a process performed by process equipment.

[0043] Additional aspects and advantages of the present disclosure will become readily apparent from the following detailed description, in which only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive. The present invention provides, for example, the following items. (Item 1) 1. A system for process monitoring and control, comprising: a data processing module configured to receive and process a plurality of data types and data sets from a plurality of different sources to generate training data; a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model; and an inference module configured to use the model to generate, in substantially real time, one or more predictive metrics usable to characterize an output of a process performed by the process equipment; and A system comprising: (Item 2) 10. The system of claim 9, wherein the inference module is configured to receive and provide process data to the model to generate the one or more predictive metrics, the process data being received from the process equipment in substantially real time as the process is performed. (Item 3) 10. The system of claim 9, wherein the inference module is configured to provide the one or more predictive metrics for the process control or for process monitoring, improvement, or troubleshooting. (Item 4) 10. The system of claim 1, further comprising a process control module configured to use the one or more predictive metrics to detect drift, deviation, or excursion in the process or the process equipment. (Item 5) 10. The system of claim 9, wherein the process control module is configured to use the one or more predictive metrics to correct or mitigate the drift, deviation, or excursion in the process or the process equipment. (Item 6) 10. The system of claim 9, wherein the process control module is configured to use the one or more predictive metrics to improve process productivity through integration with inter-run control. (Item 7) 10. The system of claim 1, wherein the model comprises a virtual measurement (VM) model. (Item 8) 10. The system of claim 1, further comprising the process equipment, the process equipment comprising semiconductor process equipment. (Item 9) 10. The system of claim 1, wherein the output of the process comprises a deposited or fabricated structure. (Item 10) 10. The system of claim 1, wherein the deposited or processed structure comprises a film, layer, or substrate. (Item 11) 2. The system of claim 1, wherein the one or more predictive metrics comprise one or more dimensions or properties of the film, the layer, or the substrate. (Item 12) 10. The system of claim 1, wherein the system is configured to be used or deployed in a manufacturing environment. (Item 13) 10. The system of claim 9, wherein the plurality of data types and data sets comprises: (1) historical process data; (2) current process data; (3) historical measurement data of the one or more predictive metrics; (4) current measurement data of the one or more predictive metrics; (5) operational data; and / or (6) equipment specification data. (Item 14) 10. The system of claim 9, wherein the data processing module is configured to validate the historical process data and the historical measurement data against the operational data and the equipment specification data. (Item 15) 10. The system of claim 9, wherein the plurality of sources comprises a database configured to store at least the historical process data or the historical measurement data. (Item 16) 2. The system of claim 1, wherein the plurality of sources comprises a database or log configured to store at least the operational data or the equipment specification data. (Item 17) 10. The system of claim 9, wherein the plurality of sources comprises the process equipment. (Item 18) 10. The system of claim 9, wherein the plurality of sources comprises measurement equipment configured to collect the current measurement data. (Item 19) 10. The system of claim 9, wherein the data processing module is configured to receive and process the plurality of data types or data sets by generating a component hierarchy of the process equipment. (Item 20) 10. The system of claim 9, wherein the component hierarchy comprises a nested structure of (i) the process equipment and (ii) one or more components used within or in conjunction with the process equipment. (Item 21) 10. The system of claim 9, wherein the one or more components comprise one or more sub-instruments including a chamber, a station, and / or a sensor. (Item 22) The system of any one of the preceding items, wherein the data processing module is configured to receive and process the plurality of data types or data sets by generating a recipe step operation hierarchy for the process. (Item 23) 10. The system of claim 9, wherein the recipe comprises a plurality of steps, each of the plurality of steps comprising a plurality of different sub-operations. (Item 24) 10. The system of claim 1, wherein the data processing module is configured to receive and process the plurality of data types or data sets by removing one or more data outliers. (Item 25) 10. The system of claim 9, wherein the data processing module is configured to pre-process and remove data outliers from the process data before the process data is input into the model within the inference module. (Item 26) 10. The system of claim 9, wherein the training data is continuously updated with the current process data and the current measurement data. (Item 27) 10. The system of claim 9, wherein the machine learning pipeline comprises two or more components from a plurality of components consisting of: (i) feature engineering, (ii) time-aware data normalization, and / or (iii) an adaptive learning algorithm. (Item 28) 10. The system of claim 1, wherein the machine learning pipeline is configured to apply the training data through the two or more components sequentially or simultaneously. (Item 29) 10. The system of claim 1, wherein the feature engineering comprises extraction of a plurality of features from raw trace data or sensor data in the training data. (Item 30) 10. The system of claim 9, wherein the feature engineering comprises the use of an algorithm to select one or more features from a list of extracted features based, at least in part, on local relationships between inputs and outputs of the model. (Item 31) 10. The system of claim 1, wherein the time-aware data normalization comprises decomposition of the time series data into one or more components including smoothed data, trended data, and / or detrended data. (Item 32) 10. The system of claim 9, wherein the time-aware data normalization is based on a model and a data type of the model. (Item 33) 10. The system of claim 9, wherein the adaptive learning algorithm is an adaptive online ensemble learning algorithm. (Item 34) 10. The system of claim 1, wherein the training and optimization module is configured to optimize the model, at least in part, using hyperparameter optimization. (Item 35) 10. The system of claim 9, wherein the training and optimization module is configured to (i) train the model using a given set of hyperparameters on output from the machine learning pipeline. (Item 36) 2. The system of claim 1, wherein the training and optimization module is further configured to (ii) evaluate the performance of the model based on validation data. (Item 37) 10. The system of claim 1, wherein the validation data is split from the training data for the hyperparameter optimization. (Item 38) 3. The system of claim 1, wherein the training and optimization module is further configured to (iii) use a hyperparameter optimization algorithm to select a set of hyperparameters for a next iteration based on past performance so as to increase or improve performance of the model. (Item 39) 10. The system of claim 1, wherein the training and optimization module is further configured to iteratively repeat (i)-(iii) until the model's performance meets a termination criterion. (Item 40) 1. A method for process monitoring and control, comprising: (a) receiving and processing a plurality of data types and data sets from a plurality of different sources to generate training data; (b) providing the training data to a machine learning pipeline to train and optimize a model; (c) generating one or more predictive metrics in substantially real time, the one or more predictive metrics usable to characterize an output of a process performed by the process equipment; A method comprising: (Summary) Systems and methods for advanced process control and monitoring are described. The systems and methods may be associated with a data processing module configured to receive and process multiple data types and data sets from multiple different sources to generate training data, a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model, and an inference module configured to use the model to generate one or more predictive metrics in substantially real time, the one or more predictive metrics usable to characterize an output of a process performed by process equipment.

[0044] (Incorporated by reference) All publications, patents, and patent applications mentioned herein are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, it is intended that the present specification supersede and / or supersede any such conflicting material. [Brief explanation of the drawings]

[0045] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings.

[0046] [Figure 1] FIG. 1 illustrates a non-limiting example of a system for adaptive online time series prediction of the present disclosure.

[0047] [Figure 2] FIG. 2 illustrates a non-limiting example of the disclosed method for adaptive online time series prediction.

[0048] [Figure 3] FIG. 3 illustrates a non-limiting example of an advantage (eg, sampling) of the present disclosure.

[0049] [Figure 4A] 4A-4B illustrate non-limiting examples of equipment, processes, and process variables in a semiconductor manufacturing process of the present disclosure. [Figure 4B] 4A-4B illustrate non-limiting examples of equipment, processes, and process variables in a semiconductor manufacturing process of the present disclosure.

[0050] [Figure 5] FIG. 5 illustrates a non-limiting example of the steps for preparing hierarchically structured data of the present disclosure.

[0051] [Figure 6] FIG. 6 illustrates a non-limiting example of the steps of extracting features from raw data and generating clean data of the present disclosure.

[0052] [Figure 7] FIG. 7 illustrates a non-limiting example of the feature engineering and model training steps of the present disclosure.

[0053] [Figure 8] FIG. 8 illustrates a non-limiting example of the feature engineering and model training steps of the present disclosure.

[0054] [Figure 9]FIG. 9 illustrates a non-limiting example of comparing methods of the present disclosure (eg, local vs. global characterization methods).

[0055] [Figure 10] FIG. 10 illustrates a non-limiting example of a local characterization method (eg, a sliding window method) of the present disclosure.

[0056] [Figure 11A] 11A-11D illustrate non-limiting examples of the disclosed time-aware normalization method. FIG. 11A illustrates drifting features and responses over time. FIG. 11B illustrates drifting features vs. responses. FIG. 11C illustrates time-aware normalized features and responses over time. FIG. 11D illustrates time-aware normalized features vs. responses. [Figure 11B] 11A-11D illustrate non-limiting examples of the disclosed time-aware normalization method. FIG. 11A illustrates drifting features and responses over time. FIG. 11B illustrates drifting features vs. responses. FIG. 11C illustrates time-aware normalized features and responses over time. FIG. 11D illustrates time-aware normalized features vs. responses. [Figure 11C] 11A-11D illustrate non-limiting examples of the disclosed time-aware normalization method. FIG. 11A illustrates drifting features and responses over time. FIG. 11B illustrates drifting features vs. responses. FIG. 11C illustrates time-aware normalized features and responses over time. FIG. 11D illustrates time-aware normalized features vs. responses. [Figure 11D] 11A-11D illustrate non-limiting examples of the disclosed time-aware normalization method. FIG. 11A illustrates drifting features and responses over time. FIG. 11B illustrates drifting features vs. responses. FIG. 11C illustrates time-aware normalized features and responses over time. FIG. 11D illustrates time-aware normalized features vs. responses.

[0057] [Figure 12] FIG. 12 illustrates a non-limiting example of the adaptive online learning method of the present disclosure.

[0058] [Figure 13A] 13A-13C illustrate non-limiting examples of the predictive performance of the present disclosure. FIG. 13A illustrates features and responses over time. FIG. 13B illustrates features versus responses grouped by time range. At time 200, the relationship between features and responses changes. FIG. 13C compares predictions over time for a shift problem for an adaptive online learning model, an online linear regressor model, and a linear regressor model. [Figure 13B] 13A-13C illustrate non-limiting examples of the predictive performance of the present disclosure. FIG. 13A illustrates features and responses over time. FIG. 13B illustrates features versus responses grouped by time range. At time 200, the relationship between features and responses changes. FIG. 13C compares predictions over time for a shift problem for an adaptive online learning model, an online linear regressor model, and a linear regressor model. [Figure 13C] 13A-13C illustrate non-limiting examples of the predictive performance of the present disclosure. FIG. 13A illustrates features and responses over time. FIG. 13B illustrates features versus responses grouped by time range. At time 200, the relationship between features and responses changes. FIG. 13C compares predictions over time for a shift problem for an adaptive online learning model, an online linear regressor model, and a linear regressor model.

[0059] [Figure 14A]Figures 14A-14D illustrate non-limiting examples of the predictive performance of the present disclosure using synthetic data. Figure 14A illustrates the synthetic response versus time. Every 200 steps, the underlying model generating the response changes. Figure 14B illustrates the first and second features over time. Both synthetic features steadily drift away from their initial points. Figure 14C illustrates the predicted and measured responses over time. At times 600 and 800, a shift occurs. The adaptive method described herein closely follows the measurements. The rolling OLS model struggles to adapt, and the online linear model does not. Figure 14D illustrates the absolute residuals over time. The absolute residuals of the adaptive method described herein decrease quickly. The residuals of the rolling OLS model are initially high but reach a low level, while the residuals of the online linear model remain high. [Figure 14B] Figures 14A-14D illustrate non-limiting examples of the predictive performance of the present disclosure using synthetic data. Figure 14A illustrates the synthetic response versus time. Every 200 steps, the underlying model generating the response changes. Figure 14B illustrates the first and second features over time. Both synthetic features steadily drift away from their initial points. Figure 14C illustrates the predicted and measured responses over time. At times 600 and 800, a shift occurs. The adaptive method described herein closely follows the measurements. The rolling OLS model struggles to adapt, and the online linear model does not. Figure 14D illustrates the absolute residuals over time. The absolute residuals of the adaptive method described herein decrease quickly. The residuals of the rolling OLS model are initially high but reach a low level, while the residuals of the online linear model remain high. [Figure 14C]Figures 14A-14D illustrate non-limiting examples of the predictive performance of the present disclosure using synthetic data. Figure 14A illustrates the synthetic response versus time. Every 200 steps, the underlying model generating the response changes. Figure 14B illustrates the first and second features over time. Both synthetic features steadily drift away from their initial points. Figure 14C illustrates the predicted and measured responses over time. At times 600 and 800, a shift occurs. The adaptive method described herein closely follows the measurements. The rolling OLS model struggles to adapt, and the online linear model does not. Figure 14D illustrates the absolute residuals over time. The absolute residuals of the adaptive method described herein decrease quickly. The residuals of the rolling OLS model are initially high but reach a low level, while the residuals of the online linear model remain high. [Figure 14D] Figures 14A-14D illustrate non-limiting examples of the predictive performance of the present disclosure using synthetic data. Figure 14A illustrates the synthetic response versus time. Every 200 steps, the underlying model generating the response changes. Figure 14B illustrates the first and second features over time. Both synthetic features steadily drift away from their initial points. Figure 14C illustrates the predicted and measured responses over time. At times 600 and 800, a shift occurs. The adaptive method described herein closely follows the measurements. The rolling OLS model struggles to adapt, and the online linear model does not. Figure 14D illustrates the absolute residuals over time. The absolute residuals of the adaptive method described herein decrease quickly. The residuals of the rolling OLS model are initially high but reach a low level, while the residuals of the online linear model remain high.

[0060] [Figure 15] 15-16 illustrate non-limiting examples of graphical user interfaces (GUIs) that may be configured to implement the systems and methods of the present disclosure. [Figure 16]15-16 illustrate non-limiting examples of graphical user interfaces (GUIs) that may be configured to implement the systems and methods of the present disclosure.

[0061] [Figure 17] FIG. 17 illustrates a non-limiting example of a computing device configured to implement the systems and methods described herein.

[0062] [Figure 18] FIG. 18 illustrates a non-limiting example of a web or mobile application provision system configured to implement the systems and methods described herein.

[0063] [Figure 19] FIG. 19 illustrates a non-limiting example of a cloud-based web / mobile application provision system configured to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0064] Detailed Description While various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, modifications, or substitutions may occur without departing from the present disclosure. It will be understood that various alternatives to the embodiments of the disclosure described herein may be employed.

[0065] Typical virtual metrology systems and methods are deficient for at least several reasons. Some systems and methods may not be able to model data drift, data shifts, or hierarchical data structures observed in manufacturing processes, such as semiconductor manufacturing processes. Data drift may generally refer to gradual changes in process dynamics, for example, due to aging of equipment used in the manufacturing process. Data shifts may generally refer to abrupt changes in process dynamics, for example, due to external operations, such as maintenance or calibration. Advanced process control methods associated with virtual metrology methods may require accurate prediction of process variables or target properties in a manufacturing process. However, some methods may not be able to model non-stationarity observed in real or actual data, for example, current or historical sensor data. Non-stationarity may occur, for example, in semiconductor manufacturing processes, such as chemical vapor deposition (CVD), etching, diffusion, or other processes. For example, data drift may cause a steady or slow decrease in the accuracy of predictions associated with a virtual metrology model. Data shifts may cause abrupt failure of a virtual metrology model. Some virtual metrology methods that use moving window-based methods may be inaccurate, for example, due to selecting a small subset of data. Some virtual metrology methods that use just-in-time learning methods cannot adapt to changes in the underlying relationships between process variables or target properties.

[0066] Recognized herein is a need for better systems and methods for advanced process control and monitoring in manufacturing processes. The systems and methods disclosed herein are capable of generating adaptive, online time series predictions for process variables or target properties in manufacturing processes. Systems and methods for process monitoring and control

[0067] In one aspect, disclosed herein (FIG. 1) is a system for process monitoring and control. The system may include, for example, a data processing module configured to receive and process multiple data types and data sets from multiple different sources to generate training data, a training and optimization module configured to provide the training data to a machine learning pipeline to train and optimize a model, and an inference module configured to use the model to generate one or more predictive metrics in substantially real time, the one or more predictive metrics usable to characterize an output of a process performed by the process equipment.

[0068] As shown in FIG. 2 , the system may be associated with a method for advanced process control and monitoring in a manufacturing process. The methods described herein may generally include methods related to adaptive online time series prediction of a target property or process variable. The target property may include, for example, a property or characteristic of a target product (e.g., film thickness or refractive index of a wafer) determined through measurements from a metrology tool or instrument. The process variable may include, for example, data generated from a sensor associated with the manufacturing process or process equipment (e.g., temperature, power, or current in a chemical vapor deposition process or equipment). The adaptive online time series method may generally include a time-aware normalizer method or an adaptive online learner method. As shown in FIG. 3 , the systems and methods described herein can be used in advanced process control and monitoring of a manufacturing process. The manufacturing process may include a semiconductor manufacturing process associated with processing semiconductor wafers. For example, the process may include a chemical vapor deposition process for semiconductor wafers. Purpose

[0069] In some embodiments, the inference module is configured to receive and provide process data to the model to generate one or more predictive metrics, the process data being received from the process equipment in substantially real time as the process is performed. In some cases, the systems and methods disclosed herein may include virtual metrology systems and methods comprising one or more inference modules. In some embodiments, the model comprises a virtual metrology (VM) model. The virtual metrology model can be configured to predict a metric or property associated with the manufacturing process. The metric or property may be related to a process variable or a target property.

[0070] In some cases, a metric, property, or characteristic may be associated with a target property or process variable in a manufacturing process of a product. The target property may include a property associated with a product generated by equipment in the manufacturing process. For example, the target property may include the thickness or refractive index of a film or layer on a wafer or substrate in a chemical vapor deposition process. The process variable may include a variable associated with equipment configured to perform the manufacturing process. For example, the process variable may include pressure, gas volume, power, temperature, current, and the like associated with equipment configured to perform a chemical vapor deposition process. The prediction of the target property or process variable may be based, in part, on measurement data, sensor data, sensor specification data, equipment data, equipment specification data, process data, or process specification data. The inference modules disclosed herein may generate predictions of the target property or process variable without physical measurement of the target property or process variable.

[0071] In some embodiments, the inference module is configured to provide one or more predictive metrics for process control or for process monitoring, improvement, or troubleshooting. Manufacturing processes, such as semiconductor manufacturing processes, may use advanced process control systems and methods to control and monitor target properties or process variables (e.g., metrics). Advanced process control and monitoring methods may include methods for monitoring, controlling, improving, or troubleshooting a process using feedback loops. Advanced process control systems and methods may include virtual metrology systems and methods. The virtual metrology systems and methods described herein may determine or predict target properties or process variables (e.g., outputs) and provide the predicted target properties or process variables (e.g., inputs) in a feedback loop for advanced process control and monitoring.

[0072] In some embodiments, the system further includes a process control module configured to use one or more predictive metrics to detect drift, deviation, or excursion in a process or process equipment. Metrics (e.g., data) from a manufacturing process (e.g., a semiconductor process) may significantly differ from other data due to, for example, a continuously changing distribution of data. For example, the data may include time-series data with non-stationary characteristics. The non-stationary characteristics may comprise data drift, data deviation, hierarchical data structure, or excursion in the process or equipment. Data drift may cause a steady or slow decrease in the accuracy of the virtual metrology model. Data drift may occur due to, for example, aging in equipment or sensors. Data deviation may cause a sudden failure of the virtual metrology model. Data deviation may occur due to, for example, changes in equipment or sensor characteristics after maintenance or calibration. The data may have a hierarchical structure associated with processes, equipment, stations, sensors, and the like. In some embodiments, the process control module is configured to use one or more predictive metrics to correct or mitigate drift, deviation, or excursion in a process or process equipment. Alternatively, or in addition, the process control module can be configured to use one or more predictive metrics to correct or mitigate excursions in a process or equipment.

[0073] In some embodiments, the process control module is configured to use one or more predictive metrics to improve process productivity through integration with run-to-run control. In some cases, the systems and methods described herein may be integrated into a manufacturing processing line as a run-to-run system in real time. In some embodiments, the system is configured to be used or deployed in a manufacturing environment. The run-to-run system may generally include a method for modifying a recipe associated with a process in real time. A recipe may include a set of instructions for performing a process in a manufacturing process. A recipe may be associated with parameters. For example, a recipe and parameters for a chemical vapor deposition process may include deposition instructions with parameters that specify a power level, e.g., a specified current and voltage. Additionally, the run-to-run system may generally include a method for modifying control parameters associated with the process in real time, e.g., process variables of equipment for chemical vapor deposition, such as pressure, gas amounts, power, temperature, current, and the like. In some cases, modifying control parameters, recipes, or models may be performed using a graphical user interface (GUI), as shown in FIGS. 15-16 . For example, users can create their own unique models incorporating their own specialized knowledge or compare actual and predicted process outcomes. The systems and methods described herein can include methods for generating real-time predictions for data drift, data shifts, hierarchical data structures, or deviations in a process or equipment, and for inferring or characterizing the output of a manufacturing process. The inferred or characterized output can be used to modify recipe or control parameters in a real-time feedback loop to improve the predictive performance of a process variable or target property.

[0074] In some embodiments, the system further comprises process equipment, which may comprise semiconductor processing equipment. For example, as shown in FIGS. 4A-4B , the process equipment may include equipment associated with semiconductor wafer production, oxidation, photolithography, etching, deposition, ion implantation, metallization, electrical die sorting, packaging, and the like. In some embodiments, the deposited or processed structure comprises a film, layer, or substrate. The systems and methods described herein may be configured to monitor or control a target property or process variable. The target property may be associated with a wafer, film, layer, or substrate. Alternatively, or in addition, the target property may include a mechanical, electrical, or structural property of the product. For example, the electrical property of a wafer may include controlling and monitoring the conductivity of the wafer via a doping process. The mechanical property of a wafer may include controlling and monitoring the warp of the wafer via a metalorganic vapor deposition process. The structural property may include controlling and monitoring the film thickness or refractive index of the film via a chemical vapor deposition process. In some embodiments, the one or more predictive metrics comprise one or more dimensions or properties of the film, layer, or substrate. Data Processing

[0075] In some embodiments, the multiple data types and data sets comprise (1) historical process data, (2) current process data, (3) historical measurement data of one or more predictive metrics, (4) current measurement data of one or more predictive metrics, (5) operational data, and / or (6) equipment specification data. As described elsewhere herein, the systems and methods may be associated with virtual metrology systems and methods. The virtual metrology systems and methods can generate substantial amounts of data. The data may include current (e.g., current or real-time) data or historical data. The current or historical data may be associated with sensor data, sensor specification data, process data, process specification data, measurement data, operational data, equipment data, or equipment specification data. For example, the data can comprise fault detection and classification (FDC) data, sensor data (e.g., temperature, pressure, power, current, gas volume, and the like), measurement data (e.g., film thickness, film refractive index, critical dimension, and the like), or operational data.

[0076] In some cases, the systems and methods disclosed herein can support processing of manufacturing data in streaming mode or batch mode. Streaming mode may generally include data received in real time during the course of a manufacturing process, e.g., current or real-time data. Batch mode may generally include data received from a manufacturing process in non-real time, e.g., historical data.

[0077] In some embodiments, the data processing module is configured to validate the historical process data and historical measurement data against the operational data and equipment specification data. Validation of the historical process data against the operational or equipment data can be associated with a threshold value. In some cases, the historical process data may be validated when the threshold value is at least about 60%, 70%, 80%, 90%, or more of the data associated with the operational or equipment data. Validation of the historical measurement data against the operational or equipment data can be associated with a threshold value. In some cases, the historical measurement data may be validated when the threshold value is at least about 60%, 70%, 80%, 90%, or more of the data associated with the operational or equipment data. The threshold value can be associated with a predetermined standard deviation. In some cases, validation of the historical process data may be validated when the threshold value is within at least about three standard deviations (3σ), two standard deviations (2σ), or one standard deviation (1σ) of the operational or equipment data. In some cases, the validation of the historical measurement data may be verified when the threshold is within at least about three standard deviations (3σ), two standard deviations (2σ), or one standard deviation (1σ) of the operational or equipment data.

[0078] In some embodiments, the multiple sources comprise a database configured to store at least the historical process data or the historical measurement data. In some cases, the multiple sources can include a database configured to store the latest or real-time process data. In some cases, the multiple sources can include a database configured to store the latest or real-time measurement data. Some or all of the data can be stored in one or more databases. In some cases, the multiple sources can include a database configured to store a subset of the latest or real-time process data. In some cases, the multiple sources can include a database configured to store a subset of the latest or real-time measurement data. The subset of data can comprise at least about 25%, 50%, 75%, or more of the historical process data, historical measurement data, latest process data, or data associated with the latest measurement data.

[0079] In some embodiments, the multiple sources comprise a database or log configured to store at least operational data or equipment specification data. In some cases, the multiple sources can include a database configured to store current, real-time, or historical operational data. In some cases, the multiple sources can include a database configured to store current, real-time, or historical equipment specification data. Some or all of the data can be stored in one or more databases. In some cases, the multiple sources can include a database configured to store a subset of current, real-time, or historical operational data. In some cases, the multiple sources can include a database configured to store a subset of current, real-time, or historical equipment specification data. The subset of data can comprise at least about 25%, 50%, 75%, or more of the data associated with historical operational data, historical equipment specification data, current operational data, or current equipment specification data.

[0080] In some embodiments, the multiple sources comprise process equipment. As described elsewhere herein, a manufacturing process may be associated with process equipment that includes one or more set points that an engineer / designer may set. For example, process equipment in a semiconductor manufacturing process may include equipment associated with wafer production, oxidation, photolithography, etching, deposition, ion implantation, metallization, electrical die sort, packaging, and the like. A user can set various set points for any of this equipment, such as temperature, angle, height, depth, width, size, amount of material, and many others.

[0081] In some embodiments, the plurality of sources comprises a metrology instrument configured to collect current metrology data. As described elsewhere herein, the manufacturing process may be associated with metrology instruments. The metrology instrument may include a metrology tool or instrument configured to measure a target property. For example, the metrology tool or instrument may be used to measure a target property in a semiconductor manufacturing process, such as film thickness, refractive index, critical dimensions, and the like, for a wafer.

[0082] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by generating a component hierarchy of the process equipment. A manufacturing process can be associated with process equipment having subsystems, e.g., components. A manufacturing process can include at least one, two, three, four, five, or more types of process equipment. The process equipment can all be the same type of process equipment. The process equipment can be different types of process equipment. Components of the process equipment can include sensors configured to collect data associated with process variables.

[0083] In some embodiments, the component hierarchy comprises (i) process equipment and (ii) a nested structure of one or more components used within or in conjunction with the process equipment. For example, as shown in FIG. 5 , a semiconductor manufacturing process may include a chemical vapor deposition process associated with process equipment, e.g., Equipment 1, Equipment 2, etc. The components of the process equipment may include chambers, e.g., Chamber A, Chamber B, etc., for performing the chemical vapor deposition process. The chambers may include stations, e.g., Station S1, Station S2, etc., configured to measure sensor data such as temperature, pressure, power, current, gas amounts, and the like. In some embodiments, the one or more components comprise one or more sub-equipment, including chambers, stations, and / or sensors.

[0084] In some cases, a manufacturing process may include many different types of processes with different types of process variables or target properties. As described elsewhere herein, the systems and methods can be expanded by aggregating data from more than one process equipment sensor or more than one sensor. Aggregating data from processes, process equipment, or sensors can improve generating predictive metrics, such as predictions of target properties or process variables. For example, by aggregating and processing data from multiple equipment chambers for the same process step, a sparse data set can still be used to generate reliable, robust, and scalable virtual metrology models by training the machine learning algorithms described herein.

[0085] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by generating a step-operation hierarchy of a recipe for the process. For example, as shown in Table 1, a recipe for chemical vapor deposition in a semiconductor manufacturing process may include a recipe or method having operations and sub-operations. An operation may include loading, preparation, deposition, unloading, or cleaning. A sub-operation for deposition may include controlling voltage or current to generate a predetermined power. In some embodiments, a recipe includes multiple steps, and each step of the multiple steps includes multiple different sub-operations. An operation or sub-operation may be associated with a data type or data set related to a process variable or target property. Process variables may include, for example, pressure, gas amounts, power, temperature, and the like for a chemical vapor deposition process. Target properties may include, for example, film thickness or refractive index of a semiconductor wafer processed through a chemical vapor deposition process. [Table 1]

[0086] In some embodiments, the data processing module is configured to receive and process multiple data types or data sets by removing one or more data outliers. The multiple data types or data sets may include data in raw or unmodified form, such as trace data. For example, FIG. 6 depicts trace data associated with process variables related to an operation or sub-operation of a chemical vapor deposition process. The process variables may include, for example, temperature, power, or current of the chemical vapor deposition process. In some cases, the trace data may be processed to generate clean data. Generating clean data may include, for example, handling irregular data or removing data outliers (e.g., within or outside a predetermined threshold) based on process specifications, equipment specifications, or sensor specifications. The data may be current data (e.g., current or real-time data) or historical data. In some embodiments, the data processing module is configured to preprocess and remove data outliers from the process data before the process data is input into a model in the inference module.

[0087] In some embodiments, the training data is continuously updated with current process data and current measurement data. Features can be extracted from the trace data or clean data to generate training, test, or validation data. A training and optimization module may use the features to train and optimize virtual metrology models via a machine learning pipeline. Model training pipeline

[0088] In some embodiments, the machine learning pipeline comprises two or more components from a plurality of components consisting of (i) feature engineering, (ii) time-aware data normalization, and / or (iii) an adaptive learning algorithm. Feature engineering may generally include feature extraction and feature selection that determine key characteristics of time-series data having data drift, data shift, hierarchical data structure, or data deviation in a process or process equipment. Feature engineering may be associated with correlation coefficients or variable contraction. Time-aware data normalization may generally include decomposing time-series data into multiple components and individually adjusting the process components. Time-aware data normalization may be associated with differentiation or moving averages. Adaptive learning methods or algorithms may generally include determining changing relationships between input data (e.g., process variables such as features or sensor data) and output data (e.g., process variables or target properties). Adaptive learning methods or algorithms may be associated with rolling regression or online regression. A combination of one or more of feature engineering, time-aware data normalization, or adaptive learning algorithms can improve the predictive performance of target properties or process variables in advanced process control and monitoring. In some embodiments, a machine learning pipeline is configured to apply training data through two or more components, either sequentially or simultaneously.

[0089] As described elsewhere herein, advanced process control and monitoring may be associated with virtual metrology systems and methods. The virtual metrology systems and methods may be associated with machine learning systems and methods. The machine learning systems and methods can use data from the manufacturing process to train, test, and validate virtual metrology models. The virtual metrology models can generate adaptive online time series predictions for use in virtual metrology. The predictions may include predictions regarding process variables or target properties in manufacturing processes, such as semiconductor manufacturing processes.

[0090] The virtual metrology models disclosed herein can address technical challenges specific to data associated with semiconductor manufacturing processes, such as data drift, data shifts, hierarchical data structures, or data deviations in the process or process equipment. As described elsewhere herein, systems and methods may compare actual data (e.g., raw or trace data from current or historical sensor data) with specified or certified process or equipment data, e.g., equipment specifications or process specifications. Actual data that does not match the specified or certified data outside of predetermined thresholds may be further processed before use in building machine learning models, such as virtual metrology models. Virtual metrology models can enable more efficient workflows for controlling and monitoring manufacturing processes.

[0091] Generating a machine learning model (e.g., a virtual metrology model) through a machine learning pipeline may generally include receiving data, preprocessing the data, selecting or engineering features from the data, training a model using the data or features, testing the model using the data or features, or validating the model using the data or features. The validated model can be deployed or integrated into a manufacturing process. The data or features can include data or features generated from multiple iterations of training, testing, or validation through the machine learning pipeline. The data or features can include data or features generated from multiple instances of the manufacturing process. The multiple instances can occur during different time periods or during the same time period. The data or features can be automatically tracked or used in real time. The data or features can be stored for review and used at a later time. In some embodiments, time-aware data normalization comprises decomposing time series data into one or more components, including smoothed data, trended data, and / or detrended data.

[0092] 7 and 8 depict operations that may be associated with feature engineering in a machine learning pipeline. Feature engineering can generally include operations related to feature creation, transformation, feature extraction, or feature selection. A feature engineering operation can comprise at least one, two, three, four, five, or more operations. A feature engineering operation can comprise up to five, four, three, two, or one operations. In some cases, a feature engineering operation can include operations such as a feature engineer (e.g., performing column combinations), a nan frac threshold (e.g., removing columns with values ​​above a threshold that are not a number (NaN)), a simple imputer (e.g., performing mean imputation of missing values), a sliding window correlation selector, or a polynomial feature (e.g., multiplying columns together). In some cases, a feature engineering operation can include operations such as a feature engineer, a nan frac threshold, a simple imputer, a function transformer such as a detrender, a differentiator, an exponentially weighted average, a standard scaler, or a sliding window correlation selector.

[0093] In some embodiments, feature engineering comprises the extraction of multiple features from raw trace data or sensor data in the training data. Compared to methods using global relationships, sliding window methods can better determine local relationships for data with non-stationary characteristics. Non-stationary characteristics may include, for example, data drift or data shifts, as described elsewhere herein. In some embodiments, feature engineering comprises the use of an algorithm to select one or more features from the list of extracted features based, at least in part, on local relationships between the model's inputs and outputs. For example, as shown in FIG. 9, a method using global relationships may inaccurately determine the relationship between input and output data. A method using global relationships may predict the relationship between output (y) and input (x) as y = -0.6x + 132 across all time points, even though there are three distinct time periods. The sliding window method can predict the changing relationship between output (y) and input (x) over three distinct periods of trace data as y = -2x + 201, y = x - 103, and y = -4.1x + 932. Figure 10 conceptually depicts the sliding window method. The sliding window method can determine the running correlation between output and input data by (i) sliding a window across a time series of data and (ii) averaging the correlations as a selection score. In some embodiments, time-aware data normalization is based on the model and the model's data type. The correlation of the ith feature can be determined using the following equation: [ka]

[0094] where i is the ith variable of the input data (e.g., X data, features, sensor data), j is the jth variable of the output data (e.g., Y data, response, target, measurement), r is the correlation coefficient, T is the total number of input data, W is the window size, and J is the total number of variables of the output data.

[0095] In some embodiments, the adaptive learning algorithm is an adaptive online ensemble learning algorithm. A single machine learning model may not be able to accurately predict all process variables or target properties at a given confidence level. Thus, the methods disclosed herein can generate, maintain, or update two or more models for use in an ensemble model to predict process variables or target properties at a given confidence level. The models may predict process variables or target properties with at least about 60%, 70%, 80%, 90%, 95%, or better confidence levels. In some cases, the models may predict process variables or target properties with at least about 95% confidence levels. The systems and methods described herein can generate, maintain, or update multiple machine learning models for selection in different manufacturing processes to improve predictive performance. In some cases, the systems and methods described herein can validate the performance of machine learning models for multiple manufacturing processes or targets, for example, to generate the best-performing model. After automatically generating benchmark tests for process variables or target properties, the best-performing machine learning models can be deployed in an agile manner with little or no human action. Model Optimization

[0096] In some embodiments, the training and optimization module is configured to optimize the model, at least in part, using hyperparameter optimization. In some cases, the virtual metrology models disclosed herein can be trained using hyperparameter optimization using streaming or batch data. Hyperparameter optimization (e.g., tuning hyperparameters in a machine learning model) may be used for one, more than one, or all of the manufacturing processes. Hyperparameter optimization for all or substantially all of the manufacturing processes can improve overall predictive performance by searching for a common set of hyperparameters across the processes, which can mitigate overfitting compared to optimizing for one process or several processes. In some cases, hyperparameter optimization can improve predictive performance by at least about 1%, 5%, 10%, 20%, or more when optimizing for all or substantially all of the processes compared to optimizing for several processes or one process. Some processes may improve predictive performance by up to about 30%, 20%, 10%, or less for all processes.

[0097] In some embodiments, the training and optimization module is configured to (i) train a model using a given set of hyperparameters on output from the machine learning pipeline. In some cases, the systems and methods disclosed herein may process data from many different sensors, different devices, or different processes and select the most relevant features. The training and optimization may include using hyperparameter optimization on one or more features for better predictive performance of a target property or process variable. For example, with reference to FIG. 7 or 8, hyperparameters associated with a sliding window correlation selector may include multicollinearity and the like. Hyperparameters associated with polynomial features may include degree and the like. Hyperparameters associated with detrenders may include half-life, change point, and the like. Hyperparameters associated with scalers may include scaler type and the like.

[0098] In some embodiments, the training and optimization module is further configured to (ii) evaluate the performance of the model based on the validation data. The systems and methods described herein can track changes in sensor data, equipment data, or process data over time to continuously update or optimize the virtual metrology model. The test data, training data, or validation data may change over time, thus changing the local or global relationship between the input data and the output data over time. The performance of the model can be continuously improved over time by training, testing, or validating the model using other training data, other test data, or other validation data that differs from the previous training data, previous test data, or previous validation data.

[0099] In some embodiments, validation data is split from training data for hyperparameter optimization. Splitting data for training, testing, or validation (e.g., split validation) can be performed using a predetermined percentage. For example, the data may be split into 80% for training and 20% for testing. The data may be split into 80% for training, 10% for testing, and 10% for validation. Alternatively, or in addition, splitting the data can be performed using cross-validation, for example, exhaustive cross-validation or non-exhaustive cross-validation such as k-fold validation.

[0100] In some embodiments, the training and optimization module is further configured to (iii) use a hyperparameter optimization algorithm to select a set of hyperparameters for the next iteration based on past performance to increase or improve the performance of the model. The systems and methods described herein may use methods associated with algorithms for selecting hyperparameters for hyperparameter optimization. For example, a set of hyperparameters can be determined to improve predictive performance over another or previous set of hyperparameters. The set of hyperparameters can be used for a subsequent integration of training, testing, or validation of the model to increase or improve the predictive performance of the model.

[0101] In some embodiments, the training and optimization module is further configured to iteratively repeat (i)-(iii) until the model's performance meets a termination criterion. In some cases, the termination criterion may be a predetermined metric based on the absolute value of the metric. Alternatively, or in addition, the termination criterion may be a predetermined metric based on a relative change in the metric. The termination criterion may include no change in the metric over a given number of epochs, an absolute change in the metric, a decrease in predictive performance observed over a given number of epochs, or an average change in the metric over a given number of epochs.

[0102] In another aspect, disclosed herein is a method for process monitoring and control. The method may include, for example, (a) receiving and processing multiple data types and data sets from multiple different sources to generate training data, (b) providing the training data to a machine learning pipeline to train and optimize a model, and (c) generating one or more predictive metrics in substantially real time, the one or more predictive metrics usable to characterize an output of a process performed by process equipment. Machine learning methods for prediction in process monitoring and control

[0103] Many machine learning (ML) methods implemented as algorithms are suitable approaches for performing the methods described herein, including, but not limited to, supervised learning approaches, unsupervised learning approaches, semi-supervised approaches, or any combination thereof.

[0104] Machine learning algorithms may include, but are not limited to, neural networks (e.g., artificial neural networks (ANNs), multilayer perceptrons (MLPs), long short-term memory (LSTM)), support vector machines, k-nearest neighbors, Gaussian mixture models, Gaussian processes, naive Bayes, decision trees, random forests, or gradient-boosted trees. Linear machine learning algorithms may include, but are not limited to, linear regression with or without a regularizer, logistic regression, naive Bayes classifiers, perceptrons, or support vector machines (SVMs). Other machine learning algorithms for use with the methods of the present disclosure may include, but are not limited to, quadratic classifiers, k-nearest neighbors, boosting, decision trees, random forests, neural networks, pattern recognition, Bayesian networks, or hidden Markov models. Other machine learning algorithms, including modifications or combinations of any of these commonly used for machine learning, may also be suitable for use with the methods described herein. Any use of machine learning algorithms in a workflow may also be suitable for use with the methods described herein. The workflow can include, for example, cross-validation, nested cross-validation, feature selection, row compression, data transformation, binning, normalization, standardization, and algorithm selection.

[0105] Machine learning algorithms can generally be trained by the following methodology to build machine learning models. In some cases, the generated models may determine or predict target properties of a product, such as film thickness or refractive index of a wafer. The input data may include, for example, process variables such as X-data, features, fault detection and classification (FDC) data, sensor data, and the like, as described elsewhere herein. The output data may include, for example, target properties such as Y-data, responses, measurements, and the like, as described elsewhere herein. In some cases, the generated models may determine or predict process variables. The input data may include, for example, process variables as described elsewhere herein. The output data may include, for example, process variables as described elsewhere herein.

[0106] 1. Collect datasets for "training" and "testing" the machine learning algorithm. The datasets can include many features, for example, features associated with sensor data, equipment, processes, and the like. The training dataset is used to "train" the machine learning algorithm. The test dataset is used to "test" the machine learning algorithm.

[0107] 2. Determine the "features" for the machine learning algorithm to use for training and testing. The accuracy of the machine learning algorithm may depend on how the features are represented. For example, feature values ​​may be transformed using one-hot encoding, binning, standardization, or normalization. Also, not all features in the dataset may be used to train and test the machine learning algorithm. Feature selection may depend, for example, on available computing resources and time or the importance of features discovered during iterative testing and training. For example, features associated with sensor data or equipment specifications may be discovered to predict process variables or target properties.

[0108] 3. Select an appropriate machine learning algorithm. For example, a machine learning algorithm described elsewhere herein may be selected. The selected machine learning algorithm may depend, for example, on available computing resources and whether the time or prediction is continuous or categorical in nature. The machine learning algorithm is used to build the machine learning model.

[0109] 4. Construct a machine learning model. A machine learning algorithm is run on the collected training dataset. The parameters of the machine learning algorithm may be adjusted by optimizing performance on the training dataset or via a cross-validation dataset. After parameter adjustment and training, the performance of the machine learning algorithm may be validated on a dataset of naive samples that is separate from the training dataset and the test dataset. The constructed machine learning model may involve feature coefficients, importance measures, or weights assigned to individual features.

[0110] Once a machine learning model has been determined ("trained") as described above, it can be used to generate predictions for process monitoring and control in manufacturing processes, such as semiconductor manufacturing processes. [Example]

[0111] While various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, modifications, or substitutions may occur without departing from the present disclosure. It will be understood that various alternatives to the embodiments described herein may be employed. Example 1 - Adaptive online time series prediction for virtual metrology in semiconductor manufacturing processes Introduction

[0112] The systems and methods described herein (e.g., virtual metrology (VM) models) have been used to statistically model wafer characteristics using sensor data and historical measurements to predict, control, and monitor semiconductor manufacturing processes, such as chemical vapor deposition (CVD) processes. The virtual metrology models have increased productivity, improved quality, and lowered maintenance costs by reducing physical or human inspection associated with the semiconductor manufacturing process.

[0113] Compared to the systems and methods described herein, other methods (e.g., algorithmic methods or models) cannot work with data having data drift or data shift observed in semiconductor manufacturing processes. Data drift generally refers to gradual changes in process dynamics due to, for example, equipment aging. Data shift generally refers to abrupt changes in process dynamics due to, for example, external operations such as maintenance, calibration, or layer changes.

[0114] To reliably use virtual metrology models in advanced process control and monitoring, model predictions should be highly accurate at all points in the process. However, some models cannot handle non-stationarity observed in sensor or measurement data generated by, for example, chemical vapor deposition, etching, diffusion, or other processes. Data drift can cause a steady or slow decrease in model accuracy. Data shifts can cause rapid model failure. For example, Gaussian process solutions or neural network architectures can generate imperfect solutions or predictions by focusing primarily on data drift. For example, window-based models can generate unreliable predictions or fail when data shifts occur due to a fixed window size. For example, just-in-time learning models cannot adapt to changes in the relationship between inputs and outputs.

[0115] The systems and methods described herein use a novel time series forecasting framework that incorporates time-aware data normalization and adaptive learning methods to generate accurate virtual metrology models (see FIGS. 1 and 2). The time-aware normalization method can remove data drift from the data. The method may use various smoothing algorithms, such as an exponentially weighted moving average, to perform this transformation. The adaptive learning method can capture the time-varying relationship between responses (e.g., outputs) and inputs. The systems and methods described herein also generate an ensemble of predictions from several models, resulting in more accurate predictions in data-shifting environments and reducing process variability on real production lines in manufacturing processes. Adaptive Online Time Series Forecasting

[0116] The systems and methods (e.g., adaptive online time series methods, algorithms, or methods) described herein may integrate two complementary strategies or operations into a pipeline process to overcome data drift or data shifts. The pipeline may include a time-aware normalization method and an adaptive online learning method. The method may normalize inputs or outputs, for example, using a moving average method, to remove data drift from the data and generate transformed data so that the distribution of the inputs or outputs remains consistent over time. The method may provide the transformed data as input to an adaptive online learning algorithm to capture changing relationships between variables by considering fluctuations in process dynamics. Both operations can complement each other to solve different problems, such as chamber drift or concept drift. Chamber drift generally occurs when changes in chamber conditions modify the output of a sensor. Concept drift generally may alter the relationship between a sensor and a measurement. The time-aware normalization method described herein can solve chamber drift to handle drifting distributions. The adaptive online learning method herein can address concept drift to handle changing relationships. Time-aware normalization method

[0117] Time-aware normalization methods can reduce the effects of data drift on prediction accuracy. In a hypothetical metrology dataset associated with a semiconductor manufacturing process, features may drift slowly and steadily over time. Data drift can create distortions between inputs and outputs, reducing prediction accuracy over time. Time-aware normalization methods can remove such variations and restore the relationship between features and targets. For example, FIGS. 11A-11D show how drifting inputs can mask a linear relationship. In FIGS. 11A-11B, the feature slowly drifts away from its initial range, while the response remains stationary. As a result, the correlation between the input and output is null. Applying time-aware normalization methods to the features can restore the linear relationship, as illustrated in FIGS. 11C-11D.

[0118] The mean, scale, sampling rate, slope, or any statistical value may change over time, affecting the predictive performance of the virtual metrology model. Therefore, the time-aware normalization method can use different approaches. For example, differencing, smoothing, detrending, online estimation of various parameters, or time series decomposition methods may be required depending on the drifting data. The systems and methods described herein may automatically transform both input and output, for example, using an exponentially weighted moving average. By appropriately weighting each data point based on the observation time, the methods described herein can ensure that the input and output are consistent over time. After transforming the data through the time-aware normalization method, the transformed data can be provided as input data for an adaptive online learning method. Adaptive Online Learning Methods

[0119] The adaptive online learning methods described herein can handle changes in input-output relationships. The methods can model these relationships as they change over time. The methods may aggregate multiple online experts (e.g., regressors) by evaluating them based on their current and past performance. The methods may emphasize the best-performing experts for any given time window by minimizing adaptive regrets. Adaptive regrets generally measure how well an algorithm performs compared to the optimum over all time intervals, in hindsight. At each time point, the best regressors contribute the most to the prediction. The pseudocode illustrated by the example in FIG. 12 can provide intuition about how individual experts contribute to the final prediction. At each time point t, an expert's weight i varies based on its residual. Higher residuals lead to smaller weights for other experts. Experts may include, for example, linear models, Gaussian processes, decision tree regressors, or any online regressors. Choosing the right regressors can be important to the performance of the adaptive online learning model, for example, in terms of speed or accuracy. To control how quickly the ensemble adapts to changes, the decay parameter η can be tuned based on the dynamics of the process.

[0120] 13A-13C provide an example of an adaptive online learning method that models a simple univariate linear relationship. In FIG. 13A, y=a t x+b t where for t≦200 (a t , b t )=(1,5), and for t≧200 (a t , b t) = (-1, 15). Assuming that no points after t = 200 are observed at test time, an offline algorithm trained on data from t ≤ 200 may provide inaccurate predictions. On the other hand, other online methods without adaptation may struggle to adapt after sudden shifts. Furthermore, while the time-aware normalization method presented above can reduce the effect of a shifting intercept, it may not be able to handle inverted coefficients, as illustrated in Figure 13B. Because the model changes completely, an adaptive online learner model may be required. Figure 13C compares the predictive performance of a linear regressor method, an online linear regressor method, and the adaptive online learning method of the present disclosure using an online linear regressor as the base expert. The method disclosed herein provides more accurate predictions than the linear regressor and the online linear regressor. The linear regressor does not adapt to model changes. The online linear regressor provides incorrect predictions and does not reach the correct regimen in time. result

[0121] The systems and methods described herein have been used to generate predictions for chemical vapor deposition processes using both synthetic data and actual data from manufacturing processes.

[0122] Synthetic Data. The synthetic data included a dataset that replicated the behavior of a carbon deposition process. The dataset shown in FIG. 14A included a concatenated sample of five regression problems of 200 samples and 10 features. Each regression problem had different coefficients and intercepts. In this concatenated dataset, model changes were generated every 200 time steps to determine how quickly each method adapted to data shifts. In addition, random data drift was generated for each feature. FIG. 14B shows two features evolving over time. In this synthetic dataset, the features drift and the response shifts. This synthetic dataset can emulate data shifts caused by, for example, maintenance, recipe changes, or calibration operations, as well as data drift resulting from, for example, equipment aging.

[0123] The systems and methods disclosed herein were compared to other virtual measurement methods that use rolling ordinary least squares (OLS) methods and online linear methods (also known as recursive least squares). Rolling OLS methods can be parameterized by a window size and stride. At each window, a linear model can be fitted. This model can make predictions until the window strides and a new model is estimated and used for prediction. In online linear methods, coefficients and intercepts can be updated whenever a response is observed. By default, these methods may not have specific adaptation mechanisms.

[0124] Each model was trained on the training data set and tuned on the evaluation data set. Performance was measured on a separate test data set. The training and evaluation data sets covered the first 600 points. The test data set included the last 400 points. By construction, the test data set contains data drift and data shifts that are not observed during training. Table 2 shows the coefficients of determination (R 2 ) and root mean square error (RMSE). The adaptive online time series method described herein has significantly improved R and R, respectively, compared to the rolling OLS method. 2 and improved the regression metrics by as much as 24% and 48% in terms of RMSE. [Table 2]

[0125] Figures 14C-14D compare the performance of the rolling OLS method, the online linear method, and the method disclosed herein. The adaptive online time series method described herein adapts to the shift much faster than the rolling OLS method. As shown in Figure 14C, at timestamps 600 and 800, the method described herein requires approximately 25 points to adapt to the data shift, while the rolling OLS method requires approximately 50 points. The online linear method cannot adapt to the rapid data shift. Figure 14D compares the absolute residuals of the three methods. The method disclosed herein limits the magnitude of the error immediately after the data shift. While the mean absolute residuals can reach as high as 2 for the rolling OLS method and the online linear method, they remain near or below 1 for the method described herein. After approximately 50-60 points after the data shift, the rolling OLS method and the method described herein recover the underlying model and perform similarly. Online linear methods cannot adapt quickly enough and never reach adequate performance.

[0126] The adaptive online time series methods described herein outperform rolling OLS methods due to their adaptation speed. Windowed methods may be limited by their window size. Performance may be affected if the window overlaps two inconsistent ranges of data. Aggregation of regressors fitted across different time ranges can reduce the impact of data shifts. The adaptive models described herein consistently train and update experts so that the relevant, untrained experts are ready as soon as shifts occur.

[0127] Actual Data. The systems and methods described herein (e.g., adaptive online time series methods) were also deployed and integrated into the advanced process control and monitoring system of a large semiconductor manufacturer. The model covered four thin film depositions on a memory chip process using 72 deposition chambers. The memory chip process included dynamic random access memory (DRAM) and "not and" (NAND) processes. By running the system in routine advanced process control operations over several months, the systems and methods described herein achieved at least about 45.2% reduction in film thickness and refractive index variance (an average of about 21.5% reduction). Computing Systems

[0128] 17, a block diagram is shown depicting an example machine, including a computer system 1700 (e.g., a processing or computing system), in which a set of instructions may be executed to cause a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components of FIG. 17 are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic components, or combinations of two or more such components, to implement a particular embodiment.

[0129] Computer system 1700 may include one or more processors 1701, memory 1703, and storage device 1708, which communicate with each other and other components via a bus 1740. The bus 1740 may also couple a display 1732, one or more input devices 1733 (which may include, for example, a keypad, keyboard, mouse, stylus, etc.), one or more output devices 1734, one or more storage devices 1735, and various tangible storage media 1736. All of these elements may interface with the bus 1740 directly or through one or more interfaces or adapters. For example, the various tangible storage media 1736 may interface with the bus 1740 via a storage media interface 1726. The computer system 1700 may have any suitable physical form, including, but not limited to, one or more integrated circuits (ICs), a printed circuit board (PCB), a mobile handheld device (such as a mobile phone or PDA), a laptop or notebook computer, a distributed computer system, a computing grid, or a server.

[0130] Computer system 1700 includes one or more processors 1701 (e.g., a central processing unit (CPU) or a general-purpose graphics processing unit (GPGPU)) that perform functions. Processor 1701 optionally contains a cache memory unit 1702 for temporary local storage of instructions, data, or computer addresses. Processor 1701 is configured to assist in the execution of computer-readable instructions. Computer system 1700 may provide functionality for the components depicted in FIG. 17 as a result of processor 1701 executing non-transitory processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 1703, storage device 1708, storage device 1735, and / or storage medium 1736. The computer-readable medium may store, and processor 1701 may execute, software that implements particular embodiments. Memory 1703 may read software from one or more other computer-readable media (such as mass storage devices 1735, 1736), or from one or more other sources through a suitable interface, such as network interface 1720. The software may cause processor 1701 to perform one or more processes, or one or more steps of one or more processes, described or illustrated herein. Performing such a process or step may include defining data structures stored in memory 1703 and modifying the data structures as directed by the software.

[0131] Memory 1703 may include various components (e.g., machine-readable media), including, but not limited to, random access memory components (e.g., RAM 1704) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phase change random access memory (PRAM), etc.), read-only memory components (e.g., ROM 1705), and any combination thereof. ROM 1705 may act to communicate data and instructions unidirectionally to processor 1701, and RAM 1704 may act to communicate data and instructions bidirectionally with processor 1701. ROM 1705 and RAM 1704 may include any suitable tangible computer-readable media, as described below. In one embodiment, a basic input / output system 1706 (BIOS), containing the basic routines that help to transfer information between elements within computer system 1700, such as during start-up, may be stored in memory 1703.

[0132] Persistent storage 1708 is optionally connected bidirectionally to processor 1701 through storage control unit 1707. Persistent storage 1708 provides additional data storage capacity and may include any suitable tangible computer-readable media described herein. Storage 1708 may be used to store operating system 1709, executables 1710, data 1711, applications 1712 (application programs), and the like. Storage 1708 may also include an optical disk drive, a solid-state memory device (e.g., a flash-based system), or any combination of the above. The information in storage 1708 may, where appropriate, be incorporated into memory 1703 as virtual memory.

[0133] In one example, storage device(s) 1735 may be removably interfaced to computer system 1700 via storage device interface 1725 (e.g., via an external port connector (not shown)). In particular, storage device(s) 1735 and associated machine-readable media may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 1700. In one example, software may reside, completely or partially, within the machine-readable media on storage device 1735. In another example, software may reside, completely or partially, within processor 1701.

[0134] The bus 1740 connects a wide variety of subsystems. As used herein, reference to a bus may encompass, where appropriate, one or more digital signal lines serving a common function. The bus 1740 may be any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures. By way of example, and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a MicroChannel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, a HyperTransport (HTX) bus, a Serial Advanced Technology Attachment (SATA) bus, and any combination thereof.

[0135] Computer system 1700 may also include input device(s) 1733. In one example, a user of computer system 1700 may type commands and / or other information into computer system 1700 via input device(s) 1733. Examples of input device(s) 1733 include, but are not limited to, an alphanumeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touchscreen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combination thereof. In some embodiments, the input device is a Kinect (登録商標) , Leap Motion (登録商標) , or the like. Input devices 1733 may be interfaced to bus 1740 via any of a variety of input interfaces 1723 (e.g., input interfaces 1723), including, but not limited to, serial, parallel, game port, USB, FIREWIRE®, THUNDERBOLT®, or any combination of the above.

[0136] In particular embodiments, when computer system 1700 is connected to network 1730, computer system 1700 may communicate with other devices, particularly mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, that are connected to network 1730. Communications to and from computer system 1700 may be transmitted through network interface 1720. For example, network interface 1720 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1730, and computer system 1700 may store the incoming communications in memory 1703 for processing. Computer system 1700 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets that are communicated from network interface 1720 to network 1730. Processor 1701 may access these communication packets stored in memory 1703 for processing.

[0137] Examples of network interface 1720 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of network 1730 or network segment 1730 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus, or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combination thereof. A network such as network 1730 may employ wired and / or wireless communication modes. In general, any network topology may be used.

[0138] Information and data can be displayed through display 1732. Examples of display 1732 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive matrix OLED (PMOLED) or active matrix OLED (AMOLED) display, a plasma display, and any combination thereof. Display 1732 can interface with other devices, such as processor 1701, memory 1703, and persistent storage 1708, and input device(s) 1733, via bus 1740. Display 1732 is coupled to bus 1740 via video interface 1722, and transport of data between display 1732 and bus 1740 can be controlled via graphics control 1721. In some embodiments, the display is a video projector. In some embodiments, the display is a head-mounted display (HMD), such as a VR headset. In further embodiments, a suitable VR headset is, by way of non-limiting example, the HTC Vive. (登録商標) , Oculus Rift (登録商標) , Samsung Gear VR (登録商標) , Microsoft HoloLens (登録商標) , Razer OSVR (登録商標) , FOVE VR (登録商標) , Zeiss VR One (登録商標) , Avegant Glyph (登録商標) , Freefly VR (登録商標) headsets, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0139] In addition to the display 1732, the computer system 1700 may include one or more other peripheral output devices 1734, including, but not limited to, audio speakers, printers, storage devices, and any combination thereof. Such peripheral output devices may be connected to the bus 1740 via an output interface 1724. Examples of the output interface 1724 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE® port, a THUNDERBOLT® port, and any combination thereof.

[0140] Additionally, or alternatively, computer system 1700 may provide functionality as a result of logic hardwired or otherwise embodied in circuitry that may operate in place of or in conjunction with software to perform one or more processes, or one or more steps of one or more processes, described or illustrated herein. References to software in this disclosure may encompass logic, and references to logic may encompass software. Also, references to computer-readable media may encompass, where appropriate, circuitry (such as an IC) that stores software for execution, circuitry that embodies logic for execution, or both. This disclosure encompasses any suitable combination of hardware, software, or both.

[0141] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0142] The various illustrative logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete-gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0143] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processors, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0144] According to the description herein, suitable computing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, subnotebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the systems described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations.

[0145] In some embodiments, the computing device includes an operating system configured to execute executable instructions. An operating system is software, including, for example, programs and data, that manages the device's hardware and provides services related to the execution of applications. A suitable server operating system is, by way of non-limiting example, FreeBSD. (登録商標) , OpenBSD (登録商標) , NetBSD (登録商標) , Linux (登録商標) , Apple (登録商標) Mac OS X Server (登録商標) , Oracle Solaris (登録商標) , Windows Server (登録商標) , and Novell NetWare (登録商標) Suitable personal computer operating systems include, by way of non-limiting example, Microsoft Windows (登録商標) , Apple Mac (登録商標) OS X, UNIX, and GNU / Linux (登録商標)In some embodiments, the operating system is provided by cloud computing. Suitable mobile smartphone operating systems include, by way of non-limiting example, Nokia Symbian. (登録商標) OS, Apple (登録商標) iOS, Research In Motion BlackBerry (登録商標) OS, Google (登録商標) Android (登録商標) , Microsoft (登録商標) Windows Phone (登録商標) OS, Microsoft (登録商標) Windows Mobile (registered trademark) OS, Linux (登録商標) , and Palm (登録商標) Suitable media streaming device operating systems include, by way of non-limiting example, Apple TV. (登録商標) , Roku (登録商標) , Boxee (登録商標) , Google TV (登録商標) , Google Chromecast (登録商標) , Amazon Fire (登録商標) , and Samsung (登録商標) HomeSync (登録商標) Suitable video game console operating systems include, by way of non-limiting example, Sony (登録商標) PS3 (登録商標) , Sony (登録商標) PS4 (登録商標) , Microsoft (登録商標) Xbox 360 (登録商標) , Microsoft Xbox One (登録商標) , Nintendo Wii (登録商標) , Nintendo Wii U (登録商標) , and Ouya (登録商標) Suitable virtual reality headset systems include, by way of non-limiting example, Meta Oculus (登録商標) Includes: Non-transitory computer-readable storage medium

[0146] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer-readable storage media encoded with a program including instructions executable by an operating system of a networked computing device. In further embodiments, the computer-readable storage medium is a tangible component of the computing device. In still further embodiments, the computer-readable storage medium is optionally removable from the computing device. In some embodiments, the computer-readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid-state memory, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the programs and instructions are encoded on the medium permanently, substantially permanently, semi-permanently, or non-transitoryly. computer program

[0147] In some embodiments, the platforms, systems, media, and methods disclosed herein include at least one computer program or the use thereof. A computer program includes a sequence of instructions that are executable by one or more processors of a computing device's CPU and are written to perform specified tasks. Computer-readable instructions may be implemented as program modules, such as functions, objects, application programming interfaces (APIs), computing data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, computer programs may be written in a variety of languages ​​and versions.

[0148] The functionality of the computer-readable instructions may be combined or distributed as desired in various environments. In some embodiments, a computer program includes one sequence of instructions. In some embodiments, a computer program includes multiple sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from multiple locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or a combination thereof. Web Applications

[0149] In some embodiments, the computer program comprises a web application. In light of the disclosure provided herein, the web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some embodiments, the web application utilizes one or more database systems, such as Microsoft (登録商標) .NET or Ruby on Rails (登録商標) In some embodiments, the web application utilizes one or more database systems, including, by way of non-limiting example, relational, non-relational, object-oriented, associative, and XML database systems. In further embodiments, suitable relational database systems are, by way of non-limiting example, Microsoft (登録商標) Structured Query Language (SQL) server, mySQL TM , and Oracle (登録商標)Web applications, in various embodiments, are written in one or more versions of one or more languages. Web applications may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, web applications are written in part in a markup language such as HyperText Markup Language (HTML), Extensible HyperText Markup Language (XHTML), or Extensible Markup Language (XML). In some embodiments, web applications are written in part in a presentation definition language such as Cascading Style Sheets (CSS). In some embodiments, web applications are written in part in Asynchronous Javascript and XML. (登録商標) (AJAX), Flash Actionscript, Javascript, or Silverlight (登録商標) In some embodiments, the web application is written in a client-side scripting language such as Active Server Pages. (登録商標) (ASP), ColdFusion (登録商標) , Perl (登録商標) , Java (registered trademark), JavaServer Pages (登録商標) (JSP), Hypertext Preprocessor (登録商標) (PHP), Python (登録商標) , Ruby (登録商標) , Tcl (登録商標) , Smalltalk (登録商標) , WebDNA (登録商標) , or Groovy (登録商標) In some embodiments, the web application is written in part in a server-side coding language such as IBM Lotus Domino. In some embodiments, the web application is written in part in a database query language such as Structured Query Language (SQL). In some embodiments, the web application is written in part in a database query language such as IBM Lotus Domino. (登録商標)In some embodiments, the web application includes a media player component. In various further embodiments, the media player component may be a media player component, such as, by way of non-limiting example, Adobe (登録商標) Flash (登録商標) , HTML5, Apple (登録商標) QuickTime (登録商標) , Microsoft Silverlight (登録商標) , Java®, and Unity (登録商標) The present invention utilizes one or more of a number of suitable multimedia technologies, including:

[0150] 18, in a particular embodiment, the application provisioning system comprises one or more databases 1800 accessed by a database management system (DBMS) 1810. Suitable DBMSs include Firebird (登録商標) , MySQL (登録商標) , NoSQL (登録商標) , PostgreSQL (登録商標) , SQLite (登録商標) , Oracle Database (登録商標) , Microsoft SQL Server (登録商標) , IBM DB2 (登録商標) , IBM Informix (登録商標) , SAP Sybase (登録商標) , SAP Sybase (登録商標) , Teradata (登録商標) , PostGIS (登録商標) , Apache (登録商標) Hive, Apache (登録商標) Impala, time series databases, graph databases, key-value stores, and the like. In this embodiment, the application provisioning system further includes one or more application servers 1820 (Java Server, .NET (登録商標) Server, PHP (登録商標) Server, and the like) and one or more web servers 1830 (Apache (登録商標) , IIS(登録商標) , G.W.S. (登録商標) , and the like). The web server optionally exposes one or more web services via an app application programming interface (API) 1840. Over a network such as the Internet, the system provides a browser-based and / or mobile-native user interface. In some cases, the DBMS may be a relational DBMS.

[0151] Referring to FIG. 19, in a particular embodiment, the application provisioning system alternatively has a distributed cloud-based architecture 1900 with resiliently load-balanced, auto-scaling web server resources 1910 and application server resources 1920, and a synchronously replicated database 1930. Mobile Applications

[0152] In some embodiments, the computer program comprises a mobile application that is provided to the mobile computing device. In some embodiments, the mobile application is provided to the mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to the mobile computing device over a computer network as described herein.

[0153] In view of the disclosure provided herein, mobile applications are created by techniques using hardware, languages, and development environments. Mobile applications are written in several languages. Suitable programming languages ​​include, by way of non-limiting example, C, C++, C#, Objective-C, Java, Javascript, and Pascal. (登録商標) , Object Pascal (登録商標) , Python TM , Ruby (登録商標) , VB.NET (登録商標) , WML(登録商標) , and XHTML / HTML with or without CSS, or a combination thereof.

[0154] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non-limiting example, Airplay SDK. (登録商標) , alcheMo (登録商標) , Appcelerator (登録商標) , Celsius (登録商標) , Bedrock (登録商標) , Flash Lite (登録商標) , .NET Compact Framework (登録商標) , Rhomobile (登録商標) , and WorkLight Mobile Platform (登録商標) As a non-limiting example, Lazarus (登録商標) , MobiFlex (登録商標) , MoSync (登録商標) , and Phonegap (登録商標) Other development environments are also available at no cost, including, by way of non-limiting example, the iPhone 6. (登録商標) and iPad (登録商標) (iOS) SDK, Android (登録商標) SDK, BlackBerry (登録商標) SDK, BREW SDK, Palm (登録商標) OS SDK, Symbian (登録商標) SDK, webOS (登録商標) SDK, and Windows (登録商標) We distribute software developer kits, including the Mobile SDK.

[0155] As a non-limiting example, Apple (登録商標) App Store, Google (登録商標) Play, Chrome (登録商標) WebStore, BlackBerry (登録商標) App World, the App Store for Palm devices (登録商標), App Catalog for webOS (登録商標) , Windows for Mobile (登録商標) Marketplace, Nokia (登録商標) Ovi Store for Samsung devices (登録商標) Apps, and Nintendo (登録商標) Several commercial sources are available for the distribution of mobile applications, including the DSi Shop. Standalone Applications

[0156] In some embodiments, the computer program includes a standalone application, which is a program that runs as an independent computer process rather than being an add-on to an existing process, e.g., not a plug-in. Standalone applications are often compiled. A compiler is a computer program that converts source code written in a programming language into binary object code, such as assembly language or machine code. Suitable compiled programming languages ​​include, by way of non-limiting example, C, C++, Objective-C (登録商標) , COBOL (登録商標) , Delphi (登録商標) , Eiffel (登録商標) , Java (registered trademark), Lisp (登録商標) , Python (登録商標) , Visual Basic (登録商標) , and VB .NET (登録商標) , or a combination thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, the computer program comprises one or more executable compiled applications. In addition, Python (登録商標) and JavaScript® related microservices may be used. Web browser plugin

[0157] In some embodiments, the computer program includes a web browser plug-in (e.g., a web extension, etc.). In computing, a plug-in is one or more software components that add specific functionality to a larger software application. Software application manufacturers support plug-ins to allow third-party developers to create the ability to extend the application, support easy addition of new features, and reduce the size of the application. When supported, plug-ins allow customization of the functionality of the software application. For example, plug-ins are commonly used in web browsers to play videos, generate interactivity, scan for viruses, and display specific file types. Some web browser plug-ins are compatible with Adobe Flash Player (登録商標) , Microsoft Silverlight (登録商標) , and Apple QuickTime (登録商標) In some embodiments, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some embodiments, the toolbar comprises one or more explorer bars, tool bands, or desk bands.

[0158] In view of the disclosure provided herein, by way of non-limiting example, C++, Delphi (登録商標) , Java (registered trademark), PHP (登録商標) , Python (登録商標) , and VB .NET (登録商標) Several plug-in frameworks are available that allow development of plug-ins in a variety of programming languages, including, for example, .NET, .NET Framework 2.0, .NET Framework 3.0, .NET Framework 4.0, .NET Framework 5.0, .NET Framework 6.0, .NET Framework 7.0, .NET Framework 8.0, .NET Framework 9.0, .NET Framework 10.0, .NET Framework 11.0, .NET Framework 12.0, .NET Framework 13.0, .NET Framework 14.0, .NET Framework 15.0, .NET Framework 16.0, .NET Framework 17.0, .NET Framework 18.0, .NET Framework 19.0, .NET Framework 20.0, .NET Framework 21.0, .NET Framework 22.0, .NET Framework 23.0, .NET Framework 24.0, .NET Framework 25.0, .NET Framework 26.0, .NET Framework 27.0, .NET Framework 28.0, .NET Framework 29.0, .NET Framework 30.0, .NET Framework 31.0, .NET Framework 32.0, .NET Framework 33.0, .NET Framework 34.0, .NET Framework 35.0, .NET Framework 36.0, .NET Framework 37.0, .NET Framework 38.0, .NET Framework 40.0, .NET Framework 41.0, .NET Framework 42.0, .NET Framework 43.0, .NET Framework 44.0, .NET Framework 45.0, .NET Framework 46.0, .NET Framework 47.0, .NET Framework 48.0, .NET Framework 50.0, .NET Framework 51.0, .NET Framework 52.

[0159] A web browser (also called an internet browser) is a software application designed for use with network-connected computing devices to retrieve, present, and traverse information resources on the World Wide Web. A suitable web browser is, by way of non-limiting example, Microsoft Internet Explorer. (登録商標) , Mozilla Firefox (登録商標) , Google Chrome (登録商標) , Apple Safari (登録商標) , Opera Software Opera (登録商標) , and KDE Konqueror (登録商標) In some embodiments, the web browser is a mobile web browser. Mobile web browsers (also called microbrowsers, minibrowsers, and wireless browsers) are designed for use on mobile computing devices, including, by way of non-limiting examples, handheld computers, tablet computers, netbook computers, sub-node computer, smartphones, music players, personal digital assistants (PDAs), and handheld video game systems. Suitable mobile web browsers include, by way of non-limiting examples, Google Android (登録商標) Browser, RIM BlackBerry (登録商標) Browser: Apple Safari (登録商標) , Palm Blazer (登録商標) , Palm WebOS (登録商標) Browser, Mozilla Firefox for mobile (登録商標) , Microsoft Internet Explorer Mobile (登録商標) , Amazon Kindle Basic Web (登録商標) , Nokia Browser (登録商標) , Opera Software Opera Mobile (登録商標) , and Sony PSP (登録商標) Including the browser. Software Module

[0160] In some embodiments, the platforms, systems, media, and methods disclosed herein include software, server, and / or database modules, or the use thereof. In view of the disclosure provided herein, software modules are created by machine, software, and language techniques. The software modules disclosed herein are implemented in numerous ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, or a combination thereof. In further various embodiments, a software module comprises multiple files, multiple sections of code, multiple programming objects, multiple programming structures, or a combination thereof. In various embodiments, one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application. In some embodiments, a software module is within one computer program or application. In other embodiments, a software module is within more than one computer program or application. In some embodiments, a software module is hosted on one machine. In other embodiments, a software module is hosted on more than one machine. In further embodiments, a software module is hosted on a distributed computing platform, such as a cloud computing platform. In some embodiments, the software modules are hosted on one or more machines in one location. In other embodiments, the software modules are hosted on one or more machines in more than one location. Database

[0161] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more databases (DBs) or the use thereof. In view of the disclosure provided herein, many databases are suitable for storing and retrieving data. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object-oriented databases, object databases, entity-relationship model databases, associative databases, XML databases, time-series databases, graph databases, and the like. Further non-limiting examples are SQL, PostgreSQL, (登録商標) , MySQL (登録商標) , Oracle (登録商標) , DB2 (登録商標) and Sybase. In some embodiments, the database is internet-based. In further embodiments, the database is web-based. In still further embodiments, the database is cloud computing-based. In certain embodiments, the database is a distributed database. In other embodiments, the database is based on one or more local computer storage devices. Terms and Definitions

[0162] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0163] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Any reference to "or" herein is intended to include "and / or" unless stated otherwise.

[0164] As used herein, the term "about" refers to an amount that, in some instances, is approximately the stated amount.

[0165] As used herein, the term "about" refers to an amount to the nearest 10%, 5%, or 1% (including increments therein) of the recited amount.

[0166] As used herein, the term "about" in connection with a percentage refers to an amount that is 10%, 5%, or 1% (including increments therein) more or less than the stated percentage.

[0167] As used herein, the phrases "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, respectively.

[0168] While preferred embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. It is not intended that the present disclosure be limited by the specific examples provided within. While the present disclosure has been described with reference to the foregoing specification, the description and illustration of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions may occur without departing from the present disclosure. Furthermore, it should be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations, or relative proportions set forth herein, which depend upon a variety of conditions and variables. It should be understood that various alternatives to the disclosed embodiments described herein may be employed in practicing the present disclosure. It is therefore contemplated that the present disclosure also covers any such alternatives, modifications, variations, or equivalents. The following claims define the scope of the present disclosure, and it is intended that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

[Claim 1] The invention described in this specification.

Images