Systems and methods for end-to-end optimization of process control or monitoring

Abstract

To provide systems and methods for optimizing process control or monitoring of manufacturing processes in manufacturing environments.SOLUTION: Systems and methods can generate predictions or recommendations for process variables, target properties, or root causes of anomalies. Systems can include data and machine learning layers, and the data and machine learning layers can include: a data collector configured to receive data from a client application layer; a dataset generator configured to enable a user to create customized datasets from the data; a model management module configured to enable the user to build, train and / or update machine learning models; and an inference module configured to use the machine learning models for generating predictions or recommendations.SELECTED DRAWING: None

Description

[Background technology]

[0001] (background) Modern manufacturing processes, such as semiconductor manufacturing processes, are becoming increasingly complex due to the amount of data generated. Typical systems and methods may generate sparse measurement data that is not sufficient to provide optimization of process control and monitoring. Optimizing process control and monitoring can help reduce the burden and cost of processing and analyzing larger data sets to improve the manufacturing process. Optimizing process control and monitoring can include the use of systems and methods for virtual metrology (VM), image metrology (IM), and root cause analysis (RCA). VM and IM can help generate predictive measurements, thereby expanding data sample sizes for more precise statistical measurements in statistical process control (SPC) or advanced process control (APC). RCA can help determine, predict, or causally analyze anomalies. Untimely control or monitoring of an abnormal manufacturing process can result in further processing anomalies, which generate alarms and interrupt the manufacturing process until the anomaly is corrected.

[0002] Optimization of process control and monitoring may result from the complementary use of VM, IM, and RCA. The complementary use of VM, IM, and RCA may improve or assist in reducing process variability, reliably managing predictive maintenance, monitoring equipment performance degradation (e.g., data drift or data shift), relating abnormal processes to yield, increasing equipment availability, expanding equipment coverage, or improving input data quality for SPC or APC. In addition, optimization of process control and monitoring may benefit from interaction with a user, e.g., an expert such as a process engineer. User interaction may include, for example, overriding process defaults, incorporating user preferences into the process, incorporating decisions and insights into the process, or experimenting through user-defined settings.

[0003] Furthermore, VM systems and methods can be useful for increasing productivity, improving quality, and reducing maintenance costs in semiconductor manufacturing by replacing typical physical metrology. Many implementations of VM can rely on predictive modeling approaches that can leverage equipment sensor data to estimate useful process outcome variables. Despite the advantages of VM implementations, the limited deployment of VM in manufacturing is due to accuracy and scalability issues, while ML-based VM faces challenges such as data drift, shift, sparsity, and inconsistent quality, especially in semiconductor processes.

[0004] A need exists for better systems and methods for optimizing process control and monitoring using VM, IM, and RCA, and for enabling user interaction thereof. Additionally, a need exists for improved systems and methods that can utilize at least rare or sparse data to optimize process control and monitoring in high volume manufacturing. Summary of the Invention [Means for solving the problem]

[0005] (overview) Disclosed herein is a system for optimizing process control or monitoring of a manufacturing process in a manufacturing environment. Optimizing process control or monitoring may include systems or methods associated with virtual metrology (VM), image metrology (IM), or root cause analysis (RCA). Process control or monitoring may be performed using, for example, statistical process control (SPC) or advanced process control (APC).

[0006] In one aspect, a system for optimizing process control or monitoring of a manufacturing environment is disclosed herein, the system comprising: a data and machine learning layer configured to interface with a client application layer associated with the manufacturing environment, the data and machine learning layer comprising: (1) a data collector configured to receive data from at least one source in the client application layer; (2) a dataset generator configured to enable a user to create one or more customized datasets from the data; (3) a model management module configured to enable a user to build, train, and / or update one or more machine learning models, at least in part, using the one or more customized datasets; and (4) an inference module configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or when equipment is being operated in the manufacturing environment; and a plurality of data stores configured to store at least data, new data updates, the one or more customized datasets, and the one or more machine learning models. In some embodiments, the one or more predictions are generated based on virtual metrology (VM). In some embodiments, the one or more machine learning models comprise aggregated adaptive online models (AggAOM). In some embodiments, the AggAOM is configured to take advantage of commonalities in the instruments and associated chambers or stations, the commonalities including one or more of control system architecture, sensor types, process flows, or quality control procedures. In some embodiments, the AggAOM is configured to employ a hierarchical structure of the instruments and associated chambers or stations. In some embodiments, the hierarchical structure includes at least two levels of hierarchy.In some embodiments, the hierarchical structure includes: (1) an equipment level that assigns a global model to each piece of equipment, the global model configured to process commonalities to evaluate a first set of residuals occurring across various chambers or stations associated with each piece of equipment; (2) a chamber level that generates a chamber model of higher resolution than the global model, the chamber model configured to process commonalities to evaluate a second set of residuals; and (3) a station level that includes a station-related model of higher resolution than the chamber model, the station-related model configured to process at least the first and second sets of residuals to generate one or more predictions. In some embodiments, the multiple data stores comprise a database configured to store data received from at least one source in the client application layer. In some embodiments, the multiple data stores comprise an object store configured to store one or more model data files associated with the one or more machine learning models. In some embodiments, the system further comprises a publishing module configured to publish one or more predictions for integration into a process control workflow associated with the manufacturing environment. In some embodiments, the one or more predictions can be used to enable substantially real-time characterization, monitoring, control, and / or modification to or to optimize one or more processes and / or equipment in the manufacturing environment. In some embodiments, the system further comprises a model performance evaluation module configured to generate one or more performance metrics for the one or more machine learning models. In some embodiments, the one or more performance metrics are derived from at least a sensitivity analysis or stress testing. In some embodiments, the system further comprises a front-end user management module comprising a graphical user interface (GUI) configured to display the one or more performance metrics on a dashboard.In some embodiments, the data comprises equipment data, process data, metadata, and / or measurement data. In some embodiments, the data collector is configured to synchronously receive data from at least two different sources. In some embodiments, the data collector is configured to receive data from at least one source based, at least in part, on a predetermined schedule. In some embodiments, the data collector is configured to receive data from at least one source based, at least in part, on detection of one or more events occurring within the manufacturing environment. In some embodiments, the one or more events are associated with a drift, excursion, shift, deviation, or anomaly in at least one process or process equipment within the manufacturing environment. In some embodiments, the data comprises a plurality of data types and datasets comprising: (1) historical process data, (2) current process data, (3) historical measurement data of one or more metrics, (4) current measurement data of one or more metrics, (5) operational data, or (6) equipment specification metadata. In some embodiments, the dataset generator is configured to automate selection of one or more features in the one or more customized datasets. In some embodiments, the model management module is configured to train one or more machine learning models including hyperparameter optimization. In some embodiments, the hyperparameter optimization is performed based on the one or more performance metrics when the one or more performance metrics degrade below a predetermined threshold. In some embodiments, the manufacturing environment is associated with at least one of semiconductor manufacturing, LCD display manufacturing, solar panel manufacturing, electronics manufacturing, battery manufacturing, automotive manufacturing, or pharmaceutical manufacturing.

[0007] In another aspect, disclosed herein is a computer program product for optimizing process control or monitoring of a manufacturing environment, the computer program product comprising at least one non-transitory computer readable medium having computer readable program code portions embodied therein, the computer readable program code portions comprising an executable portion configured to interface with a client application layer associated with the manufacturing environment, the executable portion comprising: (1) an executable portion configured to receive data from at least one source in the client application layer; (2) an executable portion configured to enable a user to create one or more customized data sets from the data; (3) an executable portion configured to enable a user to build, train, and / or update one or more machine learning models using, at least in part, the one or more customized data sets; and (4) an executable portion configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or equipment is being operated in the manufacturing environment.

[0008] Further, the present disclosure provides improved systems and methods that can utilize at least sparse or scarce data to optimize process control and monitoring in high volume manufacturing. For example, an Aggregate Adaptive Online Model (AggAOM) is disclosed herein that can effectively solve the technical challenge of data scarcity or sparsity in at least virtual metrology (VM). Data scarcity or sparsity can result from post-process measurements, such as thickness and mass, due to limited physical measurement ratios (e.g., typically less than 5%). In some cases, data scarcity or sparsity can be determined as the number of training samples, which can be further determined by two factors, such as sampling rate and data collection duration. By leveraging the hierarchical structure of manufacturing equipment, AggAOM can capture and utilize the fundamental commonality between equipment chambers in the same hierarchical level in addition to their individual variations. The systems and methods herein can enable more efficient use of limited data (e.g., sparse or scarce data) and can substantially improve the predictive accuracy of VMs employed in high volume manufacturing facilities, such as megafabs or semiconductor fabs, in a statistically significant manner. The improvements herein have been demonstrated by utilizing a data set generated from SK hynix over a period of time (e.g., 9 months) and demonstrating that AggAOM is significantly superior to other models in accuracy and usefulness. Thus, the systems and methods herein may provide improved technical solutions for optimizing VM for mass manufacturing, such as semiconductor manufacturing.

[0009] Additional aspects and advantages of the present disclosure will become readily apparent from the following detailed description, in which only illustrative embodiments of the disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the present disclosure. Thus, the drawings and description are to be regarded as illustrative in nature, and not as restrictive. The present invention provides, for example, the following items. (Item 1) 1. A system for optimizing process control or monitoring of a manufacturing environment, the system comprising: a data and machine learning layer configured to interface with a client application layer associated with a manufacturing environment, the data and machine learning layer comprising: (1) a data collector configured to receive data from at least one source in the client application layer; (2) a dataset generator configured to enable a user to create one or more customized datasets from the data; (3) a model management module configured to enable the user to build, train, and / or update one or more machine learning models using, at least in part, the one or more customized datasets; and (4) an inference module configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or equipment is being operated in the manufacturing environment; a plurality of data stores configured to store at least the data, new data updates, the one or more customized data sets, and the one or more machine learning models; A system comprising: (Item 2) The system of any preceding claim, wherein the one or more predictions are generated based on virtual measurement (VM). (Item 3) 2. The system of claim 1, wherein the one or more machine learning models comprise an aggregated adaptive online model (AggAOM). (Item 4) The system of any one of the preceding items, wherein the AggAOM is configured to take advantage of commonalities in the equipment and associated chambers or stations, the commonalities including one or more of control system architecture, sensor types, process flows, or quality control procedures. (Item 5) The system of any one of the preceding claims, wherein the AggAOM is configured to adopt a hierarchical structure of the instruments and associated chambers or stations. (Item 6) 2. The system of claim 1, wherein the hierarchical structure includes at least two levels of hierarchy. (Item 7) The system of any one of the preceding items, wherein the hierarchical structure includes: (1) an equipment level that assigns a global model to each piece of equipment, the global model configured to process commonalities to evaluate a first set of residuals occurring across various chambers or stations associated with each piece of equipment; (2) a chamber level that generates a chamber model of higher resolution than the global model, the chamber model configured to process commonalities to evaluate a second set of residuals; and (3) a station level that includes a station-related model of higher resolution than the chamber model, the station-related model configured to process at least the first set of residuals and the second set of residuals to generate the one or more predictions. (Item 8) 2. The system of claim 1, wherein the plurality of data stores comprises a database configured to store the data received from the at least one source within the client application layer. (Item 9) 2. The system of claim 1, wherein the plurality of data stores comprises an object storage device configured to store one or more model data files associated with the one or more machine learning models. (Item 10) 2. The system of claim 1, further comprising a publishing module configured to publish the one or more predictions for integration into a process control workflow associated with the manufacturing environment. (Item 11) The system of any one of the preceding claims, wherein the one or more predictions are usable to enable or optimize characterization, monitoring, control, and / or modification to the one or more processes and / or equipment in the manufacturing environment in substantially real time. (Item 12) 2. The system of claim 1, further comprising a model performance evaluation module configured to generate one or more performance metrics for the one or more machine learning models. (Item 13) 20. The system of claim 19, wherein the one or more performance metrics are derived from at least a sensitivity analysis or a stress test. (Item 14) The system of any one of the preceding claims, further comprising a front-end user management module having a graphical user interface (GUI) configured to display the one or more performance metrics on a dashboard. (Item 15) 20. The system of claim 19, wherein the data comprises equipment data, process data, metadata, and / or measurement data. (Item 16) 2. The system of claim 1, wherein the data collector is configured to synchronously receive the data from at least two different sources. (Item 17) 20. The system of claim 1, wherein the data collector is configured to receive the data from the at least one source based, at least in part, on a predetermined schedule. (Item 18) 2. The system of claim 1, wherein the data collector is configured to receive the data from the at least one source based, at least in part, on detection of one or more events occurring within the manufacturing environment. (Item 19) The system of any one of the preceding claims, wherein the one or more events are associated with a drift, excursion, shift, deviation, or anomaly in at least one process or process equipment within the manufacturing environment. (Item 20) The system of any one of the preceding claims, wherein the data comprises a plurality of data types and data sets comprising: (1) historical process data, (2) current process data, (3) historical measurement data of one or more metrics, (4) current measurement data of one or more metrics, (5) operational data, or (6) equipment specification metadata. (Item 21) 2. The system of claim 1, wherein the dataset generator is configured to automate selection of one or more features in the one or more customized datasets. (Item 22) 20. The system of claim 1, wherein the model management module is configured to train the one or more machine learning models including hyperparameter optimization. (Item 23) 2. The system of claim 1, wherein the hyperparameter optimization is performed based on one or more performance metrics when the one or more performance metrics degrade below a predetermined threshold level. (Item 24) The system of any one of the preceding claims, wherein the manufacturing environment is associated with at least one of semiconductor manufacturing, LCD display manufacturing, solar panel manufacturing, electronics manufacturing, battery manufacturing, automotive manufacturing, or pharmaceutical manufacturing. (Item 25) 1. A computer program product for optimizing process control or monitoring of a manufacturing environment, the computer program product comprising at least one non-transitory computer readable medium, the at least one non-transitory computer readable medium having computer readable program code portions embodied in the at least one non-transitory computer readable medium; 11. The computer program product of claim 10, wherein the computer readable program code portions comprise an executable portion configured to interface with a client application layer associated with a manufacturing environment, the executable portion comprising: (1) an executable portion configured to receive data from at least one source in the client application layer; (2) an executable portion configured to enable a user to create one or more customized data sets from the data; (3) an executable portion configured to enable the user to build, train, and / or update one or more machine learning models, at least in part, using the one or more customized data sets; and (4) an executable portion configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or equipment is being operated in the manufacturing environment. (Summary) The system and method may generate predictions or recommendations regarding process variables, target properties, or root causes of anomalies. The system may include a data and machine learning layer, which may include a data collector configured to receive data from a client application layer, a dataset generator configured to allow a user to create customized datasets from the data, a model management module configured to allow a user to build, train, and / or update machine learning models, and an inference module configured to use the machine learning models to generate predictions or recommendations. The machine learning models may include aggregated adaptive online models (AggAOM) for generating predictions using scarce or sparse data.

[0010] (Incorporated by reference) All publications, patents, and patent applications mentioned herein are incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, it is intended that the present specification take precedence and / or supersede any such conflicting material. [Brief description of the drawings]

[0011] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings.

[0012] [Figure 1A]1A-1C illustrate non-limiting examples for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, including (i) a high-level relationship between virtual metrology (VM), image metrology (IM), and root cause analysis (RCA) of FIG. 1A, (ii) a system architecture for the VM of FIG. 1B, and (iii) a high-level system workflow for the VM of FIG. 1C, in accordance with some embodiments. [Figure 1B] 1A-1C illustrate non-limiting examples for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, including (i) a high-level relationship between virtual metrology (VM), image metrology (IM), and root cause analysis (RCA) of FIG. 1A, (ii) a system architecture for the VM of FIG. 1B, and (iii) a high-level system workflow for the VM of FIG. 1C, in accordance with some embodiments. [Figure 1C] 1A-1C illustrate non-limiting examples for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, including (i) a high-level relationship between virtual metrology (VM), image metrology (IM), and root cause analysis (RCA) of FIG. 1A, (ii) a system architecture for the VM of FIG. 1B, and (iii) a high-level system workflow for the VM of FIG. 1C, in accordance with some embodiments.

[0013] [Figure 2A] 2A-2B illustrate non-limiting examples of manufacturing processes that may utilize virtual metrology (VM), image metrology (IM), or root cause analysis (RCA) in accordance with some embodiments. Figure 2A illustrates an exemplary manufacturing process. Figure 2B illustrates an exemplary metrology. [Figure 2B] 2A-2B illustrate non-limiting examples of manufacturing processes that may utilize virtual metrology (VM), image metrology (IM), or root cause analysis (RCA) in accordance with some embodiments. Figure 2A illustrates an exemplary manufacturing process. Figure 2B illustrates an exemplary metrology.

[0014] [Figure 3A]3A-3F illustrate non-limiting examples of high-level artificial intelligence (AI) workflows for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments. FIG. 3A illustrates an exemplary high-level architecture or workflow. FIG. 3B illustrates an exemplary data interface for solution engineering or customer system integration. FIG. 3C illustrates an exemplary product data layer for receiving or fetching data from the data interface. FIG. 3D illustrates an exemplary infrastructure module or layer for facilitating predictions or recommendations. FIG. 3E illustrates an exemplary AI layer for developing or deploying machine learning (ML) or inference models. FIG. 3F illustrates an exemplary product engineering layer for providing predictions, recommendations, performance metrics, and the like. [Figure 3B] 3A-3F illustrate non-limiting examples of high-level artificial intelligence (AI) workflows for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments. FIG. 3A illustrates an exemplary high-level architecture or workflow. FIG. 3B illustrates an exemplary data interface for solution engineering or customer system integration. FIG. 3C illustrates an exemplary product data layer for receiving or fetching data from the data interface. FIG. 3D illustrates an exemplary infrastructure module or layer for facilitating predictions or recommendations. FIG. 3E illustrates an exemplary AI layer for developing or deploying machine learning (ML) or inference models. FIG. 3F illustrates an exemplary product engineering layer for providing predictions, recommendations, performance metrics, and the like. [Figure 3C]3A-3F illustrate non-limiting examples of high-level artificial intelligence (AI) workflows for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments. FIG. 3A illustrates an exemplary high-level architecture or workflow. FIG. 3B illustrates an exemplary data interface for solution engineering or customer system integration. FIG. 3C illustrates an exemplary product data layer for receiving or fetching data from the data interface. FIG. 3D illustrates an exemplary infrastructure module or layer for facilitating predictions or recommendations. FIG. 3E illustrates an exemplary AI layer for developing or deploying machine learning (ML) or inference models. FIG. 3F illustrates an exemplary product engineering layer for providing predictions, recommendations, performance metrics, and the like. [Figure 3D] 3A-3F illustrate non-limiting examples of high-level artificial intelligence (AI) workflows for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments. FIG. 3A illustrates an exemplary high-level architecture or workflow. FIG. 3B illustrates an exemplary data interface for solution engineering or customer system integration. FIG. 3C illustrates an exemplary product data layer for receiving or fetching data from the data interface. FIG. 3D illustrates an exemplary infrastructure module or layer for facilitating predictions or recommendations. FIG. 3E illustrates an exemplary AI layer for developing or deploying machine learning (ML) or inference models. FIG. 3F illustrates an exemplary product engineering layer for providing predictions, recommendations, performance metrics, and the like. [Figure 3E]3A-3F illustrate non-limiting examples of high-level artificial intelligence (AI) workflows for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments. FIG. 3A illustrates an exemplary high-level architecture or workflow. FIG. 3B illustrates an exemplary data interface for solution engineering or customer system integration. FIG. 3C illustrates an exemplary product data layer for receiving or fetching data from the data interface. FIG. 3D illustrates an exemplary infrastructure module or layer for facilitating predictions or recommendations. FIG. 3E illustrates an exemplary AI layer for developing or deploying machine learning (ML) or inference models. FIG. 3F illustrates an exemplary product engineering layer for providing predictions, recommendations, performance metrics, and the like. [Figure 3F] 3A-3F illustrate non-limiting examples of high-level artificial intelligence (AI) workflows for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments. FIG. 3A illustrates an exemplary high-level architecture or workflow. FIG. 3B illustrates an exemplary data interface for solution engineering or customer system integration. FIG. 3C illustrates an exemplary product data layer for receiving or fetching data from the data interface. FIG. 3D illustrates an exemplary infrastructure module or layer for facilitating predictions or recommendations. FIG. 3E illustrates an exemplary AI layer for developing or deploying machine learning (ML) or inference models. FIG. 3F illustrates an exemplary product engineering layer for providing predictions, recommendations, performance metrics, and the like.

[0015] [Figure 4] FIG. 4 illustrates a non-limiting example of an AI workflow for optimizing process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments.

[0016] [Diagram 5] FIG. 5 illustrates a non-limiting example of feature engineering to optimize process control or monitoring of a manufacturing process in a manufacturing environment, according to some embodiments.

[0017] [Figure 6] FIG. 6 illustrates a non-limiting example of data for optimizing process control or monitoring of a manufacturing process in a manufacturing environment of the present disclosure.

[0018] [Figure 7] FIG. 7 illustrates a high-level architecture or flow of an adaptive online model (AOM) according to some embodiments.

[0019] [Figure 8A] 8A-8B illustrate examples of how different models may utilize instruments, chambers, and stations, according to some embodiments. Figure 8A illustrates an example of instruments, chambers, and stations used by a single model. Figure 8B illustrates an example of instruments, chambers, and stations used by a station-related model. [Figure 8B] 8A-8B illustrate examples of how different models may utilize instruments, chambers, and stations, according to some embodiments. Figure 8A illustrates an example of instruments, chambers, and stations used by a single model. Figure 8B illustrates an example of instruments, chambers, and stations used by a station-related model.

[0020] [Figure 9] FIG. 9 illustrates an example of how the Aggregated Adaptive Online Model (AggAOM) herein may utilize instruments, chambers, and stations, according to some embodiments.

[0021] [Figure 10]FIG. 10 illustrates an example of normalized wafer thickness for three pieces of equipment in a semiconductor manufacturing process, according to some embodiments.

[0022] [Figure 11] FIG. 11 illustrates an example high-level architecture or flow of the Aggregative Adaptive Online Model (AggAOM) of the present specification, according to some embodiments.

[0023] [Figure 12A] 12A-12D illustrate a comparison of performance between different models, e.g., a linear model, a partial least squares (PLS) model, an XGBoost (XGB) model, an adaptive online model (AOM), and an aggregated adaptive online model (AggAOM) of the present specification, according to some embodiments. FIG. 12A compares the performance, e.g., r-squared (R2), using a first chemical vapor deposition (CVD1) dataset. FIG. 12B compares the performance, e.g., root mean square error (RMSE), using a first chemical vapor deposition (CVD1) dataset. FIG. 12C compares the performance, e.g., R2, using a second chemical vapor deposition (CVD2) dataset. FIG. 12D compares the performance, e.g., RMSE, using a second chemical vapor deposition (CVD1) dataset. [Figure 12B] 12A-12D illustrate a comparison of performance between different models, e.g., a linear model, a partial least squares (PLS) model, an XGBoost (XGB) model, an adaptive online model (AOM), and an aggregated adaptive online model (AggAOM) of the present specification, according to some embodiments. FIG. 12A compares the performance, e.g., r-squared (R2), using a first chemical vapor deposition (CVD1) dataset. FIG. 12B compares the performance, e.g., root mean square error (RMSE), using a first chemical vapor deposition (CVD1) dataset. FIG. 12C compares the performance, e.g., R2, using a second chemical vapor deposition (CVD2) dataset. FIG. 12D compares the performance, e.g., RMSE, using a second chemical vapor deposition (CVD1) dataset. [Figure 12C] 12A-12D illustrate a comparison of performance between different models, e.g., a linear model, a partial least squares (PLS) model, an XGBoost (XGB) model, an adaptive online model (AOM), and an aggregated adaptive online model (AggAOM) of the present specification, according to some embodiments. FIG. 12A compares the performance, e.g., r-squared (R2), using a first chemical vapor deposition (CVD1) dataset. FIG. 12B compares the performance, e.g., root mean square error (RMSE), using a first chemical vapor deposition (CVD1) dataset. FIG. 12C compares the performance, e.g., R2, using a second chemical vapor deposition (CVD2) dataset. FIG. 12D compares the performance, e.g., RMSE, using a second chemical vapor deposition (CVD1) dataset. [Figure 12D] 12A-12D illustrate a comparison of performance between different models, e.g., a linear model, a partial least squares (PLS) model, an XGBoost (XGB) model, an adaptive online model (AOM), and an aggregated adaptive online model (AggAOM) of the present specification, according to some embodiments. FIG. 12A compares the performance, e.g., r-squared (R2), using a first chemical vapor deposition (CVD1) dataset. FIG. 12B compares the performance, e.g., root mean square error (RMSE), using a first chemical vapor deposition (CVD1) dataset. FIG. 12C compares the performance, e.g., R2, using a second chemical vapor deposition (CVD2) dataset. FIG. 12D compares the performance, e.g., RMSE, using a second chemical vapor deposition (CVD1) dataset.

[0024] [Figure 13] FIG. 13 illustrates a non-limiting example of a computing system according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] (Detailed Description) While various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, modifications, or substitutions may occur without departing from the present disclosure. It will be understood that various alternatives to the embodiments of the present disclosure described herein may be employed.

[0026] Typical systems for optimizing process control or monitoring in manufacturing processes are inadequate. The systems described herein may include connected systems associated with virtual metrology (VM), image metrology (IM), or root cause analysis (RCA) for optimizing process control or monitoring in manufacturing processes.

[0027] In VM or IM, measurement data on individual processes or across multiple processes or different process areas can be important for the quality of process monitoring. The manufacturing process can include, for example, semiconductor manufacturing processes for memory and logic devices. The quality of monitoring of the deformations occurring at the end of each process step can be extremely expensive or even prohibitive in terms of time, cost, and technology. As a result, the manufacturing process typically relies on very limited data, e.g., the number of sample measurements. The sampling size is further limited in the case of destructive sampling scenarios where the product resulting in the sample is destroyed or rendered unusable in the process. The limited sampling resulting from the above issues results in suboptimal process control or monitoring at various levels. Virtual metrology provides the ability to effectively predict the measurements of the product, thereby complementing the physical sampling of the product while avoiding the associated costs. Image metrology optimizes traditional metrology by generating more measurement information on the same image. The ability to sample the entire product space after each process step can enable very rigorous process control or monitoring optimization resulting in reduced variability, yield optimization, and reduced lost revenue due to scrap (e.g., inoperable product), equipment downtime, and measurement costs.

[0028] In RCA, typical systems may not be capable of modeling data drift, data shift, or hierarchical data structures observed in manufacturing processes, such as semiconductor manufacturing processes. Data drift may generally refer to gradual changes in process dynamics, for example, due to aging of equipment used in the manufacturing process. Data shift may generally refer to abrupt changes in process dynamics, for example, due to external operations, such as maintenance or calibration. Advanced process control (APC) or statistical process control (SPC) methods associated with VM or IM may require accurate prediction of process variables or target properties in manufacturing processes. However, some systems may not be capable of modeling non-stationarity observed in real or actual data, for example, current or historical sensor data. Non-stationarity may be generated in semiconductor manufacturing processes, such as, for example, chemical vapor deposition (CVD), etching, diffusion, or other processes. For example, data drift may cause a steady or slow decrease in the accuracy of predictions associated with the VM or IM model. Data shift may cause abrupt failure of the VM or IM model. Some VM or IM systems that use moving window based methods may be inaccurate, for example, due to selecting a small subset of data. Some VM or IM systems that use just-in-time learning methods may not adapt to changes in the underlying relationships between process variables or target properties.

[0029] Features of the system described herein may include, for example, domain support features, machine learning (ML) features, and product features. Domain support features may include features associated with enabling a user (e.g., an expert such as a process engineer) to incorporate domain knowledge into a process. Incorporating domain knowledge may include creating custom process experiments, generating custom logic (e.g., custom process recipes), comparing or assessing user performance against ML models for VM, RCA, and IM, including features and datasets based on domain information in a process experiment, or defining performance and evaluation metrics based on domain information. Machine learning features may include features associated with building both automated and custom VM, RCA, and IM models, benchmarking and evaluating a wide variety of models, provisioning model performance and domain-specific information for either automated or interactive measurement, experiment building for custom ML models, in-product model building, refining and tuning in response to changing operating conditions (e.g., adapting to data drift and shifts), automating scheduling for model retraining, or supporting user data privacy by supporting ML model training and inference processes in a deployed product environment as needed by the user. Product features may include features associated with supporting different data and computing infrastructures (e.g., on-site services, cloud services, custom services, or a combination thereof), or modularizing product features to enable use across and beyond supported use cases (e.g., VM output for monitoring equipment health).

[0030] (Systems for optimizing process control or monitoring) (System Overview) In an aspect, a system for optimizing process control or monitoring of a manufacturing process in a manufacturing environment is disclosed herein. The system may include a data and machine learning layer configured to interface with a client application layer associated with the manufacturing environment. The data and machine learning layer may include a data collector configured to receive data from at least one source in the client application layer. The data and machine learning layer may include a dataset generator configured to enable a user to create one or more customized datasets from the data. The data and machine learning layer may include a model management module configured to enable a user to build, train, and / or update one or more machine learning models, at least in part, using the one or more customized datasets. The data and machine learning layer may include an inference module configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or when equipment is being operated in the manufacturing environment. The system may include a plurality of data stores configured to store at least data, new data updates, one or more customized datasets, and one or more machine learning models. In some embodiments, the plurality of data stores comprises a database configured to store data received from at least one source in the client application layer.

[0031] Some VM systems may model processes and characteristics associated with controlling or monitoring manufacturing processes, e.g., processing of wafers in the semiconductor industry. Statistically modeling the processes and characteristics may, in part, use measurements from current or historical data, e.g., current or historical sensor data. Virtual metrology systems may provide more accurate measurements, for example, to control or monitor manufacturing processes. Virtual metrology systems and methods may increase productivity, improve quality, or reduce maintenance costs, for example, when compared to physical inspection of manufacturing processes using traditional metrology. For example, VM systems and methods may be capable of sampling all or substantially all units (e.g., semiconductor wafers) in a manufacturing process, while a human operator using traditional metrology may only be able to sample a small portion of the units. Virtual metrology systems and methods may, in part, use ML methods to predict process variables or target properties to optimize process control or monitoring in manufacturing processes.

[0032] Typical VM systems are inadequate for at least several reasons. Some systems may not be able to model data drift, data shift, or hierarchical data structures observed in manufacturing processes, such as semiconductor manufacturing processes. Data drift may generally refer to gradual changes in process dynamics due to, for example, aging of equipment used in the manufacturing process. Data shift may generally refer to abrupt changes in process dynamics due to, for example, external operations, such as maintenance or calibration. Advanced process control methods associated with VM may require accurate prediction of process variables or target properties in manufacturing processes. However, some systems may not be able to model non-stationarity observed in real or actual data, e.g., current or historical sensor data. Non-stationarity may be generated in semiconductor manufacturing processes, such as, for example, chemical vapor deposition (CVD), etching, diffusion, or other processes. For example, data drift may cause a steady or slow decrease in the accuracy of predictions associated with the VM model. Data shift may cause abrupt failure of the VM model. Some VM systems using moving window-based methods may be inaccurate, for example, due to selecting a small subset of data. Some VM systems that use just-in-time learning methods cannot adapt to changes in the underlying relationships between process variables or target properties.

[0033] In some embodiments, the manufacturing environment is associated with at least one of semiconductor manufacturing, electronics manufacturing, battery manufacturing, automotive manufacturing, or pharmaceutical manufacturing. A system described herein (e.g., a system for VM, RCA, or IM) may be used to optimize process control or monitoring in the manufacturing environment. For example, as illustrated in FIG. 1A, the manufacturing environment may be associated with a legacy manufacturing system 110. The legacy manufacturing system 110 may be associated with legacy data sources and legacy software systems.

[0034] As further illustrated in FIG. 1A , the legacy manufacturing system 110 may be associated with legacy data sources configured to generate or transmit data from the legacy manufacturing system 110. The data sources may include data generated or received from process equipment or machines, metrology tools or machines, inspection equipment or machines, transportation equipment or machines, process sensors, or equipment sensors. Data from the data sources may include contemporaneously generated or received data (e.g., real-time data) or non-contemporaneously received or generated data (e.g., historical data). The systems described herein may use the current data or historical data to build a more accurate model (e.g., a VM, RCA, or IM model) or improve its existing model.

[0035] 1A, the legacy manufacturing system 110 may be associated with legacy software systems configured to receive data from, analyze data from, or generate data from data sources. The legacy software systems may include manufacturing execution systems (MES), advanced process control (APC) systems, statistical process control (SPC) systems, recipe management (RMS) systems, and the like. The legacy manufacturing system 110 may be associated with a data hub configured to receive, store, or transmit data.

[0036] As further illustrated in FIG. 1A , the system 120 may receive data from the legacy manufacturing system 110 to generate predictions or recommendations for optimizing the control or monitoring of the manufacturing process. The data may include raw data (e.g., unprocessed data) and processed data for use by the system 120 to train ML models for optimizing the process control or monitoring of the manufacturing process. The raw data received from the legacy manufacturing system 110 may include, for example, sensor trace data, process responses, logs, images, and the like, as described elsewhere herein. In some cases, the system 120 may include a data interface, as described elsewhere herein, to receive, process, or transmit the data. The data interface may be configured to collect data, validate data, or wrangle data.

[0037] As further illustrated in FIG. 1A , the system 120 may be configured with artificial intelligence (AI) applications to develop ML models for generating predictions or recommendations. The predictions or recommendations may be used to optimize the control or monitoring of the legacy manufacturing system 110. The machine learning models may include inference models for generating insights from the development of the ML models, such as insights from training, testing, tuning, or validating the ML models. In some cases, the system 120 may include a serving agent configured in a feedback loop to transmit or publish predictions, recommendations, or insights for optimizing the legacy manufacturing system 110.

[0038] 1A , the system 120 may be configured to utilize infrastructure 130 described elsewhere herein to facilitate the generation of ML models to optimize process control or monitoring of the legacy manufacturing system 110. The infrastructure 130 may include services configured on-site, services configured in one or more cloud services, ML frameworks, distributed data processing systems, cloud computing frameworks, and the like, to optimize process control or monitoring of a manufacturing process in a manufacturing environment.

[0039] The manufacturing environment may be associated with a semiconductor manufacturing environment having a semiconductor manufacturing process. For example, as illustrated in FIGS. 2A-2B, the semiconductor manufacturing environment may be associated with semiconductor manufacturing equipment configured to perform the semiconductor manufacturing process. The system 120 may generate predictions or recommendations for optimizing the semiconductor manufacturing process, e.g., process control or monitoring of a process variable, a target property, or a root cause of an anomaly. The process may include a wafer fabrication process, e.g., ingot pulling, ingot slicing, wafer grinding, or oxidation. The process may include a mask fabrication process, e.g., circuit design, pattern design, or photomasking. The process may include a front-end process, e.g., pattern lithography, etching, chemical vapor deposition (e.g., ion implantation, oxidation, diffusion), photoresist coating, or planarization. The process may include intermediate inspection, e.g., inspection of the wafer. The process may include a back-end process, e.g., wafer dicing, chip mounting, wire bonding, molding, trimming, or forming. The process may include intermediate testing, e.g., burn-in temperature or voltage testing. The process may include final inspection or reliability testing. The process may include marking.

[0040] (A system for virtual instrumentation (VM)) 1B is a high-level system architecture 500 (hereinafter "architecture 500") for implementing system 120 of the present disclosure for optimizing process control or monitoring of a manufacturing process in a manufacturing environment. Architecture 500 comprises layers or modules configured to implement system 120. The layers or modules may include an application layer 510, a VM engine layer 520, an AI layer 530, a data and infrastructure layer 540, and a performance management layer 550.

[0041] As further illustrated in FIG. 1B, architecture 500 includes an application layer 510 configured to provide user interaction with system 120. Layer 510 may include layers or modules configured to provide user interaction with systems or processes associated with semiconductor manufacturing processes. The processes may include chemical vapor deposition (CVD) processes, physical vapor deposition (PVD) processes, diffusion (DIFF) processes, chemical mechanical planarization (CMP) processes, etching (D-Mass) processes, etching critical dimension (CD) processes, or processes associated with any semiconductor manufacturing process, such as those illustrated in FIG. 2A.

[0042] As further illustrated in FIG. 1B , architecture 500 includes a VM engine layer 520 configured for process management 521, data engineering 522, and AI pipeline 523. Process management 521 may include modules or layers configured to process metadata from data sources, monitor or retrain an ML pipeline, shadow an ML pipeline, or generate accuracy metrics or analytics associated with an ML pipeline. Data engineering 522 may include modules or layers configured to wrangle data, pre-process data (e.g., filtering, cleaning, augmenting, anomaly detection, standardization, and the like), or select or engineer features. AI pipeline 523 may include modules or layers configured to generate algorithm directories, process algorithm mappings, generate or conduct experiments, generate pipelines, or generate inferences.

[0043] As further illustrated in FIG. 1B , architecture 500 includes an AI layer 530 configured for data engineering 531, learning and optimization 532, inference and accuracy metrics 533, or ML operations 534. Data engineering 531 may include modules or layers configured to process data (e.g., filtering, cleaning, wrangling, and the like), augment data, detect anomalies, standardize data, or select or engineer features. Learning and optimization 532 may include modules or layers configured to determine, generate, or use algorithms for learning or optimization, optimize or learn pipelines, optimize or learn training, or optimize or learn hyperparameters. Inference and accuracy metrics 533 may include modules or layers configured to score functions associated with the ML pipeline. The ML operations 534 may comprise modules or layers configured to generate dataset lineage control, generate dataset version control, generate ML model version control, generate pipeline version control, generate continuous integration (CI) version control, generate continuous deployment (CD) version control, generate job scheduling, generate distributed training and tuning, or serve ML models.

[0044] As further illustrated in FIG. 1B , architecture 500 comprises a data and infrastructure layer 540 configured for orchestration 541, persistence 542, or compute 543. Orchestration 541 may comprise a module or layer configured to provide workflow services (e.g., Apache Airflow) or deployment services (e.g., RedisAI). Persistence 542 may comprise a module or layer configured to provide storage of data (e.g., operational data) in a relational database management system (RDBMS, e.g., MongoDB), provide storage of big data (e.g., Hive, Athena, Delta Lake, and equivalents), or provide object storage (S3, e.g., MinIO). Compute 543 may comprise a module or layer configured to provide on-site computing services (e.g., on-premise K8s, Kubernetes, managed K8s, and equivalents), cloud computing services (e.g., Elastic Kubernetes Services, Apache Spark, and equivalents), or any computing services configured to perform computing associated with developing ML models.

[0045] 1B, architecture 500 comprises a performance management layer 550 configured to receive, generate, or transmit performance data associated with architecture 500. Performance management layer 550 may comprise modules or layers configured with application and performance monitoring (APM), dashboards, alerts, logging, metrics, and the like. In some embodiments, the system further comprises a front-end user management module comprising a graphical user interface (GUI) configured to display one or more performance metrics on a dashboard.

[0046] (High-Level System Workflow for Virtual Instrumentation (VM)) 1C is a high-level system workflow 600 (hereinafter "workflow 600") for implementing the disclosed system 120 or architecture 500 for optimizing process control or monitoring of a manufacturing process in a manufacturing environment. Workflow 600 comprises workflows configured to implement system 120. The workflow may include workflows associated with source data 610, processing and storage 620, VM 630, and post-processing and monitoring 640.

[0047] As further illustrated in FIG. 1C , the workflow 600 comprises a workflow associated with source data 610. The workflow may comprise a workflow associated with receiving, generating, storing, or transmitting data from the legacy manufacturing system 110. The data sources 611 described elsewhere herein may include, for example, processing machines, metrology machines, or inspection machines. The data may be received or transmitted using an application programming interface (API) configured to receive or transmit data. The data from the data sources may be stored in a database configured to store the data. The data may be transmitted in a streaming mode or a batch mode. The data from the source data 610 may be transmitted to processing and storage 620. The processing and storage 620 may receive or fetch data from the source data 610.

[0048] As further illustrated in FIG. 1C , the workflow 600 comprises workflows associated with processing and storage 620. The workflows may comprise workflows associated with data collection services 621 or data platform 622. The data collection services 621 may include a database configured to receive or transmit data from or to the data platform 622. In some cases, the data collection services 621 may use a pub / sub service to transmit data directly to the VM 630. The data collection services 621 may comprise workflows associated with converting data from low resolution data (“bronze”) to medium resolution data (“silver”) and to high resolution data (“gold”). Workflows associated with converting data to medium resolution data may include validating the data, cleaning the data, applying business rules to the data, or other typical data transformations. Workflows associated with converting data to high resolution data may include optimizing the data, indexing the data, replicating the data, or other typical data transformations. In some cases, a user (e.g., a solution designer) may interact with processing and storage 620, for example, to customize or configure processing and storage 620. Data from processing and storage 620 may be transmitted to VM 630. VM 630 may receive or fetch data from source data 610, processing and storage 620.

[0049] As further illustrated in FIG. 1C, workflow 600 comprises a workflow associated with VM 630. The workflow may comprise a workflow associated with application 631. Application 631 described elsewhere herein may include a chemical vapor deposition (CVD) application, a physical vapor deposition (PVD) application, a diffusion (DIFF) application, a chemical mechanical planarization (CMP) application, an etch (D-Mass) application, an etch critical dimension (CD) application, or any semiconductor manufacturing application such as those described in FIG. 2A. A user, e.g., a process engineer, may interact with application 631, for example, to configure or perform an experiment, provide actionable insight, or receive actionable insight. Data from VM 630 may be transmitted to post-processing and monitoring 640. The data may include, for example, predictions, recommendations, inferences, alarms, anomaly scores, or data associated with any data described herein. Post-processing and monitoring 640 may receive or fetch data from VM 630.

[0050] As further illustrated in FIG. 1C , the workflow 600 comprises a workflow associated with post-processing and monitoring 640. The workflow may comprise a workflow associated with publishing services 641 or a monitoring and alerting service 642. Publishing services 641 may comprise a workflow associated with business rules or custom formats. Publishing services 641 may transmit data to legacy manufacturing systems 110, e.g., MES, APC, YMS, SPC, and the like. A user, e.g., a process engineer, may select an ML model. The selected ML model may be a preferred ML model. Monitoring and alerting services 642 may comprise a workflow associated with a dashboard or alerts.

[0051] 1C , workflow 600 comprises a workflow for transmitting or receiving data between processing and storage 620 and post-processing and monitoring 640. The data may include data associated with inference or operational data. Workflow 600 comprises a workflow for transmitting or receiving data between processing and storage 620 and VM 630. The data may include batch data (e.g., fault detection and classification (FDC) data, metrology data, metadata, and the like), data associated with inference, operational data, and the like.

[0052] (High-Level AI Workflow for Virtual Metrology (VM)) Illustrated in Figures 3A-3F is a high-level AI workflow 300 (hereinafter "workflow 300") for implementing the disclosed system 120, architecture 500, or workflow 600 for optimizing process control or monitoring of a manufacturing process in a manufacturing environment. As illustrated in Figure 3A, workflow 300 comprises workflows associated with a data interface 310, a product layer 320, an infrastructure layer 330, an AI layer 340, and a product engineering layer 350.

[0053] As illustrated in FIG. 3B, the workflow 300 may include a data interface 310 configured for solution engineering or customer system integration. In some cases, the data interface 310 is configured by a user, e.g., a process engineer. In some cases, the data interface 310 is configured to connect a customer data source (e.g., the legacy manufacturing system 110) to the system 120. The data interface 310 may include a database, e.g., a data warehouse, configured to receive, fetch, or store data associated with equipment or units in a manufacturing process (e.g., the legacy manufacturing system 110). The data interface 310 may receive or fetch data via an API configured to receive or fetch data.

[0054] In some embodiments, the data comprises multiple data types and data sets comprising: (1) historical process data, (2) current process data, (3) historical measurement data of one or more metrics, (4) current measurement data of one or more metrics, (5) operational data, or (6) equipment specification data. The data may include raw data, e.g., trace data from an FDC system. The data may include data of different structures, e.g., structured data, unstructured data, semi-structured data, or combinations thereof. The data may include data of different data types, e.g., integer data, floating point data, string data, Boolean data, date data, time data, object data, variant data, or combinations thereof. The data may include response data from measurement equipment, metadata or operational data from customer databases, alarm data, machine logs, system logs, and the like. The data may include data associated with system-or machine-generated logs. In some embodiments, the data comprises batch data, equipment data, process data, metadata, and / or measurement data.

[0055] The data may include data associated with response data, measurement data, FDC data, response limit data, operation data, and the like. Response data may include measurement identification (ID), wafer ID, wafer origin, rework status, wafer lot ID, equipment ID, chamber ID, module ID, response, start date or time, end date or time, creation date or time, update date or time, sample, and the like. Measurement data may include lot code, measurement operation ID, alias, process ID, process selector, process area, product ID, description, response parameters, and the like. Fault detection and classification data may include operation ID, wafer ID, process step ID, process step name, origin, recipe ID, lot ID, equipment ID, chamber ID, module ID, FDC summary, start date or time, end date or time, measurement status, sample number, rework status, operation start date or time, operation end date or time, creation date or time, update date or time, and the like. Response limit data may include target limits, control limits (e.g., upper, lower), specification limits (e.g., upper, lower), and the like. Operational data may include alarm data, lot code, operation ID, alias, process ID, process selector, process area, product ID, description, FDC name, step, recipe ID, and the like.

[0056] The data may include data associated with generating features or models. In some embodiments, the dataset generator is configured to automate the selection of one or more features in one or more customized datasets. The data may be associated with dataset specifications, feature descriptions, feature specifications, models, model predictions, model performance, predicted performance by modules, and the like. Data associated with dataset specifications may include measurement ID, operation ID, dataset name, dataset description, selected response name, process step, selected recipe ID, owner, creation date or time, wrangling configuration, and the like. Data associated with feature descriptions may include FDC name, score, step ID, and the like. Data associated with feature specifications may include dataset specification ID, features, collinear features, creation date or time, update date or time, and the like. Data associated with models may include measurement ID, dataset specification ID, name, process ID, lot code, process area, hyperparameters, selected features, creator, link to blob, status, creation date or time, update date or time, and the like. Data associated with model predictions may include model ID, wafer ID, lot code, measurement run ID, lot ID, equipment ID, chamber ID, module ID, prediction, creation date or time, run end date or time, and the like. Data associated with model performance may include deployment dates, model ID, base model end date or time, target measurement, upper or lower measurement limits (UML, LML), upper or lower control limits (UCL, LCL), statistical six sigma, root mean square error (RMSE), R-squared (R 2 ), topological data analysis (TDA), days of collection, data count, TDA data count, TDA sign count, response summary, squared response summary, error summary, creation date or time, and the like. Data associated with model performance may include predicted performance by module, days deployed, model ID, module ID, base model end date or time, target measurements, upper or lower measurement limits, statistical six sigma, root mean square error (RMSE), R-squared (R2 ), TDA, days of collection, data count, TDA data count, TDA code count, response summary, squared response summary, error summary, creation date or time, and the like.

[0057] The data interface 310 may include a data pipeline configured to modify or otherwise transform the data before storing it in the database. Modifying or transforming the data may include modifying or transforming the data into other data structures, data types, or combinations thereof. Alternatively, or in addition, the data pipeline may be configured to modify or otherwise transform the data into other data structures, data types, or combinations thereof before transmitting to the product layer 320.

[0058] As further illustrated in FIG. 3B, the data interface 310 may include a user interface, such as a graphical user interface (GUI) (not shown for clarity). The graphical user interface may allow a user to select a manufacturing process associated with, for example, a fab number, an area name, a product identification (ID), a process ID, an operation ID, a measurement operation ID, and the like. The user may define a data range for model development and select other data processing operations, such as data wrangling for development of an ML model. In some cases, the graphical user interface may be configured for selecting relevant features, building a data set, identifying and combining recipes, selecting a time window, selecting an equipment type, selecting a chamber type, and the like. The workflow 300 may perform feature extraction, feature selection, feature engineering, model development, or a combination thereof. In some cases, the user may select features to be used for model development based, in part, on the user's domain knowledge.

[0059] As illustrated in FIG. 3C, the workflow 300 may include a product data layer 320 configured to receive or fetch data from the data interface 310. The product data layer 320 may receive or fetch data via an API configured to receive or fetch data. The product data layer 320 may validate or otherwise transform the data received from the data interface 310. In some cases, the product data layer 320 may be configured to use or process metadata associated with the data to analyze ML model performance, monitoring and observability capabilities, product usage and performance measurements and statistics, model and experiment tracking, and the like. In some cases, the product data layer 320 may automatically validate the data based on metadata, operational data, statistical methods, ML methods, or any combination thereof. In some cases, the product data layer 320 may automatically validate the data based on rules (e.g., business rules) determined by a user, e.g., a process engineer, a solution engineer, a domain expert, an end user, and the like. Modifying or transforming the data may include modifying or transforming the data into other data structures, data types, or combinations thereof. The product data layer 320 may include a database, e.g., a data warehouse, configured to receive, fetch, or store data after the data has been validated or otherwise transformed. The product data layer 320 may include a data pipeline configured to modify or otherwise transform the data before transmitting to an artificial intelligence (AI) layer 340 (hereinafter, "AI layer 340"). Modifying or transforming the data may include modifying or transforming the data into other data structures, data types, or combinations thereof. The product data layer 320 may transmit or push data to the AI ​​layer 340 via an API configured to transmit or push data. The product data layer 320 may transmit or push data in a batch mode or a real-time mode. In some cases, the AI ​​layer 340 may be configured to pull data from the product layer 320.

[0060] As illustrated in FIG. 3E, the workflow 300 may include an AI layer 340 configured to develop or deploy an ML model or an inference model. The systems described herein may generate predictions or recommendations regarding process variables, target properties, root causes of anomalies, or combinations thereof. The predictions or recommendations may be used to optimize process control or monitoring of a manufacturing process in a manufacturing environment. In some cases, feature weights associated with training, testing, or validating an ML model may provide insights ("inferences") for a user. The insights may enable a user to determine the features that the ML model used to generate the predictions or recommendations. The AI ​​layer 340 may receive or fetch data from the product data layer 320 via an API configured to receive or fetch data. The AI ​​layer 340 may transmit the predictions or recommendations to the product engineering layer 350 via an API configured to transmit the predictions or recommendations.

[0061] As further illustrated in FIG. 3E, the AI ​​layer 340 may include modules configured to develop or otherwise deploy ML or inference models. In some cases, ML models are used interchangeably with inference models. In some cases, inference models derive insights from ML models, such as insights about important features or weights used to train the ML models. The modules may include a module 341 configured for data quality, a module 342 configured for model development, a module 343 configured for deploying models, a module 344 configured for analyzing model performance, a module 345 configured for model observability, a module 346 configured for continuous integration or delivery of models, a module 347 configured for scaling models, or a module 348 configured for a user to interface with the models.

[0062] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 341 configured for data quality, e.g., data scrubbing. Module 341 may be configured for validating the data, transforming the data, performing statistical methods on the data, or combinations thereof. The operations may include operations associated with row compression, one-hot encoding, binning, normalization, standardization, data enrichment, or combinations thereof. The data may be transmitted to module 342. The data may be received or fetched by module 342.

[0063] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 342 configured for model development. Data may be received or fetched from module 341. The operations may include operations associated with feature extraction, feature selection, feature engineering, model development, or combinations thereof. Statistical or ML methods may determine features for inclusion in model development. Statistical methods may be associated with detrending, linear regression, logistic regression, k-nearest neighbors, k-means clustering, support vector machines, artificial neural networks, decision trees, or combinations thereof.

[0064] For example, statistical or ML methods may process, for example, FDC trace data to extract features. In some cases, statistical methods may include summary averaging of fault detection and classification (FDC) trace data. For example, FDC trace data may be received from 200 sensors associated with equipment in a manufacturing process. The trace data may include time series data of sensor measurements taken every second over a 20 minute period. The statistical methods may include methods for determining statistical properties of the data. For example, the statistical methods may include averaging the trace data to obtain a 200 dimensional feature vector.

[0065] For example, statistical or ML methods may process, for example, FDC trace data to select features. In some cases, a sliding window correlation coefficient (SWCC) method may be used to select the most relevant features. Operations associated with the sliding window correlation coefficient (SWCC) method may include calculating correlation coefficients and averaging the correlation coefficients using a Fisher Z transform. The feature with the largest sliding window correlation coefficient (SWCC) value may be selected. In some cases, multicollinearity between features may be used for feature selection.

[0066] For example, statistical or ML methods may process, for example, FDC trace data to detrend or normalize features. The method may calculate the innovations of X and Y, where X may include features from FDC and Y may include response measurements. To generate the innovations of X and Y, the detrending method may subtract a weighted average value of past data, thereby determining the time difference with past data. The time difference of Y may better explain the relationship with X. Compared to typical methods such as deep learning or other statistical methods (e.g., random forest or partial least squares), detrending may provide better predictions or recommendations. In some cases, X and Y may be normalized after detrending.

[0067] For example, a statistical or ML method may process, for example, FDC trace data to generate a model using online learning. As illustrated in FIG. 5, the online learning method may include a method associated with follow the leading history (FLH). For example, the method may generate a model for each time point of a plurality of N data points, e.g., (x i ,y i A linear regressor f(x) having N linear regressors may be obtained from the N data points. j But, w 1 f 1 (x)+w 2 f 2(x)+w N f N For each linear regressor f(x), we can assign a linear regressor f(x) to generate an ensemble regressor f(x) as i ,y i ) for each y i can be predicted from N regressors.

[0068] In some cases, the regressors may include ridge regressors, linear regressors, or nonlinear regressors. Using hyperparameter optimization (HPO), the weights w may be updated using rules based on at least the regression error of each regressor. The hyperparameter optimization (HPO) may include a variance method for stratified modeling using Laplacian regularization. Determining the regressor error may include, for example, a method for determining the root mean square error (RMSE) or other typical descriptors or metrics for determining the performance of the ML model. In some cases, the metric may include a custom metric, for example, a metric defined by a user. Determining the regressor error may include a method for softmax, for example, normalizing the data to a probability distribution probability proportional to an exponential function of the input. In some cases, the regressors may be pruned based on a rule associated with the performance history, for example, the error of the regressor. In some cases, the regressors may be pruned based on a rule associated with the regressor's lifetime. Updating or pruning the regressors may be associated with data drift or shift in equipment or units associated with the manufacturing process. The machine learning model may be transmitted to a module 343 configured to deploy the model. The module 343 may receive or fetch the ML model.

[0069] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 343 configured to deploy a model. The module 343 may receive or fetch a model from the module 342. The module 343 may be configured to retrain a model, deploy a model, or serve a model. Retraining a model may include scheduling a model for evaluation or optimization, evaluating a model, optimizing a model, and the like. Deploying a model may include deploying a model in a stealth mode (e.g., a shadow model) or deploying a model in a staged mode (e.g., a canary model). Stealth mode may include providing data to the newly deployed model but not providing predictions or recommendations to users or other systems. The newly deployed model may be analyzed for performance, e.g., accuracy of predictions or recommendations, while the older model may continue to provide predictions or recommendations. Staged mode may include providing data to the newly deployed model and providing predictions or recommendations to some users or some other systems, e.g., to detect early issues or errors with the newly deployed model. Model serving may include packaging the model for deployment and use by users associated with the manufacturing environment. Module 343 may transmit or expose the predictions, recommendations, performance metrics, and the like to product engineering layer 350 via an API configured to transmit the predictions, recommendations, performance metrics, and the like.

[0070] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 344 configured to analyze model performance. The module 344 may be configured to perform integration testing, backtesting, sensitivity analysis, stress testing, and the like. The model performance may be used, in part, to determine further action by modules 341, 342, 343, or combinations thereof, to obtain model performance that meets a predetermined threshold. The predetermined threshold may be associated with an accuracy or confidence level of the prediction, recommendation, performance metric, and the like. In some embodiments, one or more performance metrics are derived from at least the sensitivity analysis or stress testing.

[0071] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 345 configured to determine observability of the model. The module 345 may be configured to monitor, troubleshoot, or comment on the model as it moves from development to deployment. The actions may include analyzing the model's performance, analyzing drift or shifts in the model, analyzing the model's data quality, commenting on the model's results, and the like. Insights ("inferences") from the module 345 may determine further actions by modules 341, 342, 343, or combinations thereof, to obtain model performance that meets a predetermined threshold. The predetermined threshold may be associated with an accuracy or confidence level of predictions, recommendations, performance metrics, and the like.

[0072] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 346 configured for continuous integration or delivery of models. A pipeline for continuous deployment or integration of models may be executed in response to events. Events may include on demand (e.g., ad-hoc manual execution of the pipeline), on a schedule (e.g., newly labeled data is available on a daily, weekly, or monthly schedule, etc.), in response to the availability of new training data, in response to degradation of model performance, in response to significant changes in the distribution of data, and the like.

[0073] Continuous integration may include operations for generating packages, generating container images, generating executables, generating tests, and the like. Testing may include testing feature engineering methods, testing methods implemented in models, testing for model training convergence, testing for model training outputs (e.g., not generating outputs that are not numbers (NaN)), testing that the model pipeline produces expected artifacts, testing integration between pipeline components, and the like.

[0074] Continuous delivery may include operations such as delivering the pipeline implementation to a target environment to generate predictions or recommendations. Continuous delivery may include operations such as verifying compatibility of the model with equipment or units associated with the manufacturing process, testing predictions or recommendations by comparing inputs to expected outputs, testing prediction or recommendation performance, validating data to retrain or batch predictions or recommendations, verifying that the model meets prediction or recommendation performance targets before deployment, automating deployment to a test environment, automated deployment to a pre-production environment, semi-automated deployment to a pre-production environment, manual deployment to a production environment after several successful runs of the pipeline on a pre-production environment, etc. The module 346 for continuous integration or delivery of models may perform the integration or delivery using a feature store, dataset versioning, experiment management, model registration, and the like. The feature store may include a central location for standardized definitions for features, storage of features, or access to features for training, testing, validation, serving, delivering, or deploying models.

[0075] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 347 configured to scale a model. The module 347 may be configured to scale a model by training, testing, or validating the model with data received from more than one piece of equipment, more than one unit, or more than one process. For example, the data may be associated with a chemical vapor deposition (CVD) process. The chemical vapor deposition (CVD) process may include one or more sub-processes, one or more pieces of equipment, or one or more chambers configured to perform the chemical vapor deposition (CVD) process. The data received from the sub-processes, pieces of equipment, or chambers may be used to generate a model for optimizing process control or monitoring (e.g., predicting or recommending) across all the sub-processes, pieces of equipment, or chambers.

[0076] As further illustrated in FIG. 3E, the AI ​​layer 340 may include a module 348 configured for a user to interface with the model during model development, continuous integration, continuous delivery, or scaling. The module 348 may be configured to generate logs or metrics associated with continuous integration, continuous delivery, or scaling. The module 348 may be configured to monitor the logs or metrics during continuous integration, continuous delivery, or scaling. The module 348 may be configured to generate a user dashboard or alerts from the logs. The alerts may be associated with continuous integration, continuous delivery, or scaling. The dashboards may be configured to display or transmit the logs or metrics to a user, enable a user to read the logs or metrics, enable a user to process the logs or metrics, enable a user to manipulate the logs or metrics, and the like. In some cases, the module 348 may be configured to generate ML logs associated with model development.

[0077] As illustrated in FIG. 3F, the workflow 300 may include a product engineering layer 350 configured to provide predictions, recommendations, performance metrics, and the like to equipment, units (e.g., customer systems), or users associated with a manufacturing process in a manufacturing environment (e.g., legacy manufacturing system 110). The product engineering layer 350 may be configured to receive or fetch the predictions, recommendations, performance metrics, logs, and the like from the AI ​​layer 340 via an API, e.g., a product backend, a product frontend, or any means for receiving, storing, transmitting, or fetching data. In some cases, the predictions, recommendations, performance metrics, logs, and the like may be stored in the product backend and transmitted to the equipment, units (e.g., customer systems), or users via the product backend. The provision of the predictions, recommendations, performance metrics, and the like may be configured in a feedback loop with the equipment or units to optimize process control or monitoring of the manufacturing process in the manufacturing environment.

[0078] As illustrated in FIG. 3D , the workflow 300 may include an infrastructure module or layer 330 configured to facilitate predictions or recommendations associated with optimizing process control or monitoring in a manufacturing environment. The module 330 may comprise a relational database, a multi-cloud database, a runtime database, a runtime container, a deployment, scaling or management service, a batch data processing service, a streaming data processing service or a machine learning (ML) service, an ML workflow service, a distributed scaling and modeling service, a distributed log storage database service, and the like. The module 330 may be associated with an API configured to implement the services or methods of the module 330. The module 330 may be configured to use a method similar to the module 430 of FIG. 4 . A non-limiting example of a relational database includes PostgreSQL. A non-limiting example of a multi-cloud database includes MongoDB. A non-limiting example of a runtime container includes Docker. A non-limiting example of a deployment, scaling or management service includes Kubernetes. A non-limiting example of a batch data processing service, a streaming data processing service or an ML service includes Apache Spark. A non-limiting example of an ML workflow service includes Apache Airflow. A non-limiting example of a distributed scaling and modeling service includes Ray. A non-limiting example of a distributed log storage database service includes Elastic Search Kibana. A non-limiting example of a dashboard includes Grafana or Kibana. A non-limiting example of a dashboard alert includes Grafana or PagerDuty.

[0079] (AI Workflow for Virtual Measurement (VM)) Illustrated in FIG. 4 is an AI workflow 400 (hereinafter “workflow 400”) for implementing the system 120, architecture 500, workflow 600, or workflow 300 of the present disclosure for optimizing process control or monitoring of a manufacturing process in a manufacturing environment. The workflow 400 may include modules configured to perform a method or workflow described herein. The workflow 400 may include a client application module or layer 410 configured to store or transmit data (e.g., customer data) for developing models that generate predictions or recommendations for process control or monitoring. The workflow 400 may include a front-end user management module or layer 420 configured for users to interface with the workflow 400. The workflow 400 may include an infrastructure module or layer 430 configured to manage users or models to optimize process control or monitoring. The workflow 400 may include a data and machine learning module or layer 440 configured to generate or publish predictions or recommendations for optimizing process control or monitoring.

[0080] As illustrated in FIG. 4, the workflow 400 may include a client application module or layer 410 configured to store or transmit data for developing a model that generates predictions or recommendations for optimizing process control or monitoring in a manufacturing environment. The data may include data associated with processing equipment, metrology tools, inspection equipment, automated material handling systems, and the like. The data may be stored in a database, such as a data lake, data mart, data hub, and the like. In some cases, a legacy system may transmit data for developing a model. The legacy system may include an MES system, an APC system, an SPC system, an RMS system, and the like. The data may be transmitted to a data and machine learning module or layer 440 via an API configured to transmit or receive the data. In some cases, the data may be received or fetched by a data collector module 441 in the data and machine learning module or layer 440.

[0081] As illustrated in FIG. 4, the workflow 400 may include a data and machine module or layer 440 configured to generate or publish predictions or recommendations for optimizing process control or monitoring in a manufacturing environment. The module 440 may include a module for generating or publishing predictions or recommendations. The modules may include a data collector module 441, a model management module 442, a hyperparameter optimization module 443, a model performance evaluation module 444, a prediction or recommendation module 445, a publishing module 446, a data module 447, an FDC tracing module 448, or a task module 449. The module 440 may be configured in a manner similar to the VM engine 520 and the AI ​​layer 530 of FIG. 1B.

[0082] As further illustrated in FIG. 4, the data and machine learning module or layer 440 may include a data collector module 441. The module 441 may comprise a data collector scheduler and a data collector worker. The data collector scheduler may be configured to receive or fetch data from the client application module or layer 410. The data may be received or fetched via an API configured to receive or fetch the data. In some cases, the data is received in a batch mode. In some cases, the data is received substantially in real time. The data collector worker may be configured to modify or otherwise transform the data in a manner similar to the module 341 of the artificial intelligence (AI) layer 340 of FIG. 3A or the application layer 510 of FIG. 1B.

[0083] In some embodiments, the data collector is configured to receive data synchronously from at least two different sources. In some embodiments, the data collector is configured to receive data asynchronously from at least two different sources. In some embodiments, the data collector is configured to receive data from at least one source based, at least in part, on a predetermined schedule. In some embodiments, the data collector is configured to receive data from at least one source based, at least in part, on detection of one or more events occurring within the manufacturing environment. In some embodiments, the one or more events are associated with a drift, excursion, shift, deviation, or anomaly in at least one process or process equipment within the manufacturing environment.

[0084] As further illustrated in FIG. 4, the data and machine learning module or layer 440 may include a model management module 442. Module 442 may comprise a module configured to develop a base ML model, develop a custom ML model, or update an ML model. Module 442 may be configured to use a methodology similar to module 342 of artificial intelligence (AI) layer 340 of FIG. 3A. In some embodiments, the system further comprises a model performance evaluation module configured to generate one or more performance metrics for the one or more machine learning models. Module 442 may be configured in a manner similar to VM engine 520 and AI layer 530 of FIG. 1B.

[0085] As further illustrated in FIG. 4, the data and machine learning module or layer 440 may include a hyperparameter optimization (HPO) module 443. In some embodiments, the model management module is configured to automatically update one or more machine learning models using hyperparameter optimization. The module 443 may comprise a hyperparameter optimizer scheduler or hyperparameter optimizer worker. The hyperparameter optimizer scheduler may be configured to schedule the optimization of model hyperparameters. In some embodiments, the hyperparameter optimization is performed on a predefined schedule. The hyperparameter optimizer worker may be configured to optimize the model hyperparameters. In some embodiments, the hyperparameter optimization is performed based, at least in part, on the detection of one or more events occurring within the manufacturing environment. The optimization may occur when the model is trained, tested, or validated to meet a threshold level or performance criteria for generating a prediction or recommendation. For example, the performance criteria may be associated with a root mean square error (RMSE) or softmax, as described elsewhere herein. Module 443 may be configured to use modules, methods, or workflows similar to 341 or 342 of artificial intelligence (AI) layer 340 of FIG. 3A or VM engine 520 and AI layer 530 of FIG. 1B.

[0086] As further illustrated in FIG. 4, the data and machine learning module or layer 440 may include a model performance evaluation module 444. The module 444 may comprise a daily trend scheduler or a daily trend analyzer. The daily trend scheduler may be configured to schedule the analysis of trends associated with the model's performance. The trends may be scheduled over a time interval or in response to an event. For example, the time interval or event may include on demand (e.g., manually scheduled), on a schedule (e.g., on a schedule such as hourly, daily, weekly, or monthly), in response to the availability of new training data, in response to degradation of model performance, in response to a significant change in the distribution of data, and the like. The daily trend analyzer may be configured to analyze trends associated with the model's performance. The module 444 may be configured to use a module, method, or workflow similar to 344 or 346 of the artificial intelligence (AI) layer 340 of FIG. 3A or the AI ​​layer 530 of FIG. 1B.

[0087] As further illustrated in FIG. 4, data and machine learning module or layer 440 may include a prediction or recommendation module 445. Module 445 may be configured to use methods similar to module 343 of artificial intelligence (AI) layer 340 of FIG. 3A. In some embodiments, one or more predictions are usable to enable substantially real-time characterization, monitoring, control, and / or modification to or optimize one or more processes and / or equipment in a manufacturing environment. Module 445 may be configured to use modules, methods, or workflows similar to AI layer 530 of FIG. 1B.

[0088] As further illustrated in Figure 4, the data and machine learning module or layer 440 may include a publishing module 446. Module 446 may be configured to use a module, method, or workflow similar to 343 of artificial intelligence (AI) layer 340 of Figure 3A or performance management 550 of Figure 1B. In some embodiments, the system further comprises a publishing module configured to publish one or more predictions for integration into a process control workflow associated with the manufacturing environment.

[0089] As further illustrated in FIG. 4, the data and machine learning module or layer 440 may include a data module 447. The module 447 may comprise an operation configured to obtain response data or obtain target data. The response data or target data may be received or fetched from a data hub of the module 410. The data may be received or fetched from the data hub via an API configured to receive or fetch the data. In some cases, the data may be received or fetched directly from the data hub via a specifically provisioned data store configured to transmit the data. In some cases, the data may be received or fetched directly from a processing device, a processing sensor, or a measurement device. In some cases, the data may be received or fetched directly from an SPC device, an APC device, or an FDC device. The module 447 may be configured to use a module, method, or workflow similar to 310 or 320 of FIG. 3A or the application layer 510 of FIG. 1B.

[0090] As further illustrated in FIG. 4, the data and machine learning module or layer 440 may include an FDC trace module 448. The module 448 may comprise an operation configured to obtain FDC trace data. The fault detection and classification trace data may be received or fetched from the legacy system of the module 410. The fault detection and classification trace data may be received or fetched from the module 447. The module 448 may transmit the data to the module 445 to generate a prediction or recommendation. The data may be transmitted, received, or fetched via an API configured to transmit the data, receive the data, or fetch the data. The module 448 may be configured to use a module, method, or workflow similar to 310 or 320 of FIG. 3A or the application layer 510 of FIG. 1B.

[0091] 4, data and machine learning module or layer 440 may include a task module 449. Module 449 may comprise operations configured to schedule or monitor tasks performed by module 430. Module 449 may be configured to use modules, methods, and workflows similar to 330 of FIG. 3A or VM engine 520 and AI layer 530 of FIG. 1B.

[0092] As further illustrated in FIG. 4, the workflow 400 may include a front-end management module 420 configured for users to interface with the workflow 400, e.g., core services of the workflow 400, to develop models that generate predictions or recommendations for optimizing process control or monitoring in a manufacturing environment. The core services may include services associated with VMs. The users may include executives, customers, process engineers, technicians, suppliers, and the like associated with manufacturing processes in a manufacturing environment. The modules 420 may comprise front-end application services, back-end application services, user management services, alert services, log monitoring services, or user management database services. The services described herein may be configured using APIs configured to implement the services. The front-end management module 420 may be configured to use modules, methods, or workflows similar to 350 of FIG. 3A or application layer 510 or performance management 550 of FIG. 1B.

[0093] The front-end application services of module 420 may include user applications configured to allow a user to interact with workflow 400. For example, a user may connect additional processes, connect additional equipment or units, collect data from processes, equipment, or units, generate data sets from the data, generate base ML models, generate custom ML models, view performance of ML models, and the like. Module 420 may be configured with APIs configured to implement the services. A non-limiting example of an API that may be configured to implement the front-end application services includes React. The front-end application services may be configured to use modules, methods, or workflows similar to 350 of FIG. 3A or application layer 510 or performance management 550 of FIG. 1B.

[0094] The back-end application services of module 420 may be configured to access core services (e.g., gateways) of workflow 400. The back-end application services may be configured using APIs configured to implement the services. The back-end application services may be configured to use modules, methods, and workflows similar to 350 of FIG. 3A or application layer 510 or performance management 550 of FIG. 1B.

[0095] The user management services of module 420 may be configured to allow users to access the core services through back-end application services. The user management services may be configured using APIs configured to implement the services. The user management services may be configured to use modules, methods, or workflows similar to 350 of FIG. 3A or application layer 510 or performance management 550 of FIG. 1B.

[0096] The alert service of module 420 may be configured to provide a user alert regarding an alert associated with workflow 400. The alert service may be configured using an API configured to implement the service. A non-limiting example of an API that may be configured to implement the alert service includes Elastic Kibana. The alert service may be configured to use a module, method, or workflow similar to module 350 of FIG. 3A.

[0097] The log monitoring service of module 420 may be configured for user visualization of logs associated with the workflow 400. The logs may be provided to a user via a dashboard or a graphical user interface (GUI). The logs may comprise statistics associated with the workflow 400. The statistics may include statistics related to model performance. The log monitoring service may be configured using an API configured to implement the service. The log monitoring service may be configured to use a module, method, or workflow similar to 350 of FIG. 3A or performance management 550 of FIG. 1B.

[0098] The user management database service of module 420 may be configured to receive and store user interactions with the workflow 400 in a database. The user interactions may be invoked or received by a user for subsequent user interactions with the workflow 400. The user management database service may be configured using an API configured to implement the service. A non-limiting example of a database that may be configured to implement the user management database service includes an Oracle database. The user management database service may be configured to use a module, method, or workflow similar to 350 of FIG. 3A or data and infrastructure layer 540 of FIG. 1B.

[0099] As illustrated in FIG. 4, the workflow 400 may include an infrastructure module 430 configured to manage users or models to develop models that generate predictions or recommendations for optimizing process control or monitoring in a manufacturing environment. The module 430 may comprise modules for a distributed log storage database, a multi-cloud database, an object storage device, a deployment service, and the like. The distributed log storage database may be configured to store or retrieve logs. In some embodiments, the multiple data stores comprise an object storage device configured to store one or more model data files associated with one or more machine learning models. A non-limiting example of a distributed log storage database includes Elastic Kibana. The multi-cloud database may be configured to store data associated with VMs. A non-limiting example of a multi-cloud database includes MongoDB. The object storage device may be configured to store files associated with the models. The deployment service may be configured to schedule tasks or monitor tasks associated with deploying the models. Manage job queues for a worker app service. A non-limiting example of a deployment service includes RedisAI. The infrastructure module 430 may be configured to use modules, methods, or workflows similar to 330 of FIG. 3A or the data and infrastructure layer 540 of FIG. 1B.

[0100] In some embodiments, one or more predictions are generated based on virtual metrology (VM). The manufacturing environment may be associated with equipment configured to perform a process. The process may be associated with inputs and outputs configured in a feedback control loop. The feedback control loop may generate different inputs to improve the output. The inputs and outputs may include data associated with process variables or target properties. The process variables or target properties may be detected or measured using sensors or instruments. For example, process variables associated with a semiconductor manufacturing process, e.g., chemical vapor deposition, may include temperature, pressure, power, current, gas volume, and the like. Target variables associated with a semiconductor manufacturing process, e.g., chemical vapor deposition, may include wafer film thickness, wafer refractive index, critical dimensions, and the like. The inputs and outputs may include data associated with current (e.g., current or real-time) data or historical data. The current data or historical data may be associated with sensor data, sensor specification data, process data, process specification data, measurement data, operational data, equipment data, equipment specification data, or FDC data.

[0101] A system described herein (e.g., a VM system or an IM system) may use inputs and outputs associated with a manufacturing environment to generate predictions or recommendations for optimizing process control or monitoring of a manufacturing process in the manufacturing environment. The process may be associated with an SPC process or an APC process. The advanced process control may comprise a method for monitoring, controlling, improving, or troubleshooting processes.

[0102] (Method for aggregating data and models for virtual metrology) The present disclosure also provides improved systems and methods that can utilize collected sensor data to optimize process control and monitoring in high volume manufacturing. The collected sensor data can be provided from post-process measurements such as thickness, mass, refractive index, and critical dimensions. As discussed above, virtual metrology (VM) can be a technique that predicts post-process measurements of a wafer without physical metrology. In some cases, VM can provide 100% wafer measurements that can be utilized for advanced process control (APC) or statistical process control (SPC) to improve yield in semiconductor manufacturing. Physical metrology cannot achieve such results because it can incur overhead costs of operating metrology equipment that limits processing power. Despite the advantages of VM implementation, VM may not be widely deployed in actual manufacturing processes due to lack of accuracy and scalability caused by inherent characteristics of semiconductor manufacturing data, such as extreme data drift, data shift, and data scarcity in semiconductor manufacturing data.

[0103] In semiconductor manufacturing, many different factors can cause data drift and data shift. For example, changes in equipment parameters, including process recipe variables such as temperature, pressure, and process duration, as well as periodic maintenance, can lead to data drift and shift. Data shift typically occurs due to preventive maintenance or recipe adjustments, while data drift results from gradual changes in equipment status during continuous operation. Equipment status may change drastically after a preventive maintenance event, which may lead to data shift. Traditional regression methods such as gradient boosting and neural networks often underperform in VM applications for semiconductor manufacturing. These methods typically assume a static environment where the relationship between process parameters and target measurements remains constant over time. However, the semiconductor manufacturing environment is dynamic, with frequent changes in data characteristics between model training and deployment. To address the technical challenges of data drift and data shift, adaptive online models that can capture changes in the relationship between inputs and outputs have been previously disclosed. In some cases, the inputs may include equipment sensor data and process recipe parameters, and the outputs may include post-process measurements such as thickness and mass. See, for example, Zabrocki et al., "Adaptive Online Time-Series Prediction for Virtual Metrology in Semiconductor Manufacturing," 2023 34th Annual SEMI Advanced Semiconductor Manufacturing Conference (ASMC). IEEE, 2023 (incorporated herein by reference in its entirety). Using this method, the adaptive online model achieved a level of VM prediction accuracy that is applicable in real-world high-volume manufacturing facilities. Manufacturing facilities, such as SK Hynix, have deployed adaptive online models on 1,005 data streams of key outcome variables for chemical vapor deposition (CVD), physical vapor deposition (PVD), and ETCH processes, achieving statistically significant improvements in process control and monitoring.Specifically, using the resulting virtual measurements for APC resulted in an average reduction in process variability of at least about 30%, thereby improving yield. Such technical improvements demonstrate that addressing the challenges unique to semiconductor manufacturing can be important for highly accurate VM models. This demonstration is further described below in the Examples section.

[0104] The present disclosure further improves the predictive performance of adaptive online models by focusing on another technical challenge in semiconductor manufacturing: making VM predictions with scarce or sparse data. Data scarcity or sparsity presents a significant technical challenge in VM, often resulting from low measurement sampling rates and heterogeneous data sets across various manufacturing processes and recipes. In this context, heterogeneity refers to a situation where a true relationship between input variables and output measurements established for one environment (e.g., from a specific equipment with a particular recipe) may not hold for other cases. In some cases, chip manufacturers may keep sampling rates low, for example, due to the overhead of physical metrology measurements and the vastly different characteristics of data sets across processes, recipes, or equipment. Even when equipment types and recipes are the same or similar, the status of individual chambers may vary significantly. Factors that affect chamber status include operating time, maintenance history, component wear, process residue accumulation, and environmental conditions such as temperature and humidity increase and decrease. Thus, simple data aggregation may not be effective in VMs due to such heterogeneity (or diversity), which may exacerbate the technical challenges of rare or sparse data.

[0105] To address data scarcity or sparsity, an improved VM model or algorithm, called Aggregative Adaptive Online Model (AggAOM), is provided herein that can be utilized for hierarchical structures of semiconductor manufacturing equipment. Equipment tools within the same hierarchy often exhibit a high level of commonality due to shared underlying physics and operational events, such as maintenance schedules and part replacements. The commonality can be calculated by generating correlation matrices for each tool and comparing the similarity between the matrices. These commonalities can include control system architectures, sensor types, process flows, and quality control procedures, and can contribute to the potential for knowledge transfer and unified modeling approaches across similar equipment. Such a hierarchical structure, when utilized or considered by AggAOM, can capture both the commonality and specificity of equipment tools. Such a model can successfully address the technical challenges of data scarcity or sparsity and the problems of instability resulting from data drift or data shift. By resolving these technical issues, AggAOM can significantly improve the predictive performance of VM for real-world, high-volume manufacturing datasets.

[0106] (Adaptive Online Model) Previously, Zabrocki introduced an adaptive online model (AOM) that can be adjusted to quickly accommodate significant data drifts and shifts in semiconductor data by updating the model as new measurements are observed. See Zabrocki et al. herein. The essence of the AOM can be illustrated in FIG. 7. In some cases, the AOM can be viewed as a dynamic system designed for an environment where data may continually shift and evolve as the AOM seamlessly absorbs new information. For example, the AOM can be configured to focus on incorporating the time evolution of the data, which may position the AOM as a crucial tool for real-time data analysis in VM. The AOM can achieve a relatively high level of predictive accuracy. However, using the AOM alone may inadvertently overlook a high level of chamber-related commonality regarding the fundamental physics and operational events during the manufacturing process, since it can be trained independently by the chamber.

[0107] Briefly, the AOM may include receiving sensor data and historical measurements as inputs to the AOM. The AOM may include a time-aware normalizer that normalizes the data based on its timestamp. An adaptive online learner may then learn and predict measurements based on the normalized data. The predicted measurements may be the result of a VM that may be used to supplement the sampled data.

[0108] (aggregated model) The systems and methods herein may improve both AOM and other VM approaches. For example, compared to AOM, other VM models may focus on developing a unified single model that predicts across different equipment, chambers, and stations as shown in FIG. 8A. See, for example, Chan et al., "Just-in-time modeling with variable shrinkage based on Gaussian processes for semiconductor manufacturing," IEEE Transactions on Semiconductor Manufacturing 31.3 (2018): 335-342; Chen et al., "Virtual metrology of semiconductor PVD process based on combination of tree-based ensemble model," Isa Transactions 103 (2020): 192-202; Lynn et al., "Global and local virtual metrology models for a plasma etch process," IEEE Transactions on Semiconductor Manufacturing 25.1 (2011): 94-103, each of which is incorporated herein by reference in its entirety. Compared to a unified single model, the AOM may employ individual models (M1, M2, M3, and so on) to capture the different predictive behavior of each station (S1, S2, S3, and so on) with respect to the instruments (EQ1, EQ2, EQ3, and so on) and chambers (ChA, ChB, ChC, and so on), as shown in Figure 8B. As a result, the station-related models may generate better predictions even with data drift and data shift that may appear in separate chambers and stations.

[0109] Compared with AOM and other VM approaches, the AggAOM herein may uniquely incorporate a hierarchical structure within the boosting algorithm, whereby learners or regressors are trained sequentially to correct the errors of their predecessors, ultimately creating a powerful learner / regressor. The boosting algorithm may include, for example, XGBoost, LightGBM, CatBoost, and the like. For example, consider a typical semiconductor manufacturing process in which a critical step may involve multiple pieces of equipment (e.g., equipment 1 and 2), each containing several chambers (e.g., chambers A and B), which in turn encompass two stations (e.g., stations S1 and S2). FIG. 10 illustrates an example showing the normalized thickness of wafers processed by three pieces of equipment, whereby the x-axis is the process time. Although these three time series data exhibit different behaviors, they share a discernible pattern of increase and decrease. As recognized herein, rather than building independent models in a process-related or station-related manner, as illustrated in FIG. 9, an aggregate model, e.g., AggAOM, can be developed to capture shared patterns across a multi-level hierarchy of equipment. This approach can allow for the application of collective insights from equipment, chambers, and stations. As a result, AggAOM can capture important patterns at the chamber level and improve predictive capabilities for stations of interest within the same chamber. This approach is expected to result in significant improvements in prediction accuracy. In contrast, station-related individual models cannot achieve this hierarchical information.

[0110] (Aggregative Adaptive Online Model (AggAOM)) The AggAOM herein is an ensemble model that may include several weak models (or learners). FIG. 11 illustrates the high-level architecture or flow of one potential instantiation of the AggAOM. To capture the inherent hierarchical structure in the equipment, the AggAOM may include three levels based on domain knowledge: the "equipment level", the "chamber level", and the "station level". The highest level, the "equipment level", assigns models to each equipment and captures commonalities that occur across various chambers or stations belonging to the equipment. The output from the equipment level may be used to evaluate the residuals, e.g., discrepancies or differences between predictions and actual measurements. These residuals may serve as training signals, as they may provide granularity of focus in subsequent levels, subject to removal of higher level commonalities. The second level is the "chamber level", which may generate a more granular level of modeling that may focus on individual chambers. The chamber level residuals may be utilized to train models in subsequent levels. The final level is the "station level," which may represent the most detailed layer of modeling by focusing on individual stations within the chamber. The final prediction is made by processing (e.g., adding) the predictions from all levels to generate an ensemble model.

[0111] In a semiconductor manufacturing process, each wafer is processed through a hierarchical structure, i.e., a station (e.g., S1_ChA_EQ1) in a chamber (e.g., ChA_EQ1) in an equipment (e.g., EQ1). The corresponding FDC sensor values ​​are used to iteratively refine the models at each level. First, they update the global model for EQ1. Then, the residuals from this global model update the chamber model for ChA_EQ1. Finally, the residuals from the chamber model update the station model for S1_ChA_EQ1. All wafers can be used in the same way to update the equipment, chamber, and station models. This hierarchical approach ensures that all global and local models are refined based on the structure of the equipment. In some cases, AggAOM can be applied when only the chamber and station levels may be available.

[0112] The high-level architecture or flow may provide a systematic approach to improving prediction accuracy by leveraging both global and local data patterns and by iterative refinement. In some cases, the residuals may be used as the ground truth (or target variables) of the training data, which may be determined at each level or stage using the trained model of the previous level or stage.

[0113] The data to be predicted or output using the method herein can be determined based on at least rarity. Rarity can be quantified as the number of training samples. For example, if the number of training samples is below a threshold (e.g., the number of samples per area of ​​interest, the number of samples per process time, etc.), the number of training samples can be determined by two factors, one being the sampling rate and the other being the data collection duration. Ablation studies can be performed to compare rare vs. non-rare datasets to determine and / or confirm whether a dataset is rare. For example, experiments can include selecting one dataset and then randomly lowering the target value to reduce the number of training samples.

[0114] (Machine learning methods for optimizing process control or monitoring) Many machine learning (ML) methods implemented as algorithms are suitable approaches for implementing the methods described herein, including, but not limited to, supervised learning approaches, unsupervised learning approaches, semi-supervised approaches, or any combination thereof.

[0115] Machine learning algorithms may include, but are not limited to, neural networks (e.g., artificial neural networks (ANNs), multi-layer perceptrons (MLPs), long short-term memory (LSTMs)), support vector machines, k-nearest neighbors, Gaussian mixture models, Gaussian processes, naive Bayes, decision trees, random forests, or gradient boosting trees. Linear machine learning algorithms may include, but are not limited to, linear regression with or without regularizers, logistic regression, naive Bayes classifiers, perceptrons, or support vector machines (SVMs). Other machine learning algorithms for use with the methods according to the present disclosure may include, but are not limited to, quadratic classifiers, k-nearest neighbors, boosting, decision trees, random forests, neural networks, pattern recognition, Bayesian networks, or hidden Markov models. Other machine learning algorithms, including improvements or combinations of any of these commonly used for machine learning, may also be suitable for use with the methods described herein. Any use of machine learning algorithms in a workflow may also be suitable for use with the methods described herein. The workflow may include, for example, cross-validation, nested cross-validation, feature selection, row compression, data transformation, binning, normalization, standardization, and algorithm selection.

[0116] Machine learning algorithms may generally be trained by the following methodology to build machine learning models. For example, the generated models may determine or predict process variables or target properties to optimize process control or monitoring. Input data may include, for example, process variable data, target property data, FDC data, measurement data, sensor data, features, and the like, as described elsewhere herein. Output data may include, for example, determination or prediction of process variables, target properties, root causes of anomalies, and the like, as described elsewhere herein.

[0117] 1. Collect datasets for "training" and "testing" the machine learning model. The datasets may include many features, e.g., features associated with sensor data, equipment, processes, anomalies, and the like. The training dataset is used to "train" the machine learning model. The test dataset is used to evaluate the trained machine learning model.

[0118] 2. Determine the "features" for the machine learning algorithm to use for training and testing. The accuracy of the machine learning algorithm may depend on how the features are represented. For example, feature values ​​may be transformed using one-hot encoding, binning, standardization, or normalization. Also, not all features in the dataset may be used to train and test the machine learning algorithm. Feature selection may depend, for example, on available computing resources and time or the importance of features discovered during iterative testing and training. For example, it may be discovered that features associated with sensor data or equipment specifications predict process variables, target properties, or root causes of anomalies.

[0119] 3. Select an appropriate machine learning algorithm. For example, a machine learning algorithm described elsewhere herein may be selected. The machine learning algorithm selected may depend, for example, on available computing resources and time, or whether the prediction is continuous or categorical in nature. The machine learning algorithm is used to build the machine learning model.

[0120] 4. Build a machine learning model. A machine learning algorithm is run on the collected training dataset. The parameters of the machine learning algorithm may be adjusted by optimizing performance on the training dataset or via a cross-validation dataset. After parameter adjustment and learning, the performance of the machine learning algorithm may be validated on a dataset of naive samples that is separate from the training and test datasets. The constructed machine learning model may be accompanied by feature coefficients, importance measures, or weights assigned to individual features.

[0121] Once a machine learning model has been determined ("trained") as described above, it can be used to generate predictions or recommendations regarding process variables or target properties for optimization of process control or monitoring of a manufacturing process in a manufacturing environment.

[0122] (Example) While various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, modifications, or substitutions may occur without departing from the present disclosure. It will be understood that various alternatives to the embodiments described herein may be employed.

[0123] Example 1: Statistically Significant Performance Improvement of AggAOM (benchmark) Two benchmark data sets were used and provided by SK Hynix. The data sets were collected from a chemical vapor deposition (CVD) process in a real manufacturing environment (e.g., a semiconductor fab environment) and were used to evaluate the performance of a baseline model and different models, such as a linear model, a partial least squares (PLS) model, an XGBoost (XGB) model, an AOM, and the AggAOM herein. The data sets contained thickness measurements along with fault detection and classification (FDC) summary statistics collected from various sensors attached to the equipment, chambers, or stations. Although thickness measurements are provided as an example, the methods herein may use data sets comprising delta mass, refractive index, critical dimensions, and the like. The first benchmark data set (CVD1) comprised 10,789 measured wafers from 15 equipments containing a total of 86 stations (e.g., 43 associated chambers). The second benchmark data set (CVD2) included 12,685 measured wafers from 15 equipments containing a total of 84 stations. Both data sets were collected over a span of time, e.g., nine months. In some cases, the span of time may be at least one day, one month, one year, or longer. In some cases, the span of time may be up to one year, one month, one day, or less.

[0124] (Demonstration details) Each benchmark data set was sorted by process time (see, e.g., FIG. 10). Although sorting by process time is provided as an example, the method herein may sort using other pre-processing steps, such as scaling, imputation, outlier removal, and the like. The first 70% of the wafers were utilized as the training data set, while the remaining wafers were designated as the test data set. Hyperparameter optimization (HPO) was performed for each model. In general, HPO may not depend on different machine learning models, including the AggAOM herein, and may be configured as an optimization process to find the best set of hyperparameters for a machine learning model to maximize or optimize its performance on the validation data set. For HPO, the training data set was additionally split into the first 75% for training and the remaining part for validation. The validation data set was used to assess the performance of the model with a specific hyperparameter set. After HPO, the entire training data set, including the validation data set, was used to train a model with the best or optimal hyperparameter set and evaluated against the test data set.

[0125] (result) The aggregated adaptive online model (AggAOM) of the present specification was compared with the adaptive online model (AOM) and three baseline models, namely, a linear model, a PLS model, and an XGB model. For example, see Wold et al. "PLS-regression: a basic tool of chemometrics" Chemometrics and intelligent laboratory systems 58.2 (2001): 109-130 for the PLS model, and Chen et al. "Xgboost: A scalable tree boosting system" Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016 for the XGB model (each of which is incorporated herein by reference in its entirety). As illustrated in Figures 12A-12D, AOM and AggAOM demonstrated statistically significant improved performance over baseline models (e.g., PLS and XGB models) in terms of both r-squared (R2, Figures 12A and 12C) and root mean squared error (RMSE, Figures 12B and 12D). The performance of the PLS and XGB models was slightly better than that of the simple linear model. This suggests that addressing data drift and data shift may be useful for VM. AggAOM achieved statistically significant performance improvements for both datasets compared to AOM. Specifically, as illustrated in Figures 12B and 12D, AggAOM achieved a RMSE reduction of 30.3% in CVD1 and 35.7% in CVD2 compared to the simple linear model, improving AOM by 7.3% in CVD1 and 6.1% in CVD2. This demonstrates the effectiveness of capturing shared patterns across multi-level hierarchies of devices in improving VM prediction performance.

[0126] (Conclusion) The AggAOM herein may significantly improve the accuracy of VM prediction by solving technical problems of data drift, data shift, and data scarcity or sparsity of measurements. AggAOM may incorporate or use the hierarchical structure of multiple tools, thereby exploiting physical commonality across different equipment, chambers, or stations in the same hierarchy while adapting to their unique variations. AggAOM has been demonstrated or deployed in high-volume manufacturing facilities, such as the SK-40-ynix megafab, and has been extensively evaluated with vast amounts of data from high-volume manufacturing. Results on two CVD datasets from SK-40-ynix demonstrate the technical improvement of AggAOM over other VM models. The statistically significant improvement in performance demonstrates the effectiveness of AggAOM in a high-volume semiconductor manufacturing environment, and also demonstrates the usefulness and importance of building machine learning models that can address the specific technical challenges of VM associated with manufacturing data that other models cannot solve. By utilizing AggAOM in semiconductor manufacturing facilities such as SK hynix, and by running AggAOM with daily advanced process control (APC) run-to-run (R2R) operation for over a year, the systems and methods herein have demonstrated significantly reduced process variability in the applied processes, thereby improving yield, throughput, and cost-effectiveness.

[0127] (Computing Systems) In one aspect, disclosed herein is a computer program product for optimizing process control or monitoring of a manufacturing environment, the computer program product comprising at least one non-transitory computer readable medium having computer readable program code portions embodied therein, the computer readable program code portions comprising an executable portion configured to interface with a client application layer associated with the manufacturing environment, the executable portion comprising: (1) an executable portion configured to receive data from at least one source in the client application layer; (2) an executable portion configured to enable a user to create one or more customized data sets from the data; (3) an executable portion configured to enable a user to build, train, and / or update one or more machine learning models using, at least in part, the one or more customized data sets; and (4) an executable portion configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or equipment is being operated in the manufacturing environment.

[0128] 13, a block diagram is shown depicting an example machine including a computer system 1300 (e.g., a processing or computing system) in which a set of instructions may be executed to cause the device to implement or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components of FIG. 13 are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic components, or combinations of two or more such components, to implement a particular embodiment.

[0129] The computer system 1300 may include one or more processors 1301, memory 1303, and storage device 1308, which communicate with each other and with other components via a bus 1340. The bus 1340 may also link a display 1332, one or more input devices 1333 (which may include, for example, a keypad, keyboard, mouse, stylus, etc.), one or more output devices 1334, one or more storage devices 1335, and various tangible storage media 1336. All of these elements may interface with the bus 1340 directly or through one or more interfaces or adapters. For example, the various tangible storage media 1336 may interface with the bus 1340 through a storage media interface 1326. The computer system 1300 may have any suitable physical form, including, but not limited to, one or more integrated circuits (ICs), a printed circuit board (PCB), a mobile handheld device (such as a mobile phone or PDA), a laptop or notebook computer, a distributed computer system, a computing grid, or a server.

[0130] The computer system 1300 includes one or more processors 1301 (e.g., a central processing unit (CPU) or a general purpose graphics processing unit (GPGPU)) that perform functions. The processor 1301 optionally contains a cache memory unit 1302 for temporary local storage of instructions, data, or computer addresses. The processor 1301 is configured to aid in the execution of computer-readable instructions. The computer system 1300 may provide functionality for the components depicted in FIG. 13 as a result of the processor 1301 executing non-transitory processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 1303, storage device 1308, storage device 1335, and / or storage medium 1336. The computer-readable medium may store software implementing certain embodiments, and the processor 1301 may execute the software. Memory 1303 may read software from one or more other computer-readable media (such as mass storage devices 1335, 1336), or from one or more other sources through a suitable interface, such as network interface 1320. The software may cause processor 1301 to perform one or more processes, or one or more steps of one or more processes, described or illustrated herein. Performing such a process or step may include defining data structures stored in memory 1303 and modifying the data structures as directed by the software.

[0131] The memory 1303 may include various components (e.g., machine-readable media) including, but not limited to, random access memory components (e.g., RAM 1304) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phase change random access memory (PRAM), etc.), read-only memory components (e.g., ROM 1305), and any combination thereof. The ROM 1305 may act to communicate data and instructions unidirectionally to the processor 1301, and the RAM 1304 may act to communicate data and instructions bidirectionally with the processor 1301. The ROM 1305 and the RAM 1304 may include any suitable tangible computer-readable media, as described below. In one embodiment, a basic input / output system 1306 (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1300, such as during start-up, may be stored in the memory 1303.

[0132] A persistent storage device 1308 is bidirectionally connected to the processor 1301, optionally through a storage control unit 1307. The persistent storage device 1308 provides additional data storage capacity and may include any suitable tangible computer-readable media described herein. The storage device 1308 may be used to store an operating system 1309, executable files 1310, data 1311, applications 1312 (application programs), and the like. The storage device 1308 may also include an optical disk drive, a solid-state memory device (e.g., a flash-based system), or a combination of any of the above. The information in the storage device 1308 may be incorporated as virtual memory in the memory 1303, where appropriate.

[0133] In one example, storage device 1335 may be removably interfaced to computer system 1300 via storage interface 1325 (e.g., via an external port connector (not shown)). In particular, storage device 1335 and associated machine-readable media may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 1300. In one example, software may reside, completely or partially, within the machine-readable media on storage device 1335. In another example, software may reside, completely or partially, within processor 1301.

[0134] Bus 1340 connects a wide variety of subsystems. References herein to a bus may encompass, where appropriate, one or more digital signal lines that provide a common function. Bus 1340 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof using any of a variety of bus architectures. By way of example, and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a MicroChannel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, a HyperTransport (HTX) bus, a Serial Advanced Technology Attachment (SATA) bus, and any combination thereof.

[0135] Computer system 1300 may also include input devices 1333. In one example, a user of computer system 1300 may type commands and / or other information into computer system 1300 via input devices 1333. Examples of input devices 1333 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices (e.g., mice or touch pads), touch pads, touch screens, multi-touch screens, joysticks, styluses, game pads, audio input devices (e.g., microphones, voice response systems, etc.), optical scanners, video or still image capture devices (e.g., cameras), and any combination thereof. In some embodiments, input devices are Kinect, Leap Motion, or the like. Input devices 1333 may be interfaced to bus 1340 via any of a variety of input interfaces 1323 (e.g., input interfaces 1323), including, but not limited to, serial, parallel, game port, USB, FIREWIRE®, THUNDERBOLT®, or any combination of the above.

[0136] In certain embodiments, when computer system 1300 is connected to network 1330, computer system 1300 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 1330. Communications to and from computer system 1300 may be transmitted through network interface 1320. For example, network interface 1320 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1330, and computer system 1300 may store the incoming communications in memory 1303 for processing. Computer system 1300 may similarly store outgoing communications (such as requests or responses to other devices) communicated from network interface 1320 to network 1330, in the form of one or more packets, in memory 1303. Processor 1301 may access these communication packets stored in memory 1303 for processing.

[0137] Examples of network interface 1320 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of network 1330 or network segment 1330 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus, or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combination thereof. A network such as network 1330 may employ wired and / or wireless communication modes. In general, any network topology may be used.

[0138] Information and data may be displayed through the display 1332. Examples of the display 1332 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive matrix OLED (PMOLED) or active matrix OLED (AMOLED) display, a plasma display, and any combination thereof. The display 1332 may interface with other devices such as the processor 1301, memory 1303, and persistent storage 1308, as well as input devices 1333, via a bus 1340. The display 1332 is linked to the bus 1340 via a video interface 1322, and transport of data between the display 1332 and the bus 1340 may be controlled via a graphics control 1321. In some embodiments, the display is a video projector. In some embodiments, the display is a head mounted display (HMD), such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting examples, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VR One, Avegant Glyph, Freefly VR headsets, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0139] In addition to the display 1332, the computer system 1300 may include one or more other peripheral output devices 1334, including, but not limited to, audio speakers, printers, storage devices, and any combination thereof. Such peripheral output devices may be connected to the bus 1340 via an output interface 1324. Examples of output interface 1324 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE® port, a THUNDERBOLT® port, and any combination thereof.

[0140] Additionally or alternatively, computer system 1300 may provide functionality as a result of logic hardwired or otherwise embodied in circuitry, which may operate in place of or in conjunction with software to perform one or more processes, or one or more steps of one or more processes, described or illustrated herein. References to software in this disclosure may encompass logic, and references to logic may encompass software. Also, references to computer-readable media may encompass, where appropriate, circuitry (such as ICs) that store software for execution, circuitry that embodies logic for execution, or both. This disclosure encompasses any suitable combination of hardware, software, or both.

[0141] The various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0142] The various illustrative logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0143] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processors, or in a combination of the two. The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0144] In accordance with the description herein, suitable computing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smart phones, tablet computers, personal digital assistants, video game consoles, and vehicles. Select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the systems described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations.

[0145] In some embodiments, the computing device includes an operating system configured to execute executable instructions. An operating system is software, including, for example, programs and data, that manages the device's hardware and provides services related to the execution of applications. Suitable server operating systems include, by way of non-limiting examples, FreeBSD, OpenBSD, NetBSD, Linux, Apple Mac OS X Server, Oracle Solaris, Windows Server, and Novell NetWare. Suitable personal computer operating systems include, by way of non-limiting examples, Microsoft Windows, Apple Mac OS X, UNIX, and UNIX-like operating systems such as GNU / Linux. In some embodiments, the operating system is provided by cloud computing. Suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia Symbian OS, Apple iOS, Research In Motion BlackBerry OS, Google Android, Microsoft Windows Phone OS, Microsoft Windows Mobile OS, Linux, and Palm WebOS. Suitable media streaming device operating systems include, by way of non-limiting example, Apple TV, Roku, Boxee, Google TV, Google Chromecast, Amazon Fire, and Samsung HomeSync. Suitable video game console operating systems include, by way of non-limiting example, Sony PS3, Sony PS4, Microsoft Xbox 360, Microsoft Xbox One, Nintendo Wii, Nintendo Wii U, and Ouya.Suitable virtual reality headset systems include, by way of non-limiting example, Meta Oculus.

[0146] (Non-transitory computer-readable storage medium) In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer-readable storage media encoded with a program including instructions executable by an operating system of an optionally networked computing device. In further embodiments, the computer-readable storage medium is a tangible component of the computing device. In yet further embodiments, the computer-readable storage medium is optionally removable from the computing device. In some embodiments, the computer-readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid-state memories, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the programs and instructions are encoded on the medium permanently, substantially permanently, semi-permanently, or non-transitoryly.

[0147] (Computer Program) In some embodiments, the platform, system, medium, and method disclosed herein include at least one computer program or the use thereof. A computer program includes a sequence of instructions written to perform a specified task that is executable by one or more processors of a CPU of a computing device. The computer-readable instructions may be implemented as program modules such as functions, objects, application programming interfaces (APIs), computing data structures, and the like that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, computer programs may be written in various versions of various languages.

[0148] The functionality of the computer readable instructions may be combined or distributed as desired in various environments. In some embodiments, the computer program comprises one sequence of instructions. In some embodiments, the computer program comprises multiple sequences of instructions. In some embodiments, the computer program is provided from one location. In other embodiments, the computer program is provided from multiple locations. In various embodiments, the computer program includes one or more software modules. In various embodiments, the computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or a combination thereof.

[0149] (Terms and definitions) Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0150] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Any reference to "or" herein is intended to include "and / or" unless stated otherwise.

[0151] As used herein, the term "about" refers to an amount that is, in some cases, approximately the recited amount.

[0152] As used herein, the term "about" refers to an amount that is approximately 10%, 5%, or 1% (including increments therein) of the recited amount.

[0153] As used herein, the term "about" in connection with percentages refers to an amount that is 10%, 5%, or 1% more or less than the recited percentage (including increments therein).

[0154] As used herein, the phrases "at least one," "one or more," and "and / or" are open ended expressions that are both conjunctive and disjunctive in operation. For example, the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" respectively mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0155] Although preferred embodiments of the present disclosure have been shown and described herein, such embodiments are provided only as examples. It is not intended that the present disclosure be limited by the specific examples provided herein. The present disclosure has been described with reference to the foregoing specification, but the description and illustration of the embodiments herein are not meant to be interpreted in a limiting sense. Numerous variations, changes, and substitutions may occur without departing from the present disclosure. Furthermore, it should be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations, or relative proportions described herein, which depend on various conditions and variables. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed in practicing the present disclosure. It is therefore contemplated that the present disclosure also covers any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the present disclosure, and that systems, methods, and structures within the scope of these claims and their equivalents are covered thereby.

Claims

1. 1. A system for optimizing process control or monitoring of a manufacturing environment, the system comprising: a data and machine learning layer configured to interface with a client application layer associated with a manufacturing environment, the data and machine learning layer comprising: (1) a data collector configured to receive data from at least one source in the client application layer; (2) a dataset generator configured to enable a user to create one or more customized datasets from the data; (3) a model management module configured to enable the user to build, train, and / or update one or more machine learning models using, at least in part, the one or more customized datasets; and (4) an inference module configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or equipment is being operated in the manufacturing environment; a plurality of data stores configured to store at least the data, new data updates, the one or more customized data sets, and the one or more machine learning models; A system comprising:

2. The system of claim 1 , wherein the one or more predictions are generated based on virtual instrumentation (VM).

3. 2. The system of claim 1, wherein the one or more machine learning models comprise an Aggregative Adaptive Online Model (AggAOM).

4. 4. The system of claim 3, wherein the AggAOM is configured to take advantage of commonalities in the instruments and associated chambers or stations, the commonalities including one or more of control system architecture, sensor types, process flows, or quality control procedures.

5. The system of claim 4 , wherein the AggAOM is configured to employ a hierarchical structure of the instruments and associated chambers or stations.

6. The system of claim 5 , wherein the hierarchical structure includes at least two levels of hierarchy.

7. 7. The system of claim 6, wherein the hierarchical structure includes: (1) an equipment level that assigns a global model to each piece of equipment, the global model configured to process commonalities to evaluate a first set of residuals occurring across various chambers or stations associated with each piece of equipment; (2) a chamber level that generates a chamber model of higher resolution than the global model, the chamber model configured to process commonalities to evaluate a second set of residuals; and (3) a station level that includes a station-related model of higher resolution than the chamber model, the station-related model configured to process at least the first set of residuals and the second set of residuals to generate the one or more predictions.

8. The system of claim 1 , wherein the plurality of data stores comprises a database configured to store the data received from the at least one source in the client application tier.

9. 2. The system of claim 1, wherein the plurality of data stores comprises an object store configured to store one or more model data files associated with the one or more machine learning models.

10. 10. The system of claim 1, further comprising a publishing module configured to publish the one or more predictions for integration into a process control workflow associated with the manufacturing environment.

11. 10. The system of claim 1, wherein the one or more predictions are usable to enable or optimize characterization, monitoring, control, and / or modification to the one or more processes and / or equipment in the manufacturing environment in substantially real time.

12. 10. The system of claim 1 , further comprising a model performance evaluation module configured to generate one or more performance metrics for the one or more machine learning models.

13. The system of claim 12 , wherein the one or more performance metrics are derived from at least a sensitivity analysis or a stress test.

14. The system of claim 12 , further comprising a front-end user management module comprising a graphical user interface (GUI) configured to display the one or more performance metrics on a dashboard.

15. The system of claim 1 , wherein the data comprises equipment data, process data, metadata, and / or measurement data.

16. The system of claim 1 , wherein the data collector is configured to synchronously receive the data from at least two different sources.

17. The system of claim 1 , wherein the data collector is configured to receive the data from the at least one source based, at least in part, on a predetermined schedule.

18. The system of claim 1 , wherein the data collector is configured to receive the data from the at least one source based, at least in part, on detection of one or more events occurring within the manufacturing environment.

19. 20. The system of claim 18, wherein the one or more events are associated with a drift, excursion, shift, deviation, or anomaly in at least one process or process equipment within the manufacturing environment.

20. 2. The system of claim 1 , wherein the data comprises a plurality of data types and data sets comprising: (1) historical process data, (2) current process data, (3) historical measurement data of one or more metrics, (4) current measurement data of one or more metrics, (5) operational data, or (6) equipment specification metadata.

21. The system of claim 1 , wherein the dataset generator is configured to automate selection of one or more features in the one or more customized datasets.

22. 2. The system of claim 1, wherein the model management module is configured to train the one or more machine learning models including hyperparameter optimization.

23. 23. The system of claim 22, wherein the hyper-parameter optimization is performed based on one or more performance metrics when the one or more performance metrics degrade below a predetermined threshold level.

24. 10. The system of claim 1, wherein the manufacturing environment is associated with at least one of semiconductor manufacturing, LCD display manufacturing, solar panel manufacturing, electronics manufacturing, battery manufacturing, automotive manufacturing, or pharmaceutical manufacturing.

25. 1. A computer program product for optimizing process control or monitoring of a manufacturing environment, the computer program product comprising at least one non-transitory computer readable medium, the at least one non-transitory computer readable medium having computer readable program code portions embodied in the at least one non-transitory computer readable medium; 11. The computer program product of claim 10, wherein the computer readable program code portions comprise an executable portion configured to interface with a client application layer associated with a manufacturing environment, the executable portion comprising: (1) an executable portion configured to receive data from at least one source in the client application layer; (2) an executable portion configured to enable a user to create one or more customized data sets from the data; (3) an executable portion configured to enable the user to build, train, and / or update one or more machine learning models, at least in part, using the one or more customized data sets; and (4) an executable portion configured to use the one or more machine learning models to generate one or more predictions when one or more processes are being performed and / or equipment is being operated in the manufacturing environment.

Images