Method for determining failures in manufacturing equipment, system for determining failures in manufacturing equipment, non-temporary machine-readable storage medium

Abstract

The method includes receiving first data associated with measurements obtained by a sensor during a first manufacturing procedure in a manufacturing chamber. The method further includes receiving second data. The second data includes reference data associated with the first data. The method further includes providing the first and second data to a comparison model. The method further includes receiving a similarity score from the comparison model associated with the first and second data. The method further includes performing a corrective action in consideration of the similarity score.

Description

Technical Field

[0001] The present disclosure relates to a diagnostic method for determining a failure in manufacturing equipment, and in some embodiments, to an inter-device matching method by drill-down analysis using a comparison model for determining a failure in manufacturing equipment.

Background Art

[0002] Products can be produced by performing one or more manufacturing processes using manufacturing equipment. For example, semiconductor manufacturing equipment can be used to produce substrates through semiconductor manufacturing processes. Products having specific characteristics suitable for the target application should be produced. Understanding and controlling the characteristics within the manufacturing chamber helps in the consistent production of products.

Summary of the Invention

[0003] The following is a simplified summary of the present disclosure to provide a basic understanding of some aspects of the present disclosure. This summary is not an extensive overview of the present disclosure. It is not intended to identify key or critical elements of the present disclosure, nor to delineate any scope of particular embodiments of the present disclosure or any scope of the claims. The sole purpose of this summary is to present some concepts of the present disclosure in a simplified form as a prelude to the more detailed description that follows.

[0004] In one aspect of the present disclosure, the method includes receiving first data associated with measurements obtained by a sensor during a first manufacturing procedure of a manufacturing chamber. The method further includes receiving second data, the second data including reference data associated with the first data. The method further includes providing the first data and the second data to a comparison model. The method further includes receiving, from the comparison model, a similarity score associated with the first data and the second data. The method further includes performing corrective actions in consideration of the similarity score.

[0005] In another aspect of this disclosure, a system is disclosed comprising a memory and a processing device coupled to the memory. The processing device is configured to receive first data associated with measurements obtained by a sensor during a first manufacturing process of a manufacturing chamber. The processing device is further configured to receive second data, the second data including reference data associated with the first data. The processing device is further configured to provide the first and second data to a comparison model. The processing device is further configured to receive similarity scores associated with the first and second data from the comparison model. The processing device is further configured to perform corrective actions taking the similarity scores into consideration.

[0006] In another aspect of this disclosure, a non-temporary machine-readable storage medium is enabled. The medium stores instructions causing a processing device to perform an action when executed, the action including receiving first data associated with measurements obtained by a sensor during a first manufacturing procedure of a manufacturing chamber. The action further includes receiving second data, the second data including reference data associated with the first data. The action further includes providing the first and second data to a comparison model. The action further includes receiving a similarity score from the comparison model, the similarity score associated with the first and second data. The action further includes taking corrective action in consideration of the similarity score.

[0007] The figures in the attached drawings are provided as examples, not limited to those shown in this disclosure. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows an exemplary system architecture in several embodiments. [Figure 2] This is a cross-sectional view of a manufacturing chamber according to several embodiments. [Figure 3]This is a diagram of a dataset generator for creating datasets for training, testing, and certifying models, according to several embodiments. [Figure 4A] A block diagram showing a system for generating output data according to several embodiments. [Figure 4B] This is a flowchart illustrating a method for performing inter-device matching analysis using trace sensor data, according to several embodiments. [Figure 4C] This is a flowchart illustrating a method for generating summary data according to several embodiments. [Figure 5A] This is a flowchart illustrating a method for generating a dataset for a model to generate predictive data, according to several embodiments. [Figure 5B] This is a flowchart illustrating a method for performing corrective actions using one or more models, according to several embodiments. [Figure 5C] This flowchart illustrates a method for performing a training operation of inter-device matching analysis according to several embodiments. [Figure 5D] This flowchart illustrates a method for performing inference operations for inter-device matching analysis according to several embodiments. [Figure 5E] This is a flowchart of a method 500E for performing corrective actions based on metrics indicating the performance of a processing chamber, according to several embodiments. [Figure 5F] This is a flowchart illustrating a method for performing in-lay analysis according to several embodiments. [Figure 5G] This is a flowchart illustrating a method for performing trace sensor data analysis according to several embodiments. [Figure 5H] This is a flowchart illustrating a method for performing outlier clustering analysis according to several embodiments. [Figure 5I] This is a flowchart illustrating a method for performing in-lay drift detection according to several embodiments. [Figure 6A] This figure shows a visual representation of the operation of dividing a trace into a temporary portion and a steady-state portion, according to several embodiments. [Figure 6B] This figure shows a visual representation of the digital twin output used in the summary data generation operation according to several embodiments. [Figure 6C] This figure shows a visual representation of the clustering operation results according to several embodiments. [Figure 6D] This figure shows the results of the elastic difference model applied to a reference time trace and a similar target time trace. [Figure 6E] This figure shows the results of the elastic difference model applied to the reference time trace and the different target time trace. [Figure 6F] This figure shows exemplary visual representations of trace data analysis results according to several embodiments. [Figure 6G] This figure shows exemplary visual representations of trace data analysis results according to several embodiments. [Figure 7] This diagram illustrates the operation of a model capable of reducing the dimensionality of input data, according to several embodiments. [Figure 8] This is a block diagram showing a computer system in several embodiments. [Modes for carrying out the invention]

[0009] Techniques related to an inter-device matching (TTTM) diagnostic method that can be used for diagnosing problems and / or implementing corrective measures in manufacturing machines are described herein. The manufacturing machines can be used to fabricate products such as substrates (e.g., wafers, semiconductors, displays, optoelectronic devices, etc.). The manufacturing machines (e.g., manufacturing tools) often include a manufacturing chamber that isolates the substrate being processed from the environment. The characteristics of the fabricated substrate should meet target characteristic values to promote performance, functionality, etc. Manufacturing parameters are selected to fabricate a substrate that meets the target characteristic values. The manufacturing parameters can include process parameters such as temperature, gas flow rate, plasma state, pressure, etc., and hardware parameters such as the use of components, the setting of components, the position of components, etc. The manufacturing system controls such parameters by specifying setpoints for the characteristic values and receiving data from one or more sensors disposed within the manufacturing chamber. The manufacturing system adjusts the setpoints of various components until the sensor readings match the setpoints (within the tolerance threshold range).

[0010] The manufacturing system may exhibit or indicate characteristics that result in near-optimal performance, for example, due to component drift, aging, or failures, improperly performed maintenance, changes in target process parameters, etc. Sensor data collected during the manufacturing process can convey information indicating subsystems and / or components with drift, aging, malfunction, or failures. The manufacturing system can include a large number of sensors, and in some systems, can include dozens or even hundreds to thousands of sensors. In some systems, the sensors can collect data over time (e.g., trace data) during the manufacturing procedure. A single time trace (e.g., associated with one sensor and one processing run, or one product being manufactured) can include hundreds of data points, thousands of data points, or more.

[0011] Ensuring consistency in conditions, such as consistency between runs or between chambers within a single chamber, can increase the consistency, performance, and quality of the manufactured product. Inter-machine matching (TTTM) is a process aimed at ensuring consistency. In some cases, sensor data is used as a measure of processing status. In conventional systems, using sensor data for diagnostic and corrective actions can be inconvenient. In some systems, sensor data can be used to identify the root cause of suboptimal performance of a tool. Suboptimal performance may be detected by performing measurements on finished products through, for example, random quality control or sparse sampling. Sensor data of a tool can be investigated in response to measurements of suboptimal performance (e.g., products outside of manufacturing tolerances). Sensor data can be used to identify tool subsystems, components, etc., where aging, drift, failures, malfunctions, etc., have occurred that contribute to the suboptimal performance of the tool. Identifying sensor data of interest can involve comparing trace data from many different processing operations and many different sensors, and can include comparisons with historical and current data. When such a large amount of sensor data is available, isolating sensor data that indicates corrective actions can be extremely cumbersome. This analysis system can be very inefficient in terms of the time spent, computer processing time, and associated energy costs, as well as the risk of missing other indicating sensor data by stopping the analysis after discovering multiple components that require maintenance, replacement, etc., although it does uncover some information.

[0012] In some systems, an investigation into sensor data can be caused by sub-optimal performance, for example, by the production of one or more products having characteristics outside the manufacturing specifications. In some cases, only a portion of the products are subjected to performance measurements, such as metrology measurements, to confirm the production quality. Performance measurements (e.g., metrology) can be costly, for example, it may take a significant amount of time to generate. Manufacturing equipment may continue to be used to produce products while measurements are being performed on previous products and / or while previous products are in a queue for metrology measurements. If the manufacturing equipment has deteriorated (e.g., due to component aging, drift, failure, etc., such that the performance of the equipment has changed to sub-optimal), product metrology measurements may not be performed until multiple products are processed using the sub-optimal equipment. As a result, sub-optimal products may be produced. Such systems are wasteful in terms of the time, energy, and materials wasted in processing sub-optimal products.

[0013] In conventional systems, it is difficult to separate useful sensor data (e.g., identify data indicating corrective actions to be taken in relation to a large amount of sensor data, manufacturing equipment), resulting in measurement-based defect detection. In some cases, sensor data may be observably affected by a drifting, aging, or failing component before the measurement is observably affected. In such cases, the sensor data can be used to schedule corrective actions to occur simultaneously with planned downtime (e.g., preventive maintenance operations) before the measurement is affected, reducing costly unplanned downtime for the manufacturing system. If the sensor data cannot be reliably used, corrective actions may be performed in response to sub-optimal measurement data, resulting in unplanned downtime. In some embodiments, unplanned downtime can incur additional costs, such as expedited shipping for replacement components.

[0014] The methods and devices disclosed herein address at least one or more of the aforementioned shortcomings of conventional solutions. In some embodiments, summary data is enabled and generated, which allows for rapid processing time and communication bandwidth, reduced complexity of analyzing all trace data, and the like. Generating summary data may include the application of statistical methods, machine learning methods, digital twin methods, etc., to trace data (e.g., trace sensor data).

[0015] One or more metrics (e.g., measurement standards) can be used to indicate the health of a manufacturing equipment system. In some embodiments, summary data can be used to generate one or more metric values. Metric values ​​can be used as indications that further investigation should be carried out in relation to chambers, processes, products, components, etc. In some embodiments, metric values ​​(e.g., indicators, quality scores, quality indicator scores, etc.) can be calculated to indicate the absence of observable problems. In some embodiments, inter-device matching (TTTM) indicators can be generated to enable efficient isolation of problematic systems, tools, products, components, etc. Metric values ​​and indicators can be aggregated in various ways to support drill-down analysis, for example, from identifying the presence of a problem to identifying that corrective action should be taken, thereby increasing the understanding of the tool. For example, metrics related to a large number (e.g., more than the past number) of outliers in sensor data (e.g., reflected by summary data) can be used. In some embodiments, metric values ​​that meet a threshold can trigger further analysis, including the generation of other metric values ​​associated with different metrics. In some embodiments, a set of metrics can be used as part of a drill-down analysis. For example, a first metric value (e.g., an index) can indicate that the manufacturing chamber is performing suboptimally, further indices can be aggregated to show that its performance is due to one subsystem, and further indices can be generated to show the processing operations in which performance degradation occurred, and so on.

[0016] In some embodiments, data (e.g., summary data) can be provided to one or more models (e.g., machine learning models) to identify root causes, subsystems, corrective actions to be taken, etc. In some embodiments, data at all stages of drill-down analysis can be visualized and displayed via a graphical interface.

[0017] In some embodiments, analysis can be performed using trace sensor data (for example, in response to one or more metric values ​​or indicators that demonstrate suboptimal performance). Trace data can be provided to a machine learning model. The machine learning model can be trained to identify root causes, problematic subsystems, corrective actions that should be taken, etc. Trace data analysis can be displayed via a graphical interface.

[0018] The aspects of this disclosure offer technical advantages compared to conventional solutions. This disclosure results in a more efficient substrate manufacturing process with minimized waste. If a problem occurs in the manufacturing chamber, the method of this disclosure may allow corrective actions to be taken before processing of the next product begins, before suboptimal products are submitted for quality measurement, or before quality measurement is completed. In this way, wasted material, manufacturing chamber time, and energy supplied to the manufacturing process can be minimized. Product quality can be improved by identifying and correcting the root causes of product quality variability and chamber drift. The efficiency of producing products of acceptable quality can also be increased by more precise tuning of process parameters to improve material costs, energy, and time. Aged components can be identified and flagged for replacement or maintenance, reducing unplanned downtime, costs associated with express shipping of replacement parts, etc.

[0019] In some embodiments, the disclosure describes a method that includes receiving trace sensor data associated with a first manufacturing process of a manufacturing chamber. The method further includes processing the trace sensor data by a processing device to generate summary data associated with the trace sensor data. The method further includes generating a quality index score based on the summary data. The method further includes providing an alarm to a user based on the quality index score. The alarm includes an indication that the manufacturing chamber performance does not meet a first threshold.

[0020] In some embodiments, the Disclosure describes a method comprising receiving first data associated with measurements obtained by a sensor during a first manufacturing procedure of a manufacturing chamber. The method further comprises receiving second data, the second data comprising reference data associated with the first data. The method further comprises providing the first and second data to a comparison model. The method further comprises receiving similarity scores associated with the first and second data from the comparison model. The method further comprises taking the similarity scores into consideration and performing corrective actions.

[0021] In some embodiments, the disclosure describes a method that includes receiving trace sensor data associated with a first manufacturing process of a processing chamber. The method further includes processing the trace sensor data using one or more trained machine learning models that generate a representation of the trace sensor data, and then generating reconstructed sensor data based on the representation of the trace sensor data. One or more trained machine learning models output the reconstructed sensor data. The method further includes comparing the trace sensor data with the reconstructed sensor data. The method further includes determining one or more differences between the reconstructed sensor data and the trace sensor data based on this comparison. The method further includes determining whether to recommend corrective actions associated with the processing chamber based on one or more differences between the trace sensor data and the reconstructed sensor data.

[0022] Figure 1 is a block diagram showing an exemplary system 100 (exemplary system architecture) according to several embodiments. System 100 includes a client device 120, manufacturing equipment 124, sensors 126, measuring equipment 128, a prediction server 112, and a data store 140. The prediction server 112 may be part of the prediction system 110. The prediction system 110 may further include server machines 170 and 180.

[0023] Sensor 126 can provide sensor data 142 associated with manufacturing equipment 124 (for example, associated with the manufacturing of a corresponding product such as a substrate by manufacturing equipment 124). Sensor data 142 can be used for the health of the equipment and / or the health of the product (for example, product quality). Manufacturing equipment 124 can manufacture a product according to a policy or by running a run over a period of time. In some embodiments, sensor data 142 may include one or more values ​​from among temperature (e.g., heater temperature), spacing (SP), pressure, high-frequency radio frequency (HFRF), high-frequency (RF) matched voltage, RF matched current, RF matched capacitor position, electrostatic chuck (ESC) voltage, actuator position, current, flow rate, power, voltage, etc. Sensor data 142 can be associated with or indicate manufacturing parameters such as hardware parameters of manufacturing equipment 124 (e.g., settings or components, e.g., size, type, etc.) or process parameters of manufacturing equipment 124. Alternatively, or in addition, data associated with several hardware parameters can be stored as manufacturing parameters 150, which may include past manufacturing parameters 152 and current manufacturing parameters 154. Manufacturing parameters 150 may indicate input settings for the manufacturing device (e.g., heater power, gas flow rate, etc.). When the manufacturing equipment 124 is performing the manufacturing process, sensor data 142 and / or manufacturing parameters 150 can be provided (e.g., equipment readings generated during product processing). Sensor data 142 may differ per product (e.g., per substrate). The substrate may have characteristic values ​​(e.g., film thickness, film strain, etc.) measured by the measuring equipment 128. Measurement data 160 may be the type of data stored in the data store 140.

[0024] In some embodiments, sensor data 142, measurement data 160, and / or manufacturing parameters 150 can be processed (for example, by a client device 120 and / or a prediction server 112). Processing of sensor data 142, measurement data 160, and / or manufacturing parameters 150 may include generating features. In some embodiments, these features are patterns within the sensor data 142, measurement data 160, and / or manufacturing parameters 150 (e.g., gradient, width, height, peak, etc.) or combinations of values ​​from the sensor data 142, measurement data 160, and / or manufacturing parameters 150 (e.g., power derived from voltage and current, etc.). Sensor data 142 may include features, which may be used by the prediction component 114 to perform signal processing and / or to obtain prediction data 168 for performing corrective actions.

[0025] Each instance (e.g., set) of sensor data 142 can correspond to a product (e.g., a circuit board), a set of manufacturing equipment, or the type of circuit board produced by the manufacturing equipment. Similarly, each instance of measurement data 160 and manufacturing parameter 150 can correspond to a product, a set of manufacturing equipment, or the type of circuit board produced by the manufacturing equipment. The data store can further store information relating sets of different data types; for example, information indicating a set of sensor data, a set of measurement data, and a set of manufacturing parameters can all be associated with the same product, manufacturing equipment, circuit board type, etc.

[0026] In some embodiments, summary data 162 can be generated using data associated with the processing of one or more products. The summary data 162 may include data that characterizes other data. For example, summary data 162 can be generated from trace sensor data 142. Trace sensor data 142 may include a large amount of data (e.g., data from hundreds of sensors in a tool that obtain hundreds or thousands of measurements per product). The summary data 162 may be less cumbersome to work with than the trace sensor data 142 and can be designed to present information (e.g., information indicating chamber failures). In some embodiments, data (e.g., trace sensor data 142) can be provided to a processing device (e.g., a prediction server 112, a client device 120, etc.) to generate summary data 162. The summary data 162 may include, for example, metadata (e.g., tool ID, policy name, product ID, product information, etc.), contextual data (e.g., sensor ID, step number, timestamp, subsystem, etc.), and / or basic statistics (e.g., mean, maximum, minimum, quartiles, kurtosis, control limits, etc.). In some embodiments, the generation of summary data 162 may include the use of a digital twin model of one or more components of the manufacturing equipment 124. Hereinafter, a digital twin is a digital replica of a physical asset, such as a manufactured part, chamber part, or process chamber. The digital twin includes, but is not limited to, the properties of the physical asset, including, coordinate axis dimensions, weighting properties, material properties (e.g., density, surface roughness), electrical properties (e.g., conductivity), and optical properties (e.g., reflectivity). In some embodiments, the digital twin model can be used to generate predictions of the time behavior of data, such as sensor data 142. The summary data 162 may include indications of how the characteristics of the measured data differ from the predictions of the digital twin model (e.g., overshoot, rise time, stabilization time, steady-state value error, etc.).In some embodiments, the summary data 162 can be generated based on a portion of the input data, such as only the steady-state portion or only the transient portion of the summary data 162. The summary data 162 will be described in more detail with reference to Figure 5B.

[0027] In some embodiments, the prediction system 110 can generate prediction data 168 using machine learning, such as supervised machine learning (for example, the machine learning model can be configured to produce labels associated with the input data, such as measurement predictions or performance predictions). In some embodiments, the prediction system 110 can generate prediction data 168 using unsupervised machine learning (for example, the machine learning model can be trained on unlabeled data, such as a model configured to perform clustering, dimensionality reduction, etc.). In some embodiments, the prediction system 110 can generate prediction data 168 using semi-supervised learning (for example, the machine learning model can be trained on both labeled and unlabeled input datasets).

[0028] The client device 120, manufacturing equipment 124, sensor 126, measuring equipment 128, prediction server 112, data store 140, server machine 170, and / or server machine 180 can be connected to each other via network 130 to generate prediction data 168 and perform corrective actions.

[0029] In some embodiments, network 130 is a public network that provides client devices 120 with access to the prediction server 112, the data store 140, and other publicly available computing devices. In some embodiments, network 130 is a private network that provides client devices 120 with access to the manufacturing equipment 124, sensors 126, measuring instruments 128, the data store 140, and other privately available computing devices. In some embodiments, one or more functions of server machines 170, server machine 180, and / or prediction server 112 can be performed by virtual machines, for example, by using cloud-based services. Network 130 can provide access to such virtual machines. Network 130 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof.

[0030] The client device 120 may include computing devices such as personal computers (PCs), laptops, mobile phones, smartphones, tablet computers, netbook computers, network-attached televisions ("smart TVs"), network-attached media players (e.g., Blu-ray® players), set-top boxes, over-the-top (OTT) streaming devices, and operator boxes. The client device 120 may include a corrective action component 122. The corrective action component 122 may receive user input of instructions associated with the manufacturing equipment 124 (e.g., via a graphical user interface (GUI) displayed through the client device 120). In some embodiments, the corrective action component 122 transmits these instructions to the prediction system 110, receives output from the prediction system 110 (e.g., prediction data 168), determines corrective action based on this output, and implements corrective action. In some embodiments, the corrective action component 122 retrieves sensor data 142 associated with the manufacturing equipment 124 (e.g., current sensor data 146) (e.g., from a data store 140, etc.) and provides the sensor data 142 associated with the manufacturing equipment 124 (e.g., current sensor data 146) to the prediction system 110. In some embodiments, the corrective action component 122 stores the sensor data 142 in the data store 140, and the prediction server 112 retrieves the sensor data 142 from the data store 140. In some embodiments, the prediction server 112 may store the output of a trained model 190 (e.g., prediction data 168) in the data store 140, and a client device 120 may retrieve that output from the data store 140. In some embodiments, the corrective action component 122 receives instructions for corrective action from the prediction system 110 and implements the corrective action. Each client device 120 may include an operating system that allows the user to generate, view, or edit one or more data (for example, instructions associated with manufacturing equipment 124, corrective actions associated with manufacturing equipment 124, etc.).

[0031] In some embodiments, the measurement data 160 corresponds to historical characteristic data of a product (e.g., manufactured using manufacturing parameters associated with historical sensor data 144 and historical manufacturing parameters 152), and the prediction data 168 corresponds to predicted characteristic data (e.g., predicted data of a product that should be manufactured or has been manufactured under conditions recorded by current sensor data 146 and / or current manufacturing parameters 154). In some embodiments, the prediction data 168 is predicted measurement data (e.g., virtual measurement data) of a product that should be manufactured or has been manufactured according to conditions recorded as current sensor data 146 and / or current manufacturing parameters 154. In some embodiments, the prediction data 168 is an indication of an anomaly (e.g., an abnormal product, an abnormal component, an abnormal manufacturing equipment 124, abnormal energy usage, etc.), and may be an indication of one or more causes of those anomalies. In some embodiments, the prediction data 168 is an indication of change or drift over time in some component such as manufacturing equipment 124, sensor 126, or measurement equipment 128. In some embodiments, the predictive data 168 indicates the end of life for components such as manufacturing equipment 124, sensors 126, and measuring instruments 128.

[0032] Implementing a manufacturing process that results in defective products can be costly in terms of time, energy, products, components, manufacturing equipment 124, defect identification, and disposal of defective products. By inputting sensor data 142 (e.g., manufacturing parameters used or to be used to manufacture a product) into a predictive system 110, receiving output of predictive data 168, and taking corrective action based on the predictive data 168, system 100 can have the technical advantage of avoiding the costs associated with manufacturing, identifying, and disposing of defective products.

[0033] Executing a manufacturing process that results in a failure of a component of the manufacturing equipment 124 can have significant drawbacks in terms of downtime, product damage, equipment damage, and expedited ordering of replacement components. By inputting sensor data 142 (e.g., indicating manufacturing parameters used or to be used to manufacture a product) into the predictive system 110, receiving the output of predictive data 168, and performing corrective actions (e.g., predictive operational maintenance such as component replacement, processing, or cleaning) based on the predictive data 168, the system 100 can have the technical advantage of avoiding one or more costs such as unexpected component failure, unplanned downtime, loss of production rate, unexpected equipment failure, or product disposal. Monitoring the performance of components, such as the manufacturing equipment 124, sensors 126, and measuring instruments 128, over time can provide indications of degrading components.

[0034] Manufacturing parameters may be suboptimal for producing a product, which can result in costly consequences such as increased resource (e.g., energy, coolant, gas, etc.) consumption, increased time required to produce the product, increased component failures, and an increased quantity of defective products. By inputting sensor data 142 into a trained model 190, receiving the output of predictive data 168, and performing corrective actions such as updating the manufacturing parameters (e.g., setting optimal manufacturing parameters) based on the predictive data 168, the system 100 can have the technical advantage of avoiding the costly consequences of suboptimal manufacturing parameters by using optimal manufacturing parameters (e.g., hardware parameters, process parameters, optimal design) and / or healthy equipment.

[0035] Corrective actions may be associated with one or more of the following: computational process control (CPC), statistical process control (SPC) (e.g., SPC on electronic components to determine the process under control, SPC to predict the useful life of components, SPC for comparison with a 3σ graph, etc.), advanced process control (APC), model-based process control, predictive maintenance, design optimization, manufacturing parameter updates, manufacturing policy updates, feedback control, and machine learning modifications.

[0036] In some embodiments, the corrective action includes providing an alert (for example, a warning to stop or not perform the manufacturing process on an additional substrate if the predictive data 168 indicates a predicted anomaly, such as an anomaly in the product, component, or manufacturing equipment 124). In some embodiments, the corrective action includes providing feedback control (for example, modifying manufacturing parameters in response to the predictive data 168 indicating a predicted anomaly). In some embodiments, the execution of the corrective action includes updating one or more manufacturing parameters.

[0037] Manufacturing parameters may include hardware parameters (e.g., information indicating components contained within the manufacturing equipment, instructions for recently replaced components, instructions for firmware versions or updates, etc.) and / or process parameters (e.g., temperature, pressure, flow rate, current and / or voltage, gas flow rate, rise rate, etc.). In some embodiments, corrective actions include performing preventive operational maintenance (e.g., replacing, processing, cleaning components of the manufacturing equipment 124, etc.). In some embodiments, corrective actions include performing design optimization (e.g., updating manufacturing parameters for an optimized product, updating the manufacturing process, updating the manufacturing equipment 124, etc.). In some embodiments, corrective actions include updating policies (e.g., changing the timing of instructions to put the manufacturing equipment 124 into idle mode, sleep mode, warm-up mode, etc., adjusting setpoints for temperature, gas flow rate, plasma generation, etc.).

[0038] Each of the prediction server 112, server machine 170, and server machine 180 may include one or more computing devices such as rack-mount servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, graphics processing units (GPUs), and application-specific integrated circuits (ASICs) (for example, tensor processing units (TPUs)).

[0039] The prediction server 112 may include a prediction component 114. In some embodiments, the prediction component 114 can receive current sensor data 146 and / or current manufacturing parameters 154 (e.g., received from a client device 120, retrieved from a data store 140) and, based on the current data, generate an output (e.g., prediction data 168) for taking corrective actions associated with the manufacturing equipment 124. In some embodiments, the prediction component 114 can use one or more trained models 190 to determine the output for taking corrective actions based on the current data.

[0040] In some embodiments, Model 190 may include a trained physical-based digital twin model. The physical-based model may be capable of solving systems of equations describing physical phenomena that may occur within the manufacturing chamber, such as equations governing heat flow, energy balance, gas conduction, mass balance, fluid dynamics, and electric current flow. In some embodiments, the physical-based model performs calculations of gas conduction within the manufacturing chamber. Manufacturing parameters 150 can be provided to the trained physical-based model. The trained physical-based model may provide modeled characteristic values ​​as outputs that represent the state within the chamber, corresponding to sensors 126 placed within the manufacturing chamber (e.g., manufacturing equipment 124). The outputs of the physical-based model can be stored in the data store 140.

[0041] The prediction component 114 of the prediction server 112 can receive summary data 162 generated from data collected by the sensor 126. The prediction component 114 can use the summary data 162 to generate prediction data 168. The prediction data 168 may include indications such as failures, aging, and drift in the manufacturing equipment 124, the sensor 126, etc. In some embodiments, the summary data can be used to generate metrics indicating the health and / or performance of indicators, such as tools, chambers, sets of manufacturing equipment 124, etc. The indicators can be used for corrective actions, can be displayed to the user, for example on a GUI, and can be used to perform further analysis, and so on. The indicators may indicate that further investigation should be carried out for causes such as failures, aging, and drift. In some embodiments, the indicators can be generated by providing the summary data to a trained machine learning model (e.g., model 190). In some embodiments, the prediction component 114 can use summary data associated with indicators indicating further investigation to be carried out (e.g., summary data associated with irregular indicator values) to isolate subsystems related to the performance of a tool. In some embodiments, summary data associated with indicators that indicate further investigation to be performed is used by the predictive component 114 to perform an analysis that reveals unexpected drift in sensor data. In some embodiments, trace data associated with indicators that indicate further investigation to be performed can be used to isolate subsystems that contribute to the performance of the tool. Further discussion of the methods related to the generation of summary data, the generation of indicators, and drill-down analysis is provided in relation to Figures 5A to 5G.

[0042] Using historical sensor data 144 and historical manufacturing parameters 152, one or more models 190, including physical-based models, digital twin models, and machine learning models, can be trained. In some embodiments, a digital twin model can be used to generate summary data. The digital twin model can be generated from physical principles. In some embodiments, the generated digital twin model may not capture some complex parts of the manufacturing system, such as parts that may be manufactured slightly differently (e.g., within manufacturing tolerances), parts that are aged or drifted, and so on. To increase accuracy, historical data can be used to refine the physical-based model or the digital twin model. Historical data can also be used to train one or more machine learning models. In some embodiments, machine learning models can be used to generate metrics, generate visual representations of data, generate predictions of measurement data for finished products, and so on.

[0043] In some embodiments, the sensor data 142 may include data collected from the sensor 126 during a manufacturing run that produced an acceptable product (e.g., measured by the measuring instrument 128). A manufacturing run that produces an acceptable product may be called a golden run. Sensor data associated with such a manufacturing run may be stored in the data store 140 as part of the historical sensor data 144. The prediction component 114 of the prediction server 112 can compare the golden run sensor data, the current sensor data 146, and the expected sensor data (e.g., output by a trained physical-based model) to determine whether a component failure, drift, etc., has occurred. In some embodiments, instead, some or all of these operations may be performed by different devices, such as a client device 120, a server machine 170, a server machine 180, etc.

[0044] In some embodiments, the predictive component 114 can determine which components contribute to the difference between expected sensor data (e.g., historical sensor data, sensor data from different tools, sensor data reflecting manufacturing parameters, etc.), current sensor data 146, and / or golden run sensor data. This can be done, for example, by examining historical data (e.g., by machine learning models, statistical models, etc.) of systems that perform as intended and systems that exhibit suboptimal performance. By labeling the sensor trace data with indications of the cause of failure (e.g., a subsystem that is causing suboptimal performance), the processing device can enable linking sensor differences (e.g., a specific pattern or feature in the data from a particular sensor) to failures. The processing logic can be configured to link specific features or patterns in the sensor data to a specific type of failure, aging, drift, or failure of a particular component, etc. Sensors can be assigned performance-indicating metric values, such as a metric value that aggregates the likelihood that a sensor will exhibit failure over multiple operations or runs. Sensor data can be grouped into subsystems. Sensor metric values ​​or indices can be aggregated into subsystem metric values ​​or indices. Aggregated subsystem values ​​(e.g., metric values, indicators, etc.) can be used to identify subsystem performance, among other things.

[0045] A machine learning model (for example, one or more of the models 190) can be trained using historical sensor data 144, historical manufacturing parameters 152, measurement data 160 corresponding to runs in which the historical sensor data 144 was measured, etc. One type of machine learning model that can be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network. An artificial neural network generally consists of feature representation components that have classification or regression layers that map features to a target output space. A convolutional neural network (CNN) hosts multiple layers of convolutional filters, for example. Pooling is performed so that nonlinearity can be dealt with at lower layers, and typically a multilayer perceptron is added on top of that, where the upper layer features extracted by the convolutional layers are mapped to decisions (e.g., classification outputs). Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer takes the output from the previous layer as input. Deep neural networks can be trained in supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) ways. A deep neural network has a hierarchical structure consisting of multiple layers, where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. In an example of image recognition application, for instance, the raw input could be a matrix of pixels; the first representation layer could abstract the pixels and encode the edges; the second layer could construct and encode the arrangement of the edges; the third layer could encode higher-level shapes (e.g., leaves, lips, gums, etc.); and the fourth layer could recognize the role of the scan. In particular, the deep learning process can learn which features naturally and optimally fit into which levels. The "deep" in "deep learning" refers to the number of layers to which the data is transformed. More precisely, a deep learning system has a substantial contribution distribution path (CAP) depth. CAP is a chain of transformations from input to output.A CAP describes the potentially causal connections between inputs and outputs. For a forward neural network, the CAP depth may be the network depth, which can be the number of hidden layers plus one. For a regressive neural network, where a signal can propagate through a single layer more than once, the CAP depth is potentially unlimited.

[0046] The use of a physical-based digital twin model associated with manufacturing equipment offers significant technical advantages compared to operating the equipment without such a model. Multiple components (sensors, hardware, processing, etc.) can be linked together in a closed control loop. If a problem exists with any of the components in the closed control loop, other components can adjust setpoints, etc., and readings can return to target values, but the actual state within the chamber may have changed. This can lead to inconsistent substrate fabrication, loss of fabrication rate, inefficiencies in energy, time, and materials, premature component failure, and increased maintenance costs. In some embodiments, multiple data streams exhibiting the same characteristics can be monitored to provide additional evidence of the root cause of changes in sensor data, chamber state, etc. In some embodiments, a sensor indicating chamber pressure is monitored. Sensors may include those that measure chamber pressure, actuator position of actuators affecting chamber pressure, RF-matched voltage, RF-matched current, RF-matched capacitor position, etc. Some of these characteristics can be measured by two or more sensors. For example, a manufacturing chamber may have multiple pressure sensors, some of which are included in a closed control loop, while others freely provide measurements of the chamber state. Physically based models can capture and analyze even slight changes, modifications, and drifts in sensor readings, sensor performance, and other parameters.

[0047] In some embodiments, the predictive component 114 receives current sensor data 146 and / or current manufacturing parameters 154, performs signal processing to decompose the current data into a set of current data, provides the set of current data as input to a trained model 190, and obtains an output showing predictive data 168 from the trained model 190. In some embodiments, the predictive data 168 shows measured data 160 (e.g., a prediction of substrate quality). In some embodiments, the predictive data 168 shows the health status of the component. In some embodiments, the predictive data 168 shows the performance of the component.

[0048] In some embodiments, the various models discussed in relation to Model 190 (e.g., a physically based digital twin model, a predictive machine learning model, etc.) can be combined into a single model (e.g., an ensemble model) or remain separate models. The predictive component 114 receives current sensor data 146 and current manufacturing parameters 154, provides the data to the trained model 190, and can receive information indicating how much some components in the manufacturing chamber have drifted from their previous performance. Data can be passed between several separate models included in Model 190 and the predictive component 114. In some embodiments, instead, some or all of these operations can be performed by different devices, such as a client device 120, a server machine 170, a server machine 180, etc. It will be understood by those skilled in the art that variations in data flow, which components perform which processes, and which models receive which data are within the scope of this disclosure.

[0049] The data store 140 can be memory (e.g., random access memory), a drive (e.g., a hard drive, a flash drive), a database system, or another type of component or device capable of storing data. The data store 140 can include multiple storage components (e.g., multiple drivers or multiple databases) that can span multiple computing devices (e.g., multiple server computers). The data store 140 can store sensor data 142, manufacturing parameters 150, measurement data 160, summary data 162, and predictive data 168. The sensor data 142 can include historical sensor data 144 and current sensor data 146. The sensor data can include a sensor data time trace over the duration of the manufacturing process, correlation between data and physical sensors, pre-processed data such as averages and composite data, and data showing sensor performance over time (i.e., many manufacturing processes). The manufacturing parameters 150 and measurement data 160 can include similar characteristics. The historical sensor data 144 and historical manufacturing parameters 152 can be historical data (e.g., at least a portion for training model 190). The current sensor data 146 can be the current data (for example, the portion that should be input to the learning model 190 after at least the historical data) from which predictive data 168 (for example, to perform corrective actions) should be generated. The summary data 162 may include the processed sensor data 142. The summary data may include information indicating corrective actions to be performed in relation to the manufacturing equipment 124. The summary data does not need to be as heavily manipulated as the trace sensor data.

[0050] In some embodiments, the prediction system 110 further includes server machines 170 and 180. Server machine 170 includes a dataset generator 172 capable of generating datasets (e.g., a set of data inputs and a set of target outputs) for training, certifying, and / or testing a model 190, which includes a physically based digital twin model and, in some embodiments, one or more machine learning models. Some operations of the dataset generator 172 are described in detail below with reference to Figures 3, 4, and 5A. In some embodiments, the dataset generator 172 can divide historical data (e.g., historical sensor data 144, historical manufacturing parameters 152, summary data 162 stored in data store 140) into a training set (e.g., 60 percent of the historical data), a certification set (e.g., 20 percent of the historical data), and a test set (e.g., 20 percent of the historical data). In some embodiments, the prediction system 110 generates multiple sets of features (e.g., via prediction component 114). For example, a first set of features could correspond to a first set of sensor data types corresponding to each of the datasets (e.g., a training set, a certification set, and a test set) (e.g., a first set of sensors, a first combination of values ​​from the first set of sensors, and a first pattern of values ​​from the first set of sensors), and a second set of features could correspond to a second set of sensor data types corresponding to each of the datasets (e.g., a second set of sensors different from the first set of sensors, values ​​from a second combination different from the first combination, and a second pattern different from the first pattern).

[0051] The server machine 180 includes a training engine 182, an authentication engine 184, a selection engine 185, and / or a test engine 186. The engines (e.g., training engine 182, authentication engine 184, selection engine 185, and test engine 186) can refer to hardware (e.g., circuits, dedicated logic, programmable logic, microcode, processing devices, etc.), software (e.g., instructions executed on processing devices, general-purpose computer systems, or dedicated machines), firmware, microcode, or a combination thereof. The training engine 182 may be capable of training a model 190 using one or more sets of features associated with a training set from a dataset generator 172. The training engine 182 can generate multiple trained models 190, each trained model 190 corresponding to a distinct set of features from the training set (e.g., sensor data from a distinct set of sensors). For example, a first trained model may be trained using all features (e.g., X1-X5), a second trained model may be trained using a first subset of features (e.g., X1, X2, X4), and a third trained model may be trained using a second subset of features (e.g., X1, X3, X4, and X5), where the second subset of features may partially overlap with the first subset of features. The dataset generator 172 can receive the output of the trained models (e.g., a physically based digital twin model, a machine learning model), collect that data into training, authentication, and test datasets, and use these datasets to train a second model (e.g., a machine learning model configured to output predictive data, corrective actions, etc.).

[0052] The authentication engine 184 may authenticate the trained models 190 using a corresponding set of features from the authentication set of the dataset generator 172. For example, a first trained model 190 trained using a first set of features from the training set can be authenticated using a first set of features from the authentication set. The authentication engine 184 can determine the accuracy of each trained model 190 based on the corresponding set of features from the authentication set. The authentication engine 184 may discard trained models 190 that have an accuracy that does not meet a threshold accuracy. In some embodiments, the selection engine 185 may select one or more trained models 190 that have an accuracy that meets a threshold accuracy. In some embodiments, the selection engine 185 may select the trained model 190 that has the highest accuracy.

[0053] The test engine 186 may be able to test the trained model 190 using a corresponding set of features from the test set generated by the dataset generator 172. For example, a first trained model 190 trained using a first set of features from the training set can be tested using a first set of features from the test set. Based on the test set, the test engine 186 can determine which trained model 190 has the highest accuracy among all the trained models.

[0054] In the case of a machine learning model, model 190 can refer to a model artifact created by the training engine 182 using a training set containing data inputs and corresponding target outputs (the correct response for each training input). Patterns in the dataset that map data inputs to target outputs (correct responses) can be discovered, and the machine learning model 190 is provided with mappings that capture these patterns. The machine learning model 190 can use one or more of the following: support vector machines (SVMs), radial basis functions (RBFs), clustering, supervised machine learning, semi-supervised machine learning, unsupervised machine learning, k-nearest neighbors algorithm (k-NN), linear regression, random forests, and neural networks (e.g., artificial neural networks).

[0055] The prediction component 114 can provide the current sensor data 146 and / or summary data 162 to the model 190, run the model 190 on the input, and obtain one or more outputs. The prediction component 114 may determine (e.g., extract) prediction data 168 from the output of the model 190, and may determine (e.g., extract) confidence data from the output indicating the level of confidence that the prediction data 168 is an accurate predictor of the process associated with the input data for products made or to be made using the manufacturing equipment 124 with the current sensor data 146 and / or current manufacturing parameters 154. The prediction component 114 or the corrective action component 122 may use the confidence data to determine, based on the prediction data 168, whether to trigger a corrective action associated with the manufacturing equipment 124.

[0056] Confidence data may include or indicate a confidence level that the prediction data 168 is an accurate prediction of a product or component associated with at least a portion of the input data. For example, the confidence level is a real number between 0 and 1, where 0 indicates no confidence that the prediction data 168 is an accurate prediction of a product processed according to the input data or the component health status of the components of the manufacturing equipment 124, and 1 indicates absolute confidence that the prediction data 168 accurately predicts the characteristics of a product processed according to the input data or the component health status of the components of the manufacturing equipment 124. In response to confidence data indicating a confidence level below a threshold level for a given number of cases (e.g., a percentage of cases, a frequency of cases, a total number of cases), the prediction component 114 (e.g., based on current sensor data 146, current manufacturing parameters 154, etc.) may allow the trained model 190 to be retrained.

[0057] For illustrative purposes only, not limiting, aspects of this disclosure describe training one or more machine learning models 190 using historical data (e.g., historical sensor data 144, historical manufacturing parameters 152) and inputting current data (e.g., current sensor data 146, current manufacturing parameters 154, etc.) into one or more trained machine learning models to determine predictive data 168. In other embodiments, a heuristic model or a rule-based model is used to determine predictive data 168 (e.g., without using a trained machine learning model). The predictive component 114 may monitor historical sensor data 144, historical manufacturing parameters 152, summary data 162, and measurement data 160. The heuristic or rule-based model may monitor or otherwise use any of the information described with respect to the data input 310 in Figure 3.

[0058] In some embodiments, the functions of the client device 120, prediction server 112, server machine 170, and server machine 180 can be provided by fewer machines. For example, in some embodiments, server machines 170 and 180 can be integrated into a single machine, and in some other embodiments, server machine 170, server machine 180, and prediction server 112 can be integrated into a single machine. In some embodiments, the client device 120 and prediction server 112 can be integrated into a single machine.

[0059] In general, functions described in one embodiment as being performed by the client device 120, prediction server 112, server machine 170, and server machine 180 can, where appropriate, also be performed by the prediction server 112 in other embodiments. In addition, functions attributable to a particular component can also be performed by different or multiple components working together. For example, in some embodiments, the prediction server 112 can determine corrective actions based on the prediction data 168. In another example, the client device 120 can determine the prediction data 168 based on the output from a trained machine learning model, a physics-based model, etc.

[0060] In addition, the functionality of a particular component can be performed by different or multiple components working together. One or more of the prediction server 112, server machine 170, or server machine 180 can be accessed as a service provided to other systems or devices via an appropriate application programming interface (API).

[0061] In some embodiments, “User” can represent a single individual. However, other embodiments of this disclosure also include “Users” that are entities managed by multiple users and / or automated sources. For example, a set of individual users integrated as a group of administrators can be considered a “User.”

[0062] Embodiments of this disclosure can be applied to data quality evaluation, feature enhancement, model evaluation, virtual instrumentation (VM), predictive maintenance (PdM), marginal optimization, and the like.

[0063] Figure 2 shows a cross-sectional view of a manufacturing chamber 200 (e.g., a semiconductor wafer manufacturing chamber) according to several embodiments of the present disclosure. The manufacturing chamber 200 may be one or more of the following: an etching chamber (e.g., a plasma etching chamber), a deposition chamber (including atomic layer deposition, chemical vapor deposition, physical vapor deposition, or a plasma version thereof), an annealing chamber, etc. For example, the manufacturing chamber 200 may be a chamber for a plasma etcher, a plasma cleaner, etc. Examples of chamber components may include a substrate support assembly 204, an electrostatic chuck, a ring (e.g., a process kit ring), a chamber wall, a base, a shower head 206, a gas distribution plate, a liner, a liner kit, a shield, a plasma screen, a flow balancer, a cooling base, a chamber viewport, a chamber lid, a nozzle, etc.

[0064] In one embodiment, the manufacturing chamber 200 includes a chamber body 208 and a shower head 206 that seal an internal volume 210. In some chambers, the shower head 206 can be replaced with a lid and nozzle. The chamber body 208 can be constructed from aluminum, stainless steel, or other suitable material. The chamber body 208 generally includes side walls 212 and a bottom 214.

[0065] An exhaust port 216 can be defined within the chamber body 208, and the internal volume 210 can be connected to a pump system 218. The pump system 218 may include one or more pumps and valves used to exhaust and regulate the pressure of the internal volume 210 of the manufacturing chamber 200. Actuators for controlling the gas flow rate out of the chamber and / or the pressure inside the chamber may be located at or near the exhaust port 216.

[0066] The shower head 206 can be supported on the side wall 212 of the chamber body 208 or on the top of the chamber body. The shower head 206 (or a lid in some embodiments) can be opened to allow access to the internal volume 210 of the manufacturing chamber 200 and can provide a seal for the manufacturing chamber 200 when closed. A gas panel 220 can be coupled to the manufacturing chamber 200 to provide process or cleaning gases to the internal volume 210 via the shower head 206 (or lid and nozzles). Multiple gas delivery holes can be included in the shower head 206 as a whole. Examples of process gases that can be used to process substrates in the manufacturing chamber 200 include, among other things, halogen-containing gases such as C2F6, SF6, SiCl4, HBr, NF3, CF4, CHF3, F2, Cl2, CCl4, BCl3, and SiF4, as well as other gases such as O2 or N2O. Examples of carrier gases include N2, He, Ar, and other gases that are inert to the process gas (e.g., non-reactive gases).

[0067] The substrate support assembly 204 is positioned below the shower head 206 within the internal volume 210 of the manufacturing chamber 200. In some embodiments, the substrate support assembly 204 includes a susceptor 222 and a shaft 224. The substrate support assembly 204 supports the substrate during processing. In some embodiments, one or more heaters 226 and reflectors 228 are also positioned within the manufacturing chamber 200.

[0068] Sensors are used to measure characteristic values ​​within the manufacturing chamber 200. Sensors associated with the manufacturing chamber 200 may include temperature sensors, pressure sensors (in some embodiments, at least two pressure sensors may be present), flow sensors, and the like. In some embodiments, a physical-based digital twin model of the manufacturing chamber 200 can be used to identify faulty components and take corrective action.

[0069] In some embodiments, the chamber 200 may include multiple sensors for collecting time trace data. The time trace data can be transmitted to a data storage device (e.g., data store 140 in Figure 1). Summary data can be generated from the time trace sensor data. Processing of the summary data and / or time trace data can generate data indicating problematic chambers, subsystems, components, etc. (e.g., faults, drift, aging, etc.). Data processing can be performed by one or more models, including physical-based models, machine learning models, digital twin models, etc. The summary data can be used to generate metrics. The values ​​of the metrics can indicate the performance of tools, components, fleets, etc.

[0070] Figure 3 shows a dataset generator 372 (e.g., dataset generator 172 in Figure 1) that creates datasets for training, testing, and certification of a model (e.g., model 190 in Figure 1) in several embodiments. The dataset generator 372 can be part of the server machine 170 in Figure 1. In some embodiments, model 190 can include multiple machine learning models, physical-based models, digital twin models, etc. Each model may have its own dataset generator, or multiple models may share a dataset generator. The operation of one dataset generator (associated with a model used to map sensor data to performance data) is described in detail in relation to Figure 3. Any of the other models may also be associated with a similar dataset generation procedure. For other models, the input and output data may differ from the input and output data described in relation to Figure 3.

[0071] A system 300, including a dataset generator 372 (for example, dataset generator 172 in Figure 1), creates datasets for a machine learning model (for example, model 190 in Figure 1). The dataset generator 372 can create datasets (for example, data input 310) using historical manufacturing parameters 352 (for example, historical manufacturing parameters 152 in Figure 1), historical sensor data, summary sensor data, etc. The system 300 can be used to generate datasets for training, testing, and certifying machine learning models. In some embodiments, performance data 322 is provided to the machine learning model as a target output 320. The performance data may represent characteristics of the finished product, for example, measurement data of a processed substrate. In some embodiments, the performance data may represent the state of one or more components of the manufacturing equipment, for example, faults, aging, drift, recommended maintenance, etc. In some embodiments, the dataset generator 372 can be configured to generate datasets for a physical-based model of the components of the manufacturing equipment. The output of the physical-based model can be refined by training the model using sensor data as a target output. In some embodiments, sensor data associated with a successful process (e.g., golden run data from a process that produced a product within manufacturing tolerances, golden run data from a recently certified chamber, etc.) can be used as input 310 or target output 320.

[0072] In some embodiments, the dataset generator 372 generates a dataset (e.g., a training set, a certification set, a test set) which includes one or more data inputs 310 (e.g., a training input, a certification input, a test input) and may include one or more target outputs 320 corresponding to the data inputs 310. The dataset may also include mapping data that maps the data inputs 310 to the target outputs 320. The data inputs 310 may also be referred to as “features,” “attributes,” or “information.” In some embodiments, the dataset generator 372 may provide the dataset to the training engine 182, the certification engine 184, or the test engine 186, which may use this dataset to train, certification, or test one or more models 190 (e.g., model 190, one of the models included in an ensemble model 190, etc.). In some embodiments, model 190 may include a physically based digital twin model and a model used to analyze the output of the physically based model. In some embodiments, the dataset generator 372 can supply input to a physics-based model as training input (e.g., manufacturing parameters) and can provide target output data associated with the output of a second model (e.g., component contribution data from a machine learning model), and these associated datasets can be used to train the model 190. Several embodiments for generating the training set can be further described with reference to Figure 5A.

[0073] In some embodiments, the dataset generator 372 generates a data input 310 to supply a training dataset for an unsupervised machine learning model such as a clustering model, an autoencoder, or a dimensionality reduction model, but does not generate a target output 320. The operation of such models will be further described in relation to Figure 7. In some embodiments, the dataset generator 372 generates both a data input 310 and a target output 320 (for example, to train an unsupervised or semi-supervised model). In some embodiments, the data input 310 may include one or more sets of data. As an example, the system 300 may produce a set of sensor data, such sensor data may include one or more of the following: sensor data from one or more types of sensors, combinations of sensor data from one or more types of sensors, patterns from sensor data from one or more types of sensors, manufacturing parameters from one or more manufacturing parameters, combinations of some manufacturing parameter data and some sensor data, summary data, etc.

[0074] In some embodiments, the dataset generator 372 can generate a first data input corresponding to a first set of historical sensor data 344A and / or historical manufacturing parameters 352A for training, certifying, and / or testing a first machine learning model, and the dataset generator 372 can generate a second data input corresponding to a second set of historical sensor data 344B and / or historical manufacturing parameters 352B for training, certifying, and / or testing a second machine learning model.

[0075] The data input 310 and target output 320 for training, certifying, and / or testing a machine learning model may include information for a specific manufacturing chamber (e.g., a specific substrate manufacturing machine). For example, historical manufacturing parameters 352, historical sensor data 344, summary data, and performance data 322 can be associated with the same manufacturing chamber.

[0076] In some embodiments, the information used to train a machine learning model may be from a specific type of manufacturing equipment of a manufacturing facility having specific characteristics (e.g., manufacturing equipment 124 in Figure 1), and the trained model may be able to determine the outcome for the specific group of manufacturing equipment 124 based on input to current sensor data (e.g., current sensor data 146) associated with one or more components that share the characteristics of the specific group of manufacturing equipment 124. In some embodiments, the information used to train this model may be for components from two or more manufacturing facilities, and the trained machine learning model may be able to determine the outcome for a component based on input from one manufacturing facility.

[0077] In some embodiments, a dataset can be generated, and the machine learning model 190 can be trained, validated, or tested using that dataset, after which the model 190 can be further trained, validated, tested, or tuned (for example, by adjusting the weights or parameters associated with the input data of the model 190, such as parameters in a physically based model or connection weights in a neural network).

[0078] Figure 4A is a block diagram showing a system 400 for generating output data (e.g., predictive data 168 in Figure 1) according to several embodiments. In some embodiments, system 400 can be used with a machine learning model (e.g., model 190 in Figure 1) to determine the correlation between the performance of sensors and a set of tools, chambers, or manufacturing equipment, thereby notifying the implementation of corrective actions (e.g., corrective actions can be taken considering simulated sensor data). In some embodiments, a system similar to system 400 can be used with a physical-based model to determine corrective actions associated with manufacturing equipment.

[0079] In block 410, system 400 (for example, a component of the prediction system 110 in Figure 1) performs data partitioning of historical data 464 (for example, historical performance data for model 190 in Figure 1, historical manufacturing parameters 152, and historical sensor data 144) (for example, via the dataset generator 172 of server machine 170 in Figure 1) to generate a training set 402, a certification set 404, and a test set 406. For example, the training set may consist of 60% of the historical data, the certification set may consist of 20% of the historical data, and the test set may consist of 20% of the historical data.

[0080] The generation of training set 402, authentication set 404, and test set 406 can be adjusted to suit a specific application. For example, the training set may consist of 60% of the historical data, the authentication set 20% of the historical data, and the test set 20% of the historical data. System 400 can generate multiple sets of features for each of the training set, authentication set, and test set. For example, if the historical data 464 includes sensor data from 20 sensors (e.g., sensor 126 in Figure 1) and features derived from 10 manufacturing parameters (e.g., manufacturing parameters corresponding to the sensor data from the 20 sensors), this sensor data can be divided into a first set of features including sensors 1-10 and a second set of features including sensors 11-20. Manufacturing parameters can also be divided into multiple sets, for example, a first set of manufacturing parameters including parameters 1-5 and a second set of manufacturing parameters including parameters 6-10. Either or both of the target inputs and target outputs can be divided into multiple sets, or neither can be divided. Multiple machine learning models, including physically based models, can be trained on different sets of data.

[0081] In block 412, system 400 performs model training using training set 402 (for example, via training engine 182 in Figure 1). Training of machine learning models and / or physically based models (e.g., digital twins) can be achieved in a supervised learning manner, which involves feeding a training dataset containing labeled inputs through this model, observing its output, defining the error (by measuring the difference between the output and the labeled value), and adjusting the model's weights to minimize the error using techniques such as deep gradient descent and backpropagation. In many applications, this process is repeated across many labeled inputs in the training dataset to obtain a model that can produce the correct output when presented with inputs different from those present in the training dataset.

[0082] For each training data item in the training dataset, the training data item can be input into a model (e.g., a machine learning model). The model can then process the input training data item (e.g., process policies from past runs) to produce an output. The output may include, for example, predicted performance data for the processed product. This output can be compared to the labels of the training data item (e.g., actual performance metrics measured).

[0083] Next, the processing logic can compare the generated output (e.g., predicted performance) included in the training data items with the labels (e.g., actual performance measurements). Based on the difference between the output and the labels, the processing logic determines the error (i.e., classification error). Based on this error, the processing logic adjusts one or more weights and / or values ​​of the model.

[0084] When training a neural network, an error term or Δ can be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters (weights for one or more inputs to the node) for one or more of its nodes. Parameters can be updated in a backpropagation manner, so the nodes of the highest layer are updated first, followed by the nodes of the next layer, and so on. An artificial neural network contains multiple layers of "neurons," each layer receiving input values ​​from the neurons of the previous layer. The parameters for each neuron include weights associated with the values ​​received from each of the neurons of the previous layer. Therefore, adjusting parameters can include adjusting the weights assigned to each of the inputs to one or more neurons in one or more layers within the artificial neural network.

[0085] System 400 can train multiple models using multiple sets of features from the training set 402 (e.g., a first set of features from the training set 402, a second set of features from the training set 402, etc.). For example, System 400 can train models to generate a first trained model using a first set of features in the training set (e.g., sensor data from sensors 1-10) and a second trained model using a second set of features in the training set (e.g., sensor data from sensors 11-20). In some embodiments, the first and second trained models can be combined to generate a third trained model (e.g., which may be a better predictor than the first or second trained model itself). In some embodiments, the sets of features used when comparing models may overlap (e.g., the first set of features may be sensor data from sensors 1-15, and the second set of features may be sensor data from sensors 5-20). In some embodiments, hundreds of models can be generated, including models and combinations of models with various permutations of features. In some embodiments, hundreds or more models can be stacked together to generate a single ensemble model that has greater predictive power than any individual model alone.

[0086] In block 414, system 400 performs model authentication using authentication set 404 (for example, via authentication engine 184 in Figure 1). System 400 can authenticate each of the trained models using the corresponding set of features in authentication set 404. For example, system 400 can authenticate a first trained model using a first set of features in the authentication set (e.g., sensor data from sensors 1-10) and authenticate a second trained model using a second set of features in the authentication set (e.g., sensor data from sensors 11-20). In some embodiments, system 400 can authenticate hundreds of models generated in block 412 (e.g., models with various permutations of features, combinations of models, etc.). In block 414, system 400 can determine the accuracy of each of one or more trained models (e.g., via model authentication) and determine whether one or more of the trained models have an accuracy that meets a threshold accuracy. In response to determining that none of the trained models have the accuracy to meet the threshold, the flow returns to block 412, where system 400 trains models using different sets of features from the training set. In response to determining that one or more of the trained models have the accuracy to meet the threshold, the flow proceeds to block 416. System 400 can discard trained models that have accuracy below the threshold (for example, based on the authentication set).

[0087] In block 416, system 400 performs model selection to determine which of the one or more trained models that satisfy the threshold precision has the highest precision (for example, the selected model 408 based on the authentication in block 414) (for example, via the selection engine 185 in Figure 1). In response to determining that two or more of the trained models that satisfy the threshold precision have the same precision, the flow can return to block 412, where system 400 performs model training using a more refined training set corresponding to a more refined set of features for determining the trained model with the highest precision.

[0088] In block 418, system 400 tests the selected model 408 by performing a model test using test set 406 (for example, via test engine 186 in Figure 1). System 400 can test a first trained model using a first set of features in the test set (e.g., sensor data from sensors 1-10) and determine which first trained model meets a threshold precision (e.g., based on the first set of features in test set 406). In response to the selected model 408 not meeting the threshold precision (e.g., the selected model 408 is overfitted to training set 402 and / or certification set 404 and cannot be applied to other datasets such as test set 406), the flow proceeds to block 412, where system 400 performs model training (e.g., retraining) using a different set of features (e.g., sensor data from different sensors) and a corresponding different training set. In response to determining, based on test set 406, that the selected model 408 has precision that meets the threshold precision, the flow proceeds to block 420. In at least block 412, the model can learn patterns from past data to make predictions, and in block 418, the system 400 can apply the model to the remaining data (e.g., test set 406) to test these predictions.

[0089] In block 420, system 400 receives current sensor data 446 (e.g., current sensor data 146 in Figure 1) using a trained model (e.g., selected model 408) and determines (e.g., extracts) predicted performance data 468 (e.g., predicted data 168 in Figure 1) from the output of the trained model. Taking the predicted performance data 468 into account, corrective actions associated with the manufacturing equipment 124 in Figure 1 can be performed. In some embodiments, the current sensor data 446 may correspond to the same type of features in historical manufacturing parameter data. In some embodiments, the current sensor data 446 may correspond to the same type of features as a subset of the types of features in historical manufacturing parameter data used to train the selected model 408.

[0090] In some embodiments, current data is received. Current data may include current performance data 460 (for example, measurement data 160 in Figure 1). Model 408 is retrained based on the current data. In some embodiments, a new model is trained based on current sensor data 446 and current performance data 460.

[0091] In some embodiments, one or more of operations 410-420 may be performed in various orders and / or in conjunction with other operations not presented and described herein. In some embodiments, one or more of operations 410-420 may not be performed. For example, in some embodiments, one or more of the data partitioning of block 410, model authentication of block 414, model selection of block 416, or model testing of block 418 may not be performed.

[0092] A system similar to System 400 in Figure 4A can be configured to train, certify, test, and use a physical-based digital twin model. The physical-based model is configured to accept manufacturing parameters (e.g., setpoints provided to manufacturing equipment) as input and to provide simulated sensor data (e.g., predicted sensor data) as output. Other models related to this physical-based model can also follow a similar data flow for training, certification, testing, and use. In some embodiments, a model can be configured to accept simulated and measured sensor data and to output a list of components predicted to contribute to the difference between the two datasets. The model can provide an estimate of how much each component contributes to their variation. For example, the model can provide an estimate of power delivery to a component. Sensor data describing power delivery can be provided to the physical-based model, or the output of the physical-based model and the sensor data can be provided to another model. The difference between the predicted and measured data may indicate components that have aging, faults, drift, etc. To train additional models, the splitting, training, authentication, selection, testing, and use blocks of System 400 can also be performed using data of different data types. Retraining can also be performed using current simulated sensor data and current measured sensor data. In some embodiments, golden run data can also be supplied for splitting, training, authentication, selection, testing, and use of machine learning or physics-based models.

[0093] Figure 4B is a flowchart of Method 430, which performs TTTM analysis using trace sensor data, according to several embodiments. In some embodiments, the operation of Method 430 can be performed after various machine learning models, statistical models, digital twin models, etc., which are to be used as part of the TTTM analysis module have been trained.

[0094] In block 431, the configuration operation is performed. In some embodiments, the configuration operation is performed by the user. In some embodiments, the user can configure the TTTM system by interacting with a graphical user interface. In some embodiments, the TTTM system can also be configured using other methods such as text input or command-line control. In some embodiments, the configuration settings (e.g., a document or file containing data indicating the configuration settings) can be referenced by a processing device that repeatedly performs the TTTM analysis throughout the analysis.

[0095] Configuration settings may include labeling information and specifications with labels such as date, policy label, product label, equipment and / or sensor label. Configuration settings may include instructions for using data collected by one or more tools, one or more sensors, etc., in relation to one or more processing runs or products. Configuration settings may include instructions to processing devices to perform (or not perform) certain types of analysis. Configuration settings can indicate which metrics to use, what types of summary data to generate, which models (e.g., machine learning models) should be used in the analysis, and which features of the TTTM software module should be used in the analysis (e.g., whether in-lier drift detection should be performed or whether time trace analysis should be performed). Configuration settings may include instructions for conditional thresholds to perform further analysis, such as control limits for classifying data points as outliers, and the number of outlier thresholds to trigger drill-down analysis. Configuration settings may include instructions for aggregating metric values ​​and / or indicators to be performed by the TTTM module. In some embodiments, the user may be able to remember one or more "favorite" configuration settings for use in future TTTM analyses. In some embodiments, the configuration settings may include options for visualization, such as the type of plot to generate (e.g., box plot, histogram, scatter plot, etc.), the number of plots to generate, a threshold metric or indicator value to trigger plot generation, design elements for the generated plots, and whether best fit or trend lines are calculated and / or displayed on the plots.

[0096] After the configuration settings are determined (or the default settings, favorites, etc. are selected), the automated processing procedure 432 is executed. In some embodiments, the TTTM module (guided by the configuration settings) independently performs many operations to generate an analysis of the sensor data. The operations of the automated processing procedure 432 will be discussed in more detail in relation to Figures 5B to 5I, with a brief summary of these operations included herein.

[0097] In block 433, the data acquisition operation is performed by the processing device. According to the configuration settings, the processing device extracts data for analysis. The extracted data may include trace sensor data, pre-processed sensor data, manufacturing data (e.g., hardware or processing parameters, policy information, etc.), summary data, etc.

[0098] In block 434, data processing operations are performed by the processing device. In some embodiments, data processing is performed according to configuration settings. Data processing can include generating metric values, indices, aggregated metrics and indices, trace analysis, drift analysis, etc. Data processing can include providing data (for example, data collected in block 433) to one or more models to generate outputs. Data processing models can include machine learning models, digital twin models, statistical models, etc. Data processing operations will be discussed in more detail with reference to Figure 5D.

[0099] In block 435, the processing logic performs a visualization operation. The visualization operation can be performed according to the configuration settings. In some embodiments, the generation of a visual representation can be triggered when conditions are met during data processing, such as when a metric value exceeds a threshold, or when a particular portion of data points is classified as an outlier. In some embodiments, the visual representation can be generated at multiple analysis stages, for example, the TTTM index values ​​can be plotted, and one or more aggregates of the TTTM index values ​​can also be plotted. Visualization can be performed using any convenient plotting method, such as a box plot, histogram, scatter plot, pie chart, etc. Visual representations will be discussed in more detail in relation to Figures 6A to 6G.

[0100] In block 436, the processing logic performs analysis and reporting operations. In some embodiments, the operations of block 436 are provided to the user. In some embodiments, the operations of block 436 can be triggered automatically (for example, a case evaluation procedure for evaluating the performance of the TTTM module can be performed periodically). In some embodiments, the operations of block 436 can be triggered by the user. In some embodiments, the results of the operations of block 436 can provide configuration settings for future analysis (indicated by an arrow returning to block 431 in Figure 4B).

[0101] Figure 4C is a flowchart illustrating method 450 for generating summary data according to several embodiments. Summary data generation can begin with the collection of trace data. For example, method 450 operates on a single run and all traces from a single sensor. In other embodiments, summary data can also be generated using data from multiple sensors, multiple runs, etc. In some embodiments, trace data can be separated and stored into multiple process operations, and summary data can be generated using the data separated by operations. In some embodiments, preprocessing can be performed on this data, such as smoothing, interpolation, and normalization. Preprocessing can be performed at various analysis stages.

[0102] In block 451, all trace data from a single run is divided into multiple processing operations by processing logic. In some embodiments, all trace data divided into separate processing operations (e.g., steps of a processing strategy) and the data can be provided to the processing logic to perform a basic summary operation. The data can be divided into multiple processing operations based on the timestamp of the data points, the behavior of the trace data, the behavior of the reference trace data, etc. The data divided into multiple processing operations can be provided to a processing device and further divided into transient and steady-state portions. In some embodiments, various operations can be performed in an order different from that shown in Figure 4C, which may be within the scope of this disclosure.

[0103] In block 452, the trace data, which has been divided into multiple processing operations, is further divided into transient and steady-state portions. A visual representation of the transient portion division is shown in Figure 6A. In some embodiments, process parameters such as temperature or voltage may have operating setpoints. When a characteristic value changes rapidly, it may take a certain amount of time to reach such a setpoint. The period of rapid change in the characteristic value can be separated as a transient portion of the trace data, and the period when the characteristic value is more stable can be separated as a steady-state portion of the trace data. In some embodiments, the processing policy may require a characteristic (e.g., temperature) to change (e.g., increase) over a period of time in a controlled form. Such periods that are neither steady-state nor transient can also be separated and analyzed. In some embodiments, gradient data can be analyzed as steady-state data. In some embodiments, gradient data can be analyzed as transient data. In some embodiments, gradient data can be analyzed in a way different from steady-state or transient data. In some embodiments, gradient data can be analyzed by utilizing features from transient data analysis and / or steady-state analysis.

[0104] The processing device can separate the transient and steady-state portions of trace data. In some embodiments, the separation of transient and steady-state portions can be performed according to the configuration settings. Identifying transient portions of trace data may include determining the gradient of the trace data and one or more statistical measures of the data (e.g., standard deviation, range, etc.) based on prior knowledge (e.g., time to steady-state estimate derived from golden run data). In some embodiments, a sliding window is used to separate transient and steady-state portions of trace data. A window smaller than the trace data associated with the operation (e.g., containing fewer data points) can be analyzed. In some embodiments, points within a window can be determined to be part of the steady-state portion or part of the transient portion. In some embodiments, points within a window can be determined to be part of the transient portion if the standard deviation of the points within the window is greater than a threshold. The window can then be moved to include different subsets of data points of the processing operation trace data. The size of the window, the metrics used to determine whether a portion of the trace data is transient (e.g., gradient, standard deviation, quartile values, range, etc.), the threshold at which transientity is determined, etc., can all be performed according to the configuration settings. In some embodiments, the boundary between the portion designated as steady state and the portion designated as transient can be subjected to further processing, such as changing the window size, changing the decision metric, or changing the threshold metric value. The data, divided into transient and steady state (and slope or other type) portions, is provided to the processing logic to generate summary data. In some embodiments, the steady state portion and transient portion are provided to the processing logic to perform a basic data summary operation (block 454). In some embodiments, the transient portion is provided to the processing logic to perform a digital twin summary operation (block 453).

[0105] In block 453, the processing logic performs a digital twin summary operation. In some embodiments, the digital twin summary operation is performed in relation to a temporary portion of trace data. In some embodiments, a digital twin model of a physical asset is generated. The physical asset may include one or more components of a manufacturing system. The digital twin can model the performance of the physical asset, and the modeled behavior can be compared to the behavior recorded by one or more sensors associated with the manufacturing system. A visual representation of the digital twin data for use in generating the summary data can be seen in Figure 6B.

[0106] Summary data can be generated by comparing various features of a temporary portion with the digital twin output. In some systems, characteristic values ​​may exceed the setpoint before settling at the setpoint value (e.g., overshoot). Summary data can be generated using the characteristics of the overshoot region of the trace data (e.g., the overshoot value, the duration of the overshoot, the time it took to reach the maximum or minimum overshoot). In some embodiments, characteristic values ​​may exhibit a rise time to reach the setpoint. By comparing the rise time characteristics of the sensor data with the digital twin data, summary data can be generated, such as the time required for the characteristic value to reach 50% of the indicated value, 90% of the indicated value, etc. In some embodiments, it may take some time for the characteristic value to stabilize at the setpoint (e.g., after an overshoot). This stabilization time can be used to generate summary data (e.g., the time it takes for the value to remain within 5% of the setpoint, within 2% of the setpoint, etc.). In some embodiments, the final characteristic value reached may differ from the setpoint. Summary data can be generated based on this difference. In some embodiments, the characteristics of a closed-loop control system can be modeled using a digital twin model. Summary data can also be generated by monitoring and using characteristics such as gain, phase, and bandwidth of a closed-loop control system. In some embodiments, characteristic values ​​may fluctuate (for example, as the value approaches a setpoint, it may rise or fall above the setpoint several times before stabilizing). The summary data may include detection of such fluctuations and characteristics of the fluctuations, such as frequency and amplitude. In some embodiments, maximum, minimum, slope, average, or other metrics may be included in the summary data associated with the temporal portion.

[0107] In block 454, the basic summary operation is performed by the processing logic. In some embodiments, several parts are provided to the basic summary operation logic (for example, both transient and steady-state data may be provided to the basic summary operation logic). In some embodiments, the basic summary data may include metadata, such as data associated with the processing run. The metadata may include the processing run start time, run ID, tool ID, tool name, lot name, lot ID, processing strategy name, product ID, product information (e.g., product design), product number, product count (e.g., number of products since the last maintenance event), etc.

[0108] In some embodiments, the basic summary data may include contextual data, such as data associated with sensors, processing operations, etc. Contextual data may include channel or sensor name or ID, process operation number (e.g., step number), process operation timestamp, operation name or ID, operation type (e.g., etching operation, deposition operation, etc.), manufacturing subsystem name or ID, etc.

[0109] In some embodiments, the basic summary data may include statistical information associated with the trace data. This statistical information may include data mean, median, minimum, maximum, range, quartiles, and other percentile information, standard deviation, skew, kurtosis, and a flag indicating whether the data is within a control range. The basic summary data can be generated according to configuration settings. In block 455, the summary data is stored for future use, for example, by the TTTM module. The summary data can be stored in a data store, for example, data store 140 in Figure 1.

[0110] Figures 5A to 5I are flowcharts of methods 500A to I associated with generating data and triggering corrective actions according to a particular embodiment. Methods 500A to I can be implemented by processing logic that may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, processing devices, etc.), software (e.g., instructions executed on processing devices, general-purpose computer systems, or dedicated machines), firmware, microcode, or a combination thereof. In some embodiments, methods 500A to I can be implemented in part by a prediction system 110. Method 500A can be implemented in part by a prediction system 110 (e.g., server machine 170 and dataset generator 172 in Figure 1, dataset generator 372 in Figure 3). According to embodiments of the present disclosure, the prediction system 110 can use Method 500A to generate datasets for at least one of training, certifying, or testing a machine learning model. Methods 500B to 500I can be implemented by a prediction server 112 (e.g., prediction component 114). In some embodiments, a non-temporary storage medium stores instructions that cause a processing device (for example, a prediction system 110, a server machine 180, a prediction server 112, etc.) to execute one or more of methods 500A to I when executed by the processing device.

[0111] For the sake of simplicity, methods 500A–I are illustrated and described as a series of operations. However, the operations provided herein can be performed in various orders and / or simultaneously, along with other operations not presented and described herein. Furthermore, not all shown operations are necessarily performed in order to carry out methods 500A–I according to the subject matter disclosed. In addition, it will be understood and recognized by those skilled in the art that methods 500A–I can also be represented, alternatively, through a state diagram or events as a series of correlated states.

[0112] Figure 5A is a flowchart of a method 500A for generating a dataset for a model to generate predictive data (for example, predictive data 168 in Figure 1) according to a specific embodiment.

[0113] Referring to Figure 5A, in some embodiments, in block 501, the processing logic that implements method 500A initializes the training set T to an empty set.

[0114] In block 502, the processing logic generates a first data input (e.g., a first training input, a first authentication input) which may include one or more of the following: sensor data (e.g., past sensor data 144 in Figure 1, past sensor data 344 in Figure 3), manufacturing parameters (e.g., past manufacturing parameters 152 in Figure 1), etc. In some embodiments, the first data input may include a first set of features for the data type, and the second data input may include a second set of features for the data type (e.g., as described with respect to Figure 4A).

[0115] In some embodiments, in block 503, the processing logic generates a first target output for one or more of the data inputs (e.g., a first data input). In some embodiments, the first target output is performance data (e.g., measurement data such as processed products, faulty components). In some embodiments, the input data can be in the form of sensor data, as in the case of a machine learning model configured to run in relation to a physical-based digital twin model, and the target output can be a list of components that are likely to be faulty. In some embodiments, no target output is generated (e.g., an unsupervised machine learning model can group or discover correlations in the input data rather than requiring a target output to be provided).

[0116] In block 504, the processing logic may generate mapping data that shows input / output mappings. The input / output mapping (or mapping data) may refer to data inputs (for example, one or more of the data inputs described herein), target outputs for the data inputs, and associations between the data inputs and target outputs. In some embodiments, block 504 may not be executed in relation to machine learning models, for example, for which no target outputs are provided.

[0117] In block 505, the processing logic, in some embodiments, adds the mapping data generated in block 504 to the dataset T.

[0118] In block 506, the processing logic branches based on whether the dataset T is sufficient for at least one of training, authentication, and / or testing of the model 190. If the dataset T is sufficient, execution proceeds to block 507; otherwise, execution also returns to block 502. Note that in some embodiments, it may be determined that the dataset T is sufficient simply based on the number of inputs in the dataset, and in some embodiments, the number of inputs mapped to outputs; however, in some other embodiments, it may be determined that the dataset T is sufficient based on the number of inputs, or instead, on one or more other criteria (e.g., a measure of the diversity of the data examples, precision, etc.).

[0119] In block 507, the processing logic provides a dataset T (for example, to a server machine 180) for training, authenticating, and / or testing a machine learning model 190. In some embodiments, dataset T is a training set and is provided to the training engine 182 of the server machine 180 to perform training. In some embodiments, dataset T is an authentication set and is provided to the authentication engine 184 of the server machine 180 to perform authentication. In some embodiments, dataset T is a test set and is provided to the test engine 186 of the server machine 180 to perform testing. For example, in the case of a neural network, the input values ​​of a given input / output mapping (for example, a numerical value associated with data input 310) are input to the neural network, and the output values ​​of the input / output mapping (for example, a numerical value associated with target output 320) are stored in the output nodes of the neural network. The weights of the connections in the neural network are then adjusted according to a learning algorithm (for example, backpropagation, etc.), and this procedure is repeated for other input / output mappings in dataset T. After block 507, the model (for example, model 190) can be subjected to at least one of the following: training using the training engine 182 of server machine 180, authentication using the authentication engine 184 of server machine 180, or testing using the test engine 186 of server machine 180. The trained model can then be executed by the prediction component 114 (of the prediction server 112) to generate prediction data 168 for performing corrective actions associated with signal processing or manufacturing equipment 124.

[0120] Figure 5B is a flowchart of a method 500B for determining corrective actions to be taken using one or more models, according to several embodiments. The operation in Figure 5B can be included in automated processing operations described in relation to Figure 4B.

[0121] Referring to Figure 5B, in block 510, the processing logic performs a preparatory operation associated with generating inter-device matching (TTTM) information indicating corrective actions. The preparatory operation can be triggered by manual input (e.g., when suboptimal performance is recognized, in response to suboptimal measurement data for one or more finished products), at scheduled timing intervals, or similar. The TTTM analysis preparation may include data acquisition logic. Data (e.g., sensor trace data) can be retrieved from the data store. Data acquisition can be performed according to configuration settings. The acquired data may include data indicating the date the process was performed, the tool ID of the manufacturing equipment, the policy name, key operations, sensors, or parameters. The configuration settings may include instructions on which of these types of data to collect and the procedures for preprocessing (e.g., data cleaning, resampling, averaging, smoothing, etc.). The processing logic may perform data preprocessing as part of the preparatory operation.

[0122] The preparation operation may further include the processing logic generating summary data associated with, for example, a manufacturing process. The data summarization operation will be discussed in detail in relation to Figure 4C. Data summarization can be performed according to configuration settings. In some embodiments, summary data is generated from trace sensor data. Trace sensor data can be separated into multiple parts, and summary data is extracted separately from each part. For example, trace sensor data can be separated into one or more transient parts and one or more steady-state parts. In some embodiments, a sliding window is used to provide statistics of data values ​​for a portion of the trace. If a particular statistical marker (e.g., standard deviation, variance, etc.) of a point within the window is greater than (or less than) a threshold, it can be determined that the point within that window belongs to a steady-state part of the trace data, a transient part of the trace data, etc. Other metrics (e.g., data gradient, time threshold, etc.) can also be used to specify steady-state and transient parts of the trace data. In some embodiments, steady-state and transient data are processed differently by the summarization logic. Configuration settings may include adjustments to how steady-state and transient parts are identified, such as window size, metric values ​​or identifying information for determining which part a window belongs to, etc. In some embodiments, different data are included in the summary data for different parts of the trace data. In some embodiments, a physically based (e.g., digital twin) model of the component is used to generate predictions of its behavior. In some embodiments, the difference between the digital twin prediction and the measured sensor data is used to generate the summary data. For example, the digital twin model can predict that the voltage supplied to a component (e.g., an electrostatic chuck) may have a specific rise time (e.g., the time to reach 90% of the target value), a characteristic overshoot (e.g., the percentage of the peak voltage relative to the target value), and a stabilization time (e.g., the time to fluctuate until it reaches a steady state).The summarization logic may include recording the difference between the measured temporal portion of sensor data and the prediction of such metrics by a digital twin model.

[0123] Summary data can include metadata, contextual data, statistics, and digital twin-assisted summary data. Metadata can include data describing the manufacturing process (e.g., the state in which a semiconductor wafer has been processed). Metadata can include run start time, policy ID, tool ID associated with manufacturing equipment, tool name, lot name and / or ID, product ID, product information, product number, and the number of products produced by the tool. Contextual data can include data describing trace data such as sensor ID, policy operation number, timestamp data, subsystem, and operation type. Statistical data can include mean, minimum and maximum values, percentile (e.g., quartile) values, variance, skew, kurtosis, range, interquartile range, control lower and control upper limits (e.g., calculated from other statistics such as the mean of the interquartile range outside the percentile range or three standard deviations of some multiple). Digital twin-assisted summary data may include overshoot (e.g., the percentage difference between the characteristic's target value and its peak value), rise time (e.g., the time it takes for the characteristic to adjust to 90% of the target value), stabilization time (e.g., the time it takes for the measured value to fall within some threshold range of the target value), steady-state error (e.g., the percentage difference between the measured final value and the target value), closed-loop control system parameters such as gain margin (e.g., the difference in gain between no gain and the system gain at the frequency where the phase offset between input and output is 180°), phase margin (e.g., the difference in phase offset between the system phase and 180° at the frequency with no gain), and bandwidth, variation detection, variation frequency, variation amplitude, etc.

[0124] In block 511, the processing logic checks for one or more existing models that fit the data to be analyzed. Existing models can be models trained using historical data. The suitability of existing models can be determined based on factors such as being trained on the same or similar processes with the same set of manufacturing equipment and the same type of manufacturing equipment. If no existing suitable models exist, the flow proceeds to block 512. If an existing suitable model exists, the flow proceeds to block 513.

[0125] In block 512, the processing logic performs training operations associated with performing TTTM analysis. The training operations will be discussed in more detail in relation to Figure 5C. The training operations may include retrieving data associated with one or more manufacturing processes from datastore 140 in Figure 1, for example. The training operations may include retrieving and / or generating golden runner data, retrieving summary data, retrieving trace data, etc. The training operations can be performed according to the configuration settings.

[0126] In block 513, the processing logic performs the inference operation associated with the TTTM analysis. The inference operation will be discussed in more detail in relation to Figure 5D. The inference may include an analysis using one or more models. The models may include physical-based models, digital twin models, machine learning models, etc. The inference may be performed according to the configuration settings. The inference operation may be performed in response to a previous inference operation; for example, a further drill-down analysis may be performed in response to the output of a previous operation meeting one or more criteria. The results of the inference may include corrective actions for further follow-up, such as scheduled maintenance, or instructions for the system or components. The inference operation may include visualization of the results.

[0127] In block 514, the processing logic performs a case evaluation operation associated with the TTTM analysis. The case evaluation procedure can be used to generate information demonstrating the performance of the TTTM analysis procedure. The performance of various models and algorithms can be evaluated. The case evaluation operation may include comparing predicted performance metrics with measured performance metrics. In some embodiments, the case evaluation can be triggered by a user. In some embodiments, the case evaluation can be triggered by results, such as TTTM analysis results outside a threshold range, or measurements of similarity to TTTM analysis predictions outside a threshold range. In some embodiments, the case evaluation can be performed according to a case evaluation schedule (e.g., every n processes, at a selected frequency). Data associated with the manufacturing process (e.g., sensor data, summary data) can be provided to the processing logic along with the output of the TTTM analysis (e.g., one or more components with predicted failures, subsystems with predicted drift or aging, predicted measurement data of the finished product, visualization of the health of components or subsystems, etc.). Measured performance data associated with the TTTM analysis output (e.g., measured product measurements resulting from maintenance of predicted faulty components) can also be provided to the processing logic that performs the case assessment.

[0128] The case evaluation operation may include, and / or be subject to, determining whether a TTTM module (e.g., one or more models included in a TTTM analysis) should be retrained. The processing logic may evaluate whether to perform retraining based on one or more of several metrics. In some embodiments, retraining may be performed based on a particular portion of the predictions, e.g., a portion of the predictions, failing to meet an accuracy threshold over a period of time. In some embodiments, retraining may be performed based on a series of consecutive products failing to meet an accuracy threshold. In some embodiments, retraining may be performed based on a known change to the processing state, such as a policy update, an update of incoming materials, or chamber maintenance performed. The processing logic may perform the evaluation using data from cases with sufficiently accurate TTTM predictions and cases with unsatisfactory TTTM predictions. The processing logic may perform the evaluation using both positive cases (cases where the TTTM analysis predicted failure, aging, drift, etc., e.g., true positives and false positives) and negative cases (cases where the TTTM analysis did not predict failure, e.g., true negatives and false negatives).

[0129] The processing logic can further evaluate false cases (e.g., false positives, false negatives). False cases can be evaluated in terms of their root causes, for example, in terms of possible modifications to the TTTM model to reduce false cases. In some embodiments, the effect of adjusting the weights of behaviors, sensors, etc., can be evaluated. In some embodiments, the effect of adjusting control limits can be evaluated. In some embodiments, the effect of removing sensors, behaviors, etc., from the TTTM modeling in the future can be evaluated. The processing logic can store the results of the case evaluation. The processing logic can generate recommendations, for example, to update or retrain all or part of the TTTM analysis module.

[0130] In block 515, the processing logic performs retraining of the TTTM module. Retraining can be performed according to recommendations generated during case evaluation in block 514. For example, the processing logic may generate recommendations to retrain sensor weights during the case evaluation operation. During the retraining operation, the processing logic may implement the recommendations by providing additional data to train one or more models to be used for the TTTM analysis. In some embodiments, retraining may include model validation and / or testing. In some embodiments, the retraining operation may include training a new model using different sensor data (e.g., a different set of sensors as described in relation to Figure 4A). In some embodiments (e.g., for recommendations generated during the case evaluation operation), the retraining operation may not be performed. After retraining, the flow can return to the inference stage (block 513) for additional products, processes, manufacturing equipment, etc., for the TTTM analysis.

[0131] Figure 5C is a flowchart showing a method 500C for performing training operations of a TTTM analysis method according to several embodiments. These operations may focus on preparing the TTTM analysis for inference, rather than training a specific machine learning model, a physics-based model, etc. In block 521, data is retrieved for use in the TTTM analysis. The retrieved data may include trace data, pre-processed trace data, summarized data, etc. The retrieved data may include data that has been identified, generated, stored, etc., in the preparation operations for the TTTM analysis. Data can be retrieved according to configuration settings, such as settings indicating which analytical techniques, which trained models, etc., should be used in the TTTM analysis.

[0132] In block 522, the processing logic determines whether one or more golden datasets should be used in the TTTM analysis. Golden data can include trace data associated with well-manufactured products, properly functioning equipment, etc., cleaned or pre-processed trace data, summary data, etc. The use of one or more golden datasets can be based on configuration settings, etc., and one or more models should be used as part of the TTTM analysis, etc., based on the configuration settings. If the golden datasets should not be used, the flow proceeds to block 526. If one or more golden datasets should be used, the flow proceeds to block 523. In block 523, the processing logic determines whether one or more golden datasets are available (for example, data designated as golden data is stored in datastore 140). If golden data (for example, golden data for all sensors, processes, systems, etc., for which golden data should be used) is available, the flow proceeds to block 525, and one or more golden datasets are retrieved. If the golden datasets are not available (for example, one or more golden datasets that should be used in the TTTM analysis are not available), the flow proceeds to block 524.

[0133] In block 524, the processing logic generates one or more golden datasets. The processing logic can select data to act as one or more golden datasets (for example, from datastore 140). Data can be selected as golden data based on one or more indications that the data corresponds to acceptable processing parameters (for example, processing parameters that facilitate the production of products within manufacturing tolerances). In some embodiments, a set of recently certified (e.g., recently maintained, recently installed, etc.) tools, chambers, and / or manufacturing equipment can be designated as golden tools. Data from processes performed by golden tools can be designated as golden data. In some embodiments, data indicating acceptable product production can be associated with sensor data (e.g., measurement data, classification, or quality data, etc.). Sensor data associated with acceptable product production (including pre-processed sensor data, summarized sensor data, etc.) can be designated as golden run data. In some embodiments, one or more sensors may have associated golden run data. In some embodiments, one or more processing operations may have associated golden run data.

[0134] In block 526, the processing logic defines the control limits. The control limits can be set according to the configuration settings. In some embodiments, the control limits can be established by user input (for example, the user can select the acceptable processing range). In some embodiments, the control limits can be calculated by the processing logic (for example, three standard deviations from the mean). In some embodiments, a composite system can also be used (for example, three standard deviations from the mean, but within user-defined limits).

[0135] Figure 5D is a flowchart showing a method 500D for performing the inference operation of a TTTM analysis method according to several embodiments. In block 530, one or more models to be used in the TTTM analysis are imported. These can include trained machine learning models, physically based models, digital twin models, statistical models, etc. Models can be imported according to the configuration settings.

[0136] In block 531, the processing logic performs an outlier detection operation. By comparing the summary data to the control limits, it is possible to determine whether a data point (e.g., summary data associated with a set of trace data) is an outlier. Each sensor in each processing operation (e.g., policy step) can be considered individually. A data point can be considered an outlier if any one summary data is outside the control limits, if multiple summary data are outside the control limits, or if the overall weight of the summary data (e.g., a higher weight is given to a summary with greater predictive ability) is outside the control limits. In some embodiments, individual outlier signals can be aggregated to generate an index (e.g., a TTTM index) that indicates the performance and / or quality of a chamber, tool, process, equipment, product, set of manufacturing equipment, etc.

[0137] In some embodiments, the generation of the TTTM index may depend on the weights of the sensor and process operation, as well as control limits. Some sensor-operation combinations may be found to have higher predictive capabilities than others. For example, in the case of processing operations of semiconductor substrates utilizing plasma, a sensor indicating plasma parameters may have high predictive capabilities for chamber performance. In the case of processing operations where plasma processing is not used, the same sensor may not provide information that is readily correlated with tool performance. Weighting coefficients may indicate the importance of the contribution of associated data to performance.

[0138] In some embodiments, data points classified as high outliers (e.g., above the control limit), low outliers (e.g., below the control lower limit), and inliers (e.g., between the control upper and lower limits) can be processed in different ways. In some embodiments, one or more weighting coefficients can be applied to sensor data, summary data, etc. The weighting coefficients (e.g., weights) can indicate the impact that characteristic values ​​measured in the processing chamber have on performance. In some embodiments, weights can be applied to sensors (e.g., sensors closer to the substrate can produce data more closely correlated with performance). In some embodiments, weights can be applied to processing operations (e.g., sensor data during etching operations can correlate more closely with the performance of the etching strategy than non-etching operations such as warm-up or cool-down operations). In some embodiments, weights can be applied to processing runs (e.g., processing runs with known problems can be given lower weights). In some embodiments, weighting coefficients can be applied to data types (for example, sensor types such as temperature sensors, summary data types such as means may be given a weight greater than the standard deviation, summary data from temporal parts may be given a different weight than steady-state parts of trace data, and so on). In some embodiments, weighting coefficients can be applied to combinations of states, such as specific sensors during specific operations (for example, a sensor that measures the voltage applied to generate plasma may produce results that correlate more highly with the performance of processing operations involving plasma, but not as highly with the performance in operations without plasma).

[0139] In some embodiments, weights can be used to generate the TTTM index. In some embodiments, weights can be selected so that the sum of weights from all sensors can be normalized (e.g., equal to 1) for each processing operation (e.g., processing step). Outlier sensors contribute zero to the TTTM index value for an operation, while inliers may contribute to their weights. If all sensors are inliers, the TTTM index can be 1 (or scaled to a different value). A TTTM index value lower than 1 (or a scaled value) may indicate that further analysis of the manufacturing process, operation, tools, equipment, etc., is recommended.

[0140] In some embodiments, a weight discovery model can assign (or assist in assigning weights). In some embodiments, the weight discovery model can include a machine learning model. The machine learning model can be trained using sensor-associated data (e.g., summary data) as training input and performance data (e.g., measurement data) as the target output. Training the machine learning model adjusts the weights that connect the input data to the target output data. The machine learning model can output weights that connect action / step combinations to performance. The weights can be used in TTTM analysis, for example, as the output of a machine learning model adjusted by the user. In some embodiments, a partial least squares model can be used to generate the weight function. In some embodiments, a stacked partial least squares model can be used (e.g., multiple partial least squares models arranged as an ensemble model).

[0141] In block 532, the processing logic performs the operation of sensor contribution calculation. In some embodiments, the performance of the operation in blocks 532-535 may depend on the value of the TTTM metric. For example, the processing logic may be configured to perform further analysis (e.g., drill-down analysis to explore root causes) if the TTTM metric meets a threshold (e.g., falls below a threshold established in the configuration settings). In block 532, sensor contribution information can be generated indicating "how" an outlier one or more data points are. The previous operation includes labeling data points if they are within the control limits. The sensor contribution value indicates how far outside the control limits an outlier is. An exemplary sensor contribution metric is the severity distance. The severity distance is a measure of how far outside the control limits a data point (e.g., summary data point) is. The severity distance shows the relationship between a data point (or aggregate of data points) and the associated control limits. In some embodiments, the severity distance can be generated for one or more sensor / behavior data points for a manufacturing run (e.g., a single product). Severity distance can be based on one or more summary data (e.g., one or more types of summary data). In some embodiments, the calculation of severity distance can include a calculation of how far one or more data points are outside the control limits. In some embodiments, the calculation of severity distance can include a calculation of how far a data point is outside the control limits relative to the size of the control range (for example, a temperature sensor may be assigned a control range of 20 to 40 degrees, and a data point at 50 degrees can be calculated to be 50% of the size of the control range above the upper control limit). For example, a data point below the lower control limit (e.g., below the control range) may have a severity distance calculated as follows:

[0142]

number

[0143] Aggregated metric values ​​or indices can be generated based on one or more severity distances. An example of an aggregated metric is a severity score. A severity score may include an aggregation of severity distances across multiple process runs (e.g., several products manufactured). A severity score may indicate the likelihood that a particular sensor / operation pair (e.g., a sensor that measures the voltage applied to generate plasma in a plasma etching operation) will produce data indicating chamber faults, chamber drift, etc., across several runs. The calculation of aggregated severity distances, such as a severity score, may include a coefficient indicating the frequency of outlier data points. For example, a severity score may be calculated as follows:

[0144]

number

[0145] In some embodiments, metric values ​​can be further aggregated to support analysis. For example, the processing logic can aggregate severity scores (corresponding to a single sensor and a single action in some embodiments) across policy actions to discover a severity index per sensor (e.g., by adding the product of action weights and severity scores across process actions). Such analysis can be triggered by severity scores reaching a threshold (e.g., quality index, quality score, severity score, etc. that meet a criterion), the number or portion of severity scores that meet one or more thresholds, etc. Trigger states for aggregating data to generate new indices can be stored as configuration settings. By aggregating severity scores in this way, additional information can be provided to problematic or faulty sensors. By adding across sensors, similar processes can be run to determine whether one or more actions in a manufacturing process are being performed suboptimally. In some embodiments, the TTTM index (which in some embodiments is associated with a single run or product and a single processing operation) can be aggregated to generate a TTTM index per run by adding weights for the product (e.g., per run, per operation) and operations across multiple operations (e.g., warm-up, etching, deposition, and cool-down operations) of the TTTM index, for example, to generate a TTTM index per run. This can help determine whether a particular operation in a manufacturing process is being performed suboptimally. A similar procedure can be performed, for example, to isolate a failure originating from a single run (e.g., a single semiconductor wafer). The processing logic can use various aggregates of the index to recommend that a component or subsystem should be maintained or replaced, and this can be fed as input to a machine learning model to generate recommendations for corrective actions, etc. Corrective actions may include scheduling preventive maintenance, updating processing policies, sending alarms to the user, etc. In some embodiments, clustering analysis can be performed on the index and / or metric values. Clustering analysis will be discussed in more detail in relation to Figure 5H.The visualization of clustering will be discussed in more detail in relation to block 535.

[0146] In block 533, the processing logic performs in-lie drift detection. The method for in-lie drift detection is described in detail with reference to Figure 5I. TTTM index analysis, severity distance, severity score, etc., can be associated with outlier data. Inlier data (e.g., data within control limits) can also provide valuable information for analysis. Inlier analysis can be performed using summary data. In some embodiments, summary data from a run (e.g., one product such as one semiconductor wafer) can be compared with golden summary data (e.g., summary data records during the production of a product that meets a performance threshold). In some embodiments, summary data from a series of manufacturing runs (e.g., processing over a day, a week, a month, etc.) can be compared with golden summary data from a series of manufacturing runs.

[0147] The comparison between the golden data and the data to be analyzed can be performed using any technique for comparing datasets. In some embodiments, techniques for quantifying time series similarity can be utilized. One metric that can be used is to preprocess and / or wash the data so that each data point in the dataset to be analyzed has a corresponding data point in the golden run data (e.g., data with the same timestamp, data associated with the same processing operation, etc.). The differences between the values ​​of the corresponding data points can then be aggregated (e.g., by adding squares, absolute values, etc.). The metric can be further processed, for example, by taking the square root, which is obtained from the added squares and divided by the number of data points for normalization, and so on. The higher the value of the metric, the more it indicates that the two datasets are different. In some embodiments, a discrete Fourier transform can be performed on both datasets, and similarly, the corresponding Fourier coefficients can be compared to the above analysis comparing the corresponding data points. In some embodiments, an autoregressive model can be used to compare datasets. An autoregressive model is a representative example of a random process, where the values ​​of the data points in the process depend on the values ​​of previous data points. An autoregressive model can be used to determine the likelihood that the dataset to be analyzed and the golden dataset are associated with the same process (for example, that processing conditions, chamber quality, etc., are the same between the two sets).

[0148] In some embodiments, an elastic difference metric can be used to determine the similarity between two datasets. In some embodiments, dynamic time warping can be used in in-lieured drift analysis. Dynamic time warping works by aligning or warping datasets in the time domain so that the accumulated cost of alignment is minimized. The cost function can be determined according to the configuration settings. The elastic difference metric can be thought of as adjusting the spacing between data points in one dataset to better align with a second dataset. The elastic difference metric adjusts only the spacing between data points, not the order of the data points. The elastic difference metric can return a metric or index value indicating how much spacing adjustment is performed to make the two datasets optimally aligned (determined, for example, by minimizing the cost function). In some embodiments, after elastic warping, each point in the first dataset can be associated with a point in the second dataset. For example, the data to be analyzed can be warped to optimally align with the golden data. After warping, each point in the data to be analyzed can be assigned a “nearest neighbor” point in the golden run dataset (for example, some points in the analyzed set can be assigned the same nearest neighbor point in the golden dataset). In some embodiments, each data point in both datasets can be spliced ​​together with one or more nearest neighbor points in the other dataset. For example, each point in the warped dataset being analyzed can be assigned a nearest neighbor point in the golden dataset. This assignment may include some points in the golden dataset being associated with multiple points in the dataset to be analyzed. The assignment may also include some points in the golden dataset not being assigned as nearest neighbors to some points in the analyzed dataset. Then, each data point in the golden set can be assigned a nearest neighbor point in the temporally warped dataset to be analyzed.If the two datasets are aligned (for example, the warping cost function is minimized when each data point 1...n in the dataset to be analyzed is its nearest neighbor to the corresponding point 1...n in the golden dataset), the number of nearest neighbor associations can be equal to the number of data points in the set. If the two datasets are significantly out of alignment, one or more points in each dataset may have several points designated as nearest neighbors in the other dataset. In this case, the number of nearest neighbor associations can be greater than the number of data points in the set. The increased number of associations can be used as a metric to indicate how well warping produces optimal alignment of the two datasets (in some embodiments, this involves preprocessing such as normalization or subtraction of the number of data points). In-lie drift detection and elastic difference measures will be discussed further in relation to Figures 5I and 6D-6E.

[0149] In some embodiments, each tool, chamber, set of manufacturing equipment, etc., may operate in a somewhat different manner. Each tool, chamber, set of equipment, etc., may have its own associated one or more golden datasets. Each type of process (e.g., each target product design, each policy, etc.) may have its own associated one or more golden datasets.

[0150] In the implementation of corrective actions, in-lie drift detection behavior can be used. In some embodiments, an in-lie drift detection metric exceeding a specific threshold (depending on configuration settings in some embodiments) can trigger a corrective action. Corrective actions may include providing an alert to the user, scheduling maintenance, or updating policies.

[0151] In block 534, processing logic allows additional features of the TTTM module to be utilized for further analysis. The use of additional features can be subject to configuration settings. In some embodiments, various metric values ​​can be used when determining whether additional features should be utilized. Additional features can assist in drill-down analysis to isolate causes such as chamber failures, drift, unexpected results, and insufficient product quality. Additional features can be used to determine corrective actions that should be taken. Additional features may include providing data associated with the manufacturing process to one or more models (e.g., machine learning, physics-based, etc.) to provide additional analysis.

[0152] In some embodiments, additional features of the TTTM module may include trace data analysis, such as a trace data exploration analysis module. Trace data analysis may include the use of a trained model (e.g., a machine learning model). In some embodiments, trace data analysis can be performed by providing trace data to a trained machine learning model and receiving output from the machine learning model indicating corrective actions, equipment components to be maintained or replaced, faults within the manufacturing equipment subsystem, etc. In some embodiments, an autoencoder model can be used to perform trace data analysis. Autoencoder operation will be discussed in more detail in relation to Figure 7. Examples of visual representations associated with the results of trace data analysis will be discussed in more detail in relation to Figures 6F-6G.

[0153] Trace data analysis can include data preparation operations, such as data cleaning, reformatting, normalization, and interpolation. The trace data can then be fed into a trained model. The trained model can generate output instructions for sensors, subsystems, products, etc., that are not performing as expected. In some embodiments, trace data associated with many (e.g., all) sensors associated with a single product processed by manufacturing equipment can be fed into the trained model. A set of data associated with a set of products can be fed into the trained model. Based on the trace data analysis of the trace sensor data, the model can generate data indicating suboptimal predicted product performance, measurements, characteristics, etc.

[0154] In block 535, the processing logic executes the results visualization operation. Metrics, analyses, etc., performed or generated by the TTTM analysis process can be incorporated into one or more visualization steps, and in some embodiments, the processing logic can execute the visualization operation according to the configuration settings. Indicators, metrics, etc., can be visualized in any convenient or appropriate form, such as scatter plots, histograms, box plots, line graphs, etc.

[0155] In some embodiments, data can be clustered for visualization. In some embodiments, severity distance can be used for clustering and visualization of outliers. Data from multiple runs and multiple sensors can be used for clustering and visualization. In some cases, the majority of sensor data is inlier data (e.g., within control limits). Data can be preprocessed to remove extraneous data (e.g., all inlier data can be removed). In some embodiments, dimensionality reduction can be applied to simplify the remaining outlier data. Dimensionality reduction can assist in data visualization and, for example, can reduce data to two or three dimensions for more favorable analysis. In some embodiments, a nonlinear dimensionality reduction model can be used. A nonlinear dimensionality reduction model can capture correlations that linear dimensionality reduction models (e.g., principal component analysis) do not capture. A dimensionality reduction model can translate affinity or similarity between data points into the probability that data points are close to each other in the reduced dimensional space (e.g., by Euclidean distance, Minkowski distance, correlation distance, or another distance function). In some embodiments, the dimensionality reduction model can include a t-distribution type stochastic nearest neighbor embedding algorithm. In some embodiments, parameters that control the behavior of the dimensionality reduction model can be stored as configuration settings such as perplexity (a metric that determines the expected number of nearest neighbors within a cluster) and distance metrics in high-dimensional space.

[0156] In some embodiments, dimensionality-reduced data can be clustered using a clustering model. In some embodiments, a clustering model that self-selects the number of clusters in the dataset can be selected. In some embodiments, density-based spatial clustering of an application with a noise clustering algorithm can be utilized. Parameters that determine the details of how the clustering model works can be stored as configuration settings (e.g., metrics that affect clustering distance, cluster size, distance metric, etc.).

[0157] In some embodiments, data points (e.g., clustered and dimensionally reduced outlier points) may have a contribution to a ranked severity score (e.g., the severity distance of sensors contributing to the severity score associated with an outlier point). Clusters can be labeled using the sensors that contribute most to the cluster (e.g., sensors represented by multiple data points within a cluster, sensors contributing to one cluster but not another). In some embodiments, sensors and associated data can be assigned to subsystems of manufacturing equipment (e.g., heater power subsystem, chamber temperature subsystem, chamber pressure subsystem, etc.). Clusters can be labeled by the subsystem that contributes most severely to the outlier state of the points associated with the cluster. In this way, clusters can be grouped by their root cause (e.g., subsystem failure). Outlier clustering can be used in implementing corrective actions, such as providing alerts to users, scheduling maintenance, and adjusting process policies.

[0158] The processing logic can generate visual representations related to in-lie drift detection. The results of the elastic difference scale model can be visualized. Changes can be represented by scatter plots, bar graphs, line graphs, etc., the relationships between data points can be plotted on a plot showing the relationship between dataset 1 and dataset 2, or other methods can be used to visualize the in-lie drift detection data.

[0159] The processing logic can generate a visual representation associated with anomaly trace detection. In some embodiments, sensors can be grouped into subsystems, and metrics describing deviations from expected sensor results can be aggregated into subsystem metrics. Subsystem analysis metrics per subsystem and per run (product manufactured) can be displayed, for example, as a heatmap.

[0160] In some embodiments, visual representations can be provided to the user as part of a graphical user interface. In some embodiments, additional data can be made available through user interaction with the graphical user interface. In some embodiments, additional analysis can be performed based on the user's interaction with the graphical user interface. For example, a heatmap displaying the results of anomaly trace detection analysis may be based on only a single metric (e.g., per subsystem, per run), but when selecting data points in the visual representation, more data can be displayed (e.g., details of the difference between analyzed trace data and expected trace data, details of the output of a machine learning model, details of recommended corrective actions, etc.). In another example, a visualization tool may display representations of multiple TTTM metrics for various tools, products, etc. The user can select a particular representation (e.g., one displaced from other points in a group) and perform additional analysis (e.g., in-lie drift detection, anomaly trace detection, etc.) on the data associated with that data point.

[0161] Figure 5E is a flowchart of method 500E, in several embodiments, of performing corrective actions based on metrics indicating the performance of a processing chamber. In block 540, the processing logic receives trace sensor data associated with a first manufacturing process of the manufacturing chamber. The trace sensor data may be collected by the manufacturing chamber during the processing of a product such as a semiconductor wafer.

[0162] In block 542, the processing logic processes the trace sensor data to generate summary data associated with the trace sensor data. In some embodiments, generating summary data includes identifying at least one steady-state portion and at least one transient portion of the trace sensor data. The transient portion includes a set of trace data associated with a period in which the characteristic value in the chamber changes (e.g., the variance is above a threshold). The steady-state portion includes a set of trace data associated with a period in which the characteristic value measured by the sensor does not change (e.g., the variance is below a threshold). In some embodiments, a sliding window is used to distinguish the steady-state portion from the transient portion. In some embodiments, determining whether a set of data points (e.g., points in a window) of the trace sensor data belongs to a transient portion or a steady-state portion may include determining whether the statistical measure (e.g., standard deviation) of the set of data points meets a threshold. The transient portion may include a set of points where the standard deviation is above a threshold, and the steady-state portion may include a set of points where the standard deviation is below a threshold. Determining whether a set of points belongs to a transient portion or a steady-state portion may include considering the statistical measures of adjacent windows (e.g., overlapping sets) of the data points. In some embodiments, the processing strategy includes multiple processing operations. In some embodiments, trace data from sensors of a process operation (e.g., one or more process operations of the process strategy) may include a transient beginning portion (e.g., when the state in the manufacturing chamber is adjusted to a set point), a steady-state portion (e.g., when the state is maintained at the set point), and a transient end portion.

[0163] In some embodiments, summary data is generated separately for each part (e.g., each steady-state part, each transient part, etc.). The summary data for the steady-state part may include a statistical description of the data, such as mean, maximum, minimum, range, standard deviation, kurtosis, etc. The summary data for the transient part may also include a statistical description. The summary data for the transient part may further include metrics generated considering the output of the digital twin model. A digital twin model can be generated of one or more components of a manufacturing chamber associated with one or more sensors (e.g., a sensor can measure the power supplied to a component, and a digital twin model of the component and the power supply for the component can be constructed). The digital twin model can take information indicating a processing policy (e.g., setpoints, near-field characteristic values, etc.) as input and produce time-trace data predicting sensor readings as output. In some embodiments, the output of the digital twin model (e.g., simulated sensor data) can be compared with measured sensor data. In some embodiments, the difference between the features of the simulated sensor data and the features of the measured sensor data (e.g., a difference greater than a threshold) may be included in the transient part summary data. The characteristics of the temporary data may include metrics such as overshoot relative to the target or final value, rise time, stabilization time, parameters of detected variation, and the difference between the setpoint and the final value.

[0164] In block 544, quality index scores are generated based on summary data. The quality index scores can be associated with a particular run (e.g., a product manufactured by a manufacturing machine), process operation, sensor, or any combination of these features (for example, a quality index score can be associated with one sensor and one operation across multiple runs, or a quality index score can be associated with a group of sensors such as all temperature sensors, and so on). In some embodiments, quality indices are generated that indicate the quality of runs and process operations. In some embodiments, the generated quality indices can be used to take measures to increase the consistency of process parameters such as inter-run, inter-chamber, etc., e.g., inter-machine matching (TTTM).

[0165] In some embodiments, quality metrics are generated by considering one or more classification states. In some embodiments, data (e.g., summary data) can be classified as inlier data or outlier data. In some embodiments, data can be classified as inlier data, low outlier data (e.g., below the control lower limit), or high outlier data (e.g., above the control upper limit). Classification can be performed by comparing data values ​​to control upper and lower limits, for example, to determine whether data values ​​are within the control range or whether data values ​​meet thresholds. Control limits can be assigned by the user and can be statistical in nature (e.g., three standard deviations away from the mean of historical data), and so on. In some embodiments, inliers do not contribute to the quality metric score, and the quality metric can be based solely on outliers.

[0166] In some embodiments, a quality index is generated by considering multiple weighting factors. These weighting factors can represent the data's contribution to performance, for example, to the characteristics of the finished product. For instance, in some processes, data from one sensor may correlate more strongly to performance than data from another sensor. In such processes, the quality index score calculation can assign a smaller weight to data from a first sensor that does not correlate as strongly to product performance than to data from a second sensor that does correlate more strongly to product performance. The weighting factors can be associated with sensors, process operation, sensor / operation pairs (e.g., a sensor measuring the voltage supplied to a plasma generator during plasma operation might be given a larger weight, while a sensor measuring the voltage supplied to the plasma generator might be given a smaller or zero weight during operation without plasma), and data characteristics (e.g., one aspect of summary data might correlate more strongly to product performance than another, or part of trace data might correlate more strongly to product performance than another). In some embodiments, the weighting factors can be extracted from a trained machine learning model. The machine learning model can be trained to correlate process data (e.g., sensor data, summary sensor data, etc.) with performance data (e.g., measurement data). A machine learning model can be configured to map one or more relationships between sensor data and measurement data. Weighting coefficients can be extracted from a machine learning model (e.g., a stacked partial least squares model), determined by considering factors extracted from the machine learning model, calculated based on data including those extracted from the machine learning model (in addition to process knowledge), and so on.

[0167] In some embodiments, quality metric scores can be generated as an aggregation of calculated metrics. For example, a TTTM metric score can be generated by aggregating data associated with many sensors (e.g., via a weighted sum) to calculate a quality metric score that represents the quality of one operation in one processing run (e.g., one product). Aggregations of other metrics can also be applied to the TTTM process, for example, by aggregating many runs to show process quality over time, or by aggregating across operations and sensors to provide an approximate quality value for a processing run. The processing logic can generate many metrics to provide information (e.g., to the user, further processing logic, etc.) associated with metric values ​​that can indicate problems associated with the manufacturing equipment (e.g., all metric values ​​that meet a threshold, a specific percentage or number of metric values ​​that are most likely to be associated with a problem or failure). Using the metric values ​​that indicate a problem or failure, further analysis, such as drill-down analysis, can be initiated to determine the root cause, corrective actions that should be taken, etc.

[0168] In some embodiments, the quality index score may include information associated with the severity of an outlier, such as how far a data point is outside the control range. In some embodiments, a severity distance may be assigned to a data point. Calculating the severity distance (e.g., a severity distance score) may include calculating how far a data point is outside the control range, for example, as a fraction of the size of the control range (for example, an outlier below the lower control limit may have a severity distance calculated using the ratio between how far below the lower control limit the data point is and the difference between the upper and lower control limits). Control limits and control ranges may be associated with sensors, process operations, chambers, product designs, etc. The calculation of the severity distance may include weighting factors. The severity distance may be associated with sensors, process operations, and processing runs (e.g., products).

[0169] In some embodiments, severity distances can be aggregated (e.g., into a severity score). The aggregated severity distance score can be used in drill-down analysis. For example, generating a severity score (e.g., aggregated severity distance) may involve adding severity distances for multiple runs (e.g., to generate a severity score associated with sensor and process operations). In some embodiments, the severity score may include factors related to how many points are outliers, such as the frequency (e.g., percentage) of outlier points per sensor and process operation.

[0170] In some embodiments, one or more metrics indicating quality (e.g., TTTM index, severity score, etc.) can be provided to the processing logic for visualization. In some embodiments, data points (e.g., severity distance) can be provided to a dimensionality reduction model. In some embodiments, the dimensionality reduction model can be a nonlinear model such as a t-distribution type stochastic nearest neighbor embedding model. In some embodiments, data points (e.g., dimensionality reduction severity distance) can be provided to a clustering model. In some embodiments, the clustering model can be a nonlinear clustering model such as density-based spatial clustering of applications by a noise model. In some embodiments, data points of clusters can be assigned to subsystems (e.g., based on the top-level contribution of the data points to the severity score). In some embodiments, each cluster can be assigned the root cause of a subsystem of the manufacturing equipment.

[0171] In block 546, the processing logic provides the user with an alert based on the quality metric score, the alert including an indication that the manufacturing chamber performance does not meet a first threshold. In some embodiments, a metric (such as a TTTM metric score) can be provided to the user. In some embodiments, a visual representation of the metric (e.g., a bar graph, histogram, box plot) can be provided to the user. In some embodiments, the user can use the visual representation to initiate, be notified of, or continue a drill-down analysis, for example, to identify the root cause of a failure or corrective actions to be taken.

[0172] Figure 5F is a flowchart of method 500F for performing in-lier analysis according to several embodiments. In one embodiment, method 500F is performed after method 500E has been performed on data for which no outliers were identified. In block 550, the processing logic receives first data associated with measurements obtained by sensors during a first manufacturing procedure of a manufacturing chamber. In some embodiments, the data may include summary data. The data may include in-lier data points, which are points that satisfy a criterion (e.g., a threshold). In block 552, the processing logic receives second data. The second data includes reference data associated with the first data. The reference data may be data associated with a process that produces an acceptable product, a process in which an acceptable state is achieved, etc. The reference data may be called golden data. The reference data may be associated with the same process, sensor, chamber, equipment, etc., as first data.

[0173] In block 554, the first data and reference data (e.g., golden run data) are provided to the comparison model. In some embodiments, the comparison model is an elastic difference model, for example, a model that manipulates the space between data points in two datasets without changing the order of the data points in those datasets in order to achieve the closest possible match. In some embodiments, the comparison model can be a dynamic time warping model. In some embodiments, data (e.g., summary data) over multiple processing runs (e.g., days, weeks, or more product manufacturing) can be provided to the comparison model. A metric can be provided that shows how different the measured data is from the golden data (e.g., by an elastic difference model). For example, the model can show the best match between each data point in both datasets and the data points in the other set. If a point exists in one dataset, two identical datasets can have the same number of matched pairs. Two sets that are not sufficiently matched can have a larger number of pairs. The metric can relate to the number of matched pairs between the two sets, for example, the ratio of data points to matched pairs.

[0174] In block 556, the processing logic receives similarity scores from the comparison model, associated with the first data and the reference data. In some embodiments, the similarity scores relate to multiple matching pairs between the two datasets. In block 558, the processing logic takes the similarity scores into account and performs corrective actions. Corrective actions may include sending an alert to the user. The alert may include a visual representation, such as a visual representation of multiple similarity scores, or a visual representation of the nearest point matching between the two sets of data. Corrective actions may include scheduling preventive or corrective maintenance. Corrective actions may include updating process policies.

[0175] Figure 5G is a flowchart of method 500G for performing trace sensor data analysis according to several embodiments. In block 560, the processing logic receives trace sensor data associated with a first manufacturing process in the processing chamber. In some embodiments, the processing logic may provide trace data in response to values ​​of one or more metrics that satisfy a threshold. For example, the TTTM metric may satisfy the threshold. In response to the metric satisfying the threshold (e.g., meeting a criterion), the trace data associated with the TTTM metric (e.g., trace data from sensors with high severity scores) can be further analyzed. In some embodiments, the processing logic performs preprocessing on the trace data. Preprocessing may include smoothing, cleaning (e.g., resizing, removal of empty data, etc.), interpolation, etc.

[0176] In block 562, the processing logic processes trace sensor data using one or more trained machine learning models. The trained machine learning models are trained to generate a representation of the trace sensor data (e.g., a dimensionally reduced representation). One or more trained machine learning models are trained to reconstruct the sensor data based on the representation of the trace sensor data. One or more trained machine learning models output the reconstructed data. In some embodiments, the trace sensor data can be provided to an autoencoder model. In some embodiments, the trace sensor data can be provided to a trained encoder model to generate a representation of the trace sensor data. This representation can then be provided to a trained decoder model to generate reconstructed sensor data. The autoencoder model can be an encoder-decoder model. In some embodiments, the machine learning models can be trained by providing the model with data (e.g., preprocessed sensor trace data). The model can be configured to generate a representation of the input data (e.g., a dimensionally reduced representation). The model can be configured to generate reconstructed data from the representation of the input data. The model can then be trained by minimizing the difference between the input data and the reconstructed data (depending on the configuration settings, for example, to avoid making changes to the trace sensor data to generate the representation).

[0177] In block 564, the processing logic compares the trace sensor data with the reconstructed sensor data. In some embodiments, the machine learning model is trained using data that meets a threshold, such as data within control limits, data where a specified portion is within control limits, data associated with a process in which a product meeting the performance threshold was manufactured, golden run data, etc. The trace sensor data provided to the trained machine learning model may or may not be associated with irregular processing runs, such as processing runs with data outside the control limits.

[0178] In block 566, the processing logic determines one or more differences between the reconstructed sensor data and the trace sensor data. The differences can be determined based on comparisons performed in relation to block 564. In some embodiments, the trace sensor data may differ in some way from the training data, for example, the trace sensor data may be associated with a processing run that did not produce an acceptable product, or the trace sensor data may be outside one or more control limits, and so on. Differences between the training set and the trace sensor data may cause the trace sensor data and the reconstructed data to differ significantly. Reconstruction errors can be used to determine whether the input data deviates significantly from the range of the training data. Since the machine learning model is trained using data from a healthy tool (e.g., a tool with a relatively new part that produces products that meet quality standards), the machine learning model produces an accurate reproduction of the trace data when the trace data is from a healthy tool. However, when the reconstructed data differs from the trace data, this indicates that the trace data deviates in some way from that of a healthy tool. In some embodiments, reconstruction errors (e.g., errors for each data point in the trace data) can be aggregated to generate a metric of errors associated with, for example, a sensor and a processing run (e.g., a single product).

[0179] In block 568, the processing logic determines whether to recommend corrective actions associated with the processing chamber. The determination may be based on one or more differences between trace sensor data and reconstructed sensor data. The determination may be based on aggregated error metrics. In some embodiments, sensors can be associated with subsystems. Error metrics can be further aggregated in subsystems (e.g., as root mean square error). In some embodiments, a visual representation can be generated. The visual representation may be organized by runs, subsystems, sensors, etc.

[0180] Figure 5H is a flowchart of method 500H for performing outlier clustering analysis according to several embodiments. Details of triggering and processing outlier clustering can follow configuration settings. In some embodiments, outlier clustering can be triggered by multiple outlier points in previous analyses that meet a threshold. In block 570, sensor contribution values ​​are extracted by processing logic. The calculation of sensor contributions is discussed in more detail in relation to block 532 in Figure 5D. In some embodiments, each sensor contribution metric used for clustering (e.g., severity index, severity distance, severity score, etc.) can be associated with a single run and sensor. In some embodiments, the clustering method can be used to analyze multiple data points (e.g., from multiple sensors across multiple processing runs). As a large number of sensors are associated with the tool, this data can occupy a high-dimensional space. In some embodiments, the majority of the data is inlier data, e.g., data within control limits. In some embodiments, the sensor contribution value of an inlier can be defined as 0. In some embodiments, only data points with sensor contribution values ​​that meet a threshold can be used for clustering analysis.

[0181] In block 572, data preprocessing for clustering analysis is performed. Data preprocessing may include, for example, washing the dataset to include only runs with sensors exhibiting outlier behavior. Other preprocessing, such as normalization, may also be performed. In block 574, dimensionality reduction operations are performed by the processing device. In some embodiments, high-dimensional datasets can be reduced to a more easily visualized space for increased analytical power, such as to accommodate clustering models. In some embodiments, the dataset can be reduced to a two-dimensional dataset. In some embodiments, the algorithm is likely to make data points that are close to each other in the high-dimensional space also close to each other in the low-dimensional space. Proximity can be determined by Euclidean distance, cosine distance, or other distance metrics. In some embodiments, a nonlinear dimensionality reduction model can be used. In some embodiments, a t-distribution type stochastic nearest neighbor embedding method can be used for dimensionality reduction. Dimensionality reduction can be performed according to configuration settings, such as distance metrics, the expected number of nearest neighbors, etc.

[0182] In block 576, the clustering operation is performed by the processing device. The clustering operation can be performed on data reconstructed in a low-dimensional space. In some embodiments, a clustering model that self-selects the number of clusters in the data can be utilized. In some embodiments, density-based spatial clustering of applications by a noise method can be utilized for the clustering operation. Clustering can identify the number of sets of data points that are represented close to each other in a low-dimensional space, which can indicate a high probability that the data points are represented close to each other in a high-dimensional space. Such data points are likely to be related to the same root cause. The clustering operation can be performed according to configuration settings. Configuration settings may include a distance metric, a minimum number of samples in a cluster, and a maximum distance between two samples to be considered neighbors.

[0183] In block 578, the processing logic ranks the contributions of the sensor systems to one or more clusters, according to several embodiments. The data points of a cluster can be analyzed by examining the sensor contribution values ​​that contribute to the data points. In some embodiments, the sensors associated with the data points can be ordered by their sensor contribution values ​​(e.g., severity distance). In some embodiments, the sensors with the highest contributions (e.g., the three sensors with the highest contribution values) can be used to assign primary subsystems of the manufacturing system to data points, clusters, etc. In some embodiments, the processing logic can visualize the clustering results. Visualization of outlier clustering is discussed in more detail in relation to block 535 in Figure 5D and Figure 6C.

[0184] Figure 5I is a flowchart of Method 500I for performing in-lie drift detection according to several embodiments. In-lie drift detection can be triggered and / or performed according to configuration settings. Some operations of the TTTM analysis module utilize outliefer data points (e.g., TTTM indices, sensor contribution / severity metrics, etc.). In some embodiments, analysis of inliefer data points (e.g., data points within control limits) can be used to determine system failures, drift, corrective actions to be taken, etc.

[0185] In block 580, reference data for in-lie drift detection is set. In some embodiments, in-lie drift detection includes comparing data (e.g., data collected over a period of time) with reference data. In some embodiments, a golden dataset is provided or selected. The golden dataset may include data from processing runs associated with products manufactured within specifications. The golden dataset may include data from processing runs performed immediately after successful tool installation or preventive maintenance.

[0186] In block 582, warp pathfinding is performed. In some embodiments, in-layer drift detection involves the use of an elastic difference model. Such a model modifies the space between data points in a dataset (without changing the order of the data points) to generate the best fit between two sets, measured by adding, for example, the differences (or interpolations of data values) between the two sets. In some embodiments, a dynamic time warping algorithm is applied to a reference dataset and a dataset to be analyzed. In some embodiments, the dataset includes summary data. In some embodiments, the dataset includes summary data collected over a period of time, for example, over many processing runs. The warp path defines how to stretch one dataset (e.g., the data to be analyzed) to fit the pattern of the other dataset (e.g., the golden data).

[0187] In block 584, the warp point count is determined. The warp point count is a metric that indicates how much a dataset must be warped to provide the best fit with another dataset. Other metrics can also be used to indicate the degree of warping, such as an aggregated measure of how much data points in one set must be shifted. The warp point count is a measure of how many matched pairs exist at comparison points between two datasets. In some embodiments, each point in the target dataset (e.g., the dataset being analyzed) is matched to one “nearest neighbor” point in the reference dataset (e.g., the golden dataset). In some embodiments, each point in the reference dataset is matched to one nearest neighbor point in the target dataset. If the two datasets are sufficiently aligned, each data point in one set can be matched to one other data point in the other set, and the number of matches will be equal to the number of data points. If the two datasets are not sufficiently aligned, some data points in one or both sets may be matched to multiple points in the other set, and the number of matches may be greater than the number of data points. The matching point count can be extended to situations where the reference data and target data have different numbers of data points. In some embodiments, each data point has at least one matching point, so the matching point count can be used to subtract the number of data points (e.g., the number of guaranteed matches) and leave only the number of additional matches due to warping of a single dataset, e.g., the warp point count. In block 586, the warp rate is calculated by the processing logic. The warp rate is a comparison that shows how much the dataset is warped overall (e.g., compensated for the size of the dataset). In some embodiments, the warp rate can be calculated as follows:

[0188]

number

[0189] Figures 6A to 6G show exemplary visual representations associated with aspects of TTTM analysis according to several embodiments. Figure 6A shows a visual representation 600A of the operation of dividing the trace into transient and steady-state parts according to several embodiments. Visual representation 600A includes a time trace 602. The time trace 602 can be associated with one sensor and one processing operation (for example, the entire runtime trace has already been divided by the operation divider 451 in Figure 4C to generate the time trace 602). A window 604 is used to determine whether a set of data points (e.g., data points in a window) belongs to the transient part of the data or the steady-state part. The metric values ​​of the points in the window can be used to classify the points in the window as transient or steady-state. In some embodiments, the points in the window are determined to be transient if the standard deviation of the points in the window exceeds a threshold.

[0190] In some embodiments, the window 604 is moved through the time trace, as indicated by an arrow and dashed window to the right of the trace 602, to classify the data points of the trace as transient or steady state. In some embodiments, the time trace 602 can be separated into two or more parts. In some embodiments, the time trace 602 can be separated into a transient beginning portion 606, a steady state portion 607, and a transient end portion 608.

[0191] Figure 6B shows a visual representation of the digital twin output used in the summary data generation operation in several embodiments. One or more components of a manufacturing machine can be modeled by the digital twin. The digital twin includes a digital reproduction of one or more physical assets. The digital twin can be used to model a system response (for example, controlling adjustments to satisfy a setpoint). Various features of the digital twin response can be compared with sensor data to generate summary data.

[0192] Visual representation 600B includes a time trace 610. In some embodiments, the time trace 610 can be associated with a time period of processing operation. The digital twin model may include a target setpoint 611 associated with characteristic values ​​measured by sensors in a physical manufacturing system. A feature of the digital twin time trace is an overshoot 612. The time trace 610 may exceed the setpoint (or, in some embodiments, fall below the setpoint) before settling at the setpoint value. One or more of the overshoot 612 values ​​(or normalized or otherwise modified) or the peak time 613 can be used to generate summary data. In some embodiments, the rise time 614 can be used to generate summary data. In some embodiments, the rise time 614 can be measured for the first time when the time trace 610 meets the setpoint 611. In some embodiments, the rise time 614 can be measured for the first time when the time trace 610 reaches a threshold (for example, 80% of the difference between the setpoint and the starting value, as shown in visual representation 600B).

[0193] In some embodiments, summary data can be generated using the duration of a variation, such as the variation 615. Summary data can be generated using characteristics of the variation, such as frequency and amplitude. Summary data can be generated using the stabilization time 617 (for example, the time the time trace 610 remains within the error window 616 of the setpoint 611). In some embodiments, summary data can be generated using the setpoint error (for example, the difference between the final value of the time trace 610 and the setpoint 611).

[0194] Figure 6C shows a visual representation of the results of clustering operations, such as those discussed in relation to Figure 5H, according to several embodiments. Outlier data points (e.g., data points exhibiting unusual or suboptimal performance) can be used in the clustering operation. The clustering operation may target the identification of the root cause (e.g., subsystem) that is responsible for the outlier data (e.g., suboptimal performance of manufacturing equipment). In some embodiments, high-dimensional sensor data is transformed into a lower-dimensional space, for example, by a dimensionality reduction model. In some embodiments, the high-dimensional space is reduced to a two-dimensional space. In some embodiments, a dimensionality reduction algorithm brings points that were close to each other in the high-dimensional space (e.g., by Euclidean distance, cosine distance, etc.) closer to each other in the lower-dimensional space. In some embodiments, data points having the same (or related) root cause (e.g., subsystem that caused the data point to be an outlier) can be represented in close proximity to each other in the lower-dimensional space. In some embodiments, subsystems can be assigned to clusters using the sensor contribution to the data point.

[0195] Visual representation 600C shows outlier points in low-dimensional space. Using a clustering algorithm (e.g., a density-based spatial clustering application), data points can be separated into groups that are likely related to the same root cause. The exemplary visual representation 600C is separated into four clusters 640, 641, 642, and 643. Each cluster can be labeled with a manufacturing subsystem using the apex contributing sensor (e.g., the sensor with the highest severity metric). For example, cluster 640 can be seen as associated with the heater power subsystem, cluster 642 as associated with electrostatic chuck temperature, cluster 643 as associated with gas flow rate in some part of the manufacturing system, and so on.

[0196] Figures 6D–6E show visual representations of the results of an in-lyer drift detection elastic difference model in several embodiments. Figure 6D shows the results of the elastic difference model 600D applied to a target time trace 650 and a reference time trace 651. Figures 6D (and 6E) show the original sequential data, e.g., summary data collected over multiple processing runs, and the results of warping (e.g., application of the elastic difference model) are indicated by dashed lines showing pairs of matched points. Result 600D is generated from two fairly similar traces. Only a few points in one trace match several points in the other trace, as highlighted by the circles, and in each case, the number of matched points is small. The target trace 650 and the reference trace 651 are temporally similar and are indicated by fairly ordered (e.g., mostly parallel) dashed lines.

[0197] Figure 6E shows the results of the elastic variance model 600E applied to a target time trace 660 and a reference time trace 661 that differ from those shown in Figure 6D. Numerous metrics can be used to quantify the degree of warping in the target trace 660, which can, for example, trigger corrective actions. Result 600E includes several data points aligned with a large number of points in the other trace. One way to indicate how much warping is associated with a pair of time traces involves comparing the number of alignments that should exist in a well-aligned pair of traces with the number of alignments present in the analysis. Other methods include characterizing the aggregated displacement of alignments over time (e.g., how much the dashed line in Figure 6E is stretched from left to right), indications of the gradient of the alignment line, etc.

[0198] Figures 6F to 6G show exemplary visual representations 600F and 600G of the results of trace data analysis according to several embodiments. Figure 6F includes a heatmap 670 and a legend 671. Reconstruction errors of the autoencoder model may be used to generate the heatmap 670. In some embodiments, reconstruction errors (e.g., relating to the difference between input data and reconstructed data at each timestamp of the time trace) can be aggregated into a single error number per trace. In some embodiments, reconstruction errors can be further aggregated by a sensor subsystem. In some embodiments, a single value can be generated for one processing run and one manufacturing subsystem. The heatmap 670 can generate multiple subsystems (e.g., subsystems A, B, C, and D such as a vacuum subsystem, a temperature subsystem, and a chucking subsystem) and multiple processing runs (e.g., fabricated substrates). The legend 671 can visually indicate subsystem / run pairs with high reconstruction errors, e.g., subsystem / run pairs that are likely to have performed suboptimally. Exemplary visual representation 600F shows a particularly high reconstruction error associated with run number 5, subsystem B. In some embodiments, the visual representation 600F can be generated as a corrective action and can be considered, for example, as an alert provided to the user. In some embodiments, a visual representation such as the visual representation 600F can be used to perform a corrective action, for example, the visual representation 600F can indicate that maintenance is appropriate for subsystem B.

[0199] Figure 6G shows an exemplary visual representation 600G of the output from an autoencoder model associated with trace data analysis, according to several embodiments. Visual representation 600G includes the input data pattern 680 and the reconstructed data pattern 681. In some embodiments, visual representation 600G can be generated (or displayed to the user) in response to the detection of a high reconstruction error, for example, in response to the detection of a high reconstruction error within a heatmap visual representation such as Figure 6F. In some embodiments, the user can choose to view more detail in a portion of the visual representation associated with the time trace analysis (for example, by clicking on a portion of the heatmap 670 associated with subsystem B and processing run 5) to display a more detailed figure (e.g., visual representation 600G). In some embodiments, a high reconstruction error, a large difference between the input trace data pattern 680 and the reconstructed data pattern 681, can be associated with suboptimal performance, as shown, for example, in visual representation 600G.

[0200] Figure 7 illustrates the operation of a model 700 (e.g., a machine learning model), such as an autoencoder model, which is capable of reducing the dimensionality of input data, according to several embodiments.

[0201] The input data 710 of Model 700 is data associated with the fabrication of the substrate. In some embodiments, the input data 710 includes one or more of the following: measurement data, manufacturing parameters, sensor data, or a combination thereof. The input data 710 may be pre-processed data. In some embodiments, the input data 710 is measurement data associated with the substrate. The measurement data may be of any (or many) types, including thickness, in-plane displacement, chemical properties, electronic properties, optical properties, etc. In some embodiments, the input data 710 includes trace sensor data and / or a pre-processed version of the trace sensor data associated with the manufacturing of the product.

[0202] Model 700 includes a first part 720 (e.g., an encoder) and a second part 740 (e.g., a decoder). In some embodiments, the model is one or more of the following: an autoencoder, a neural network model, a convolutional neural network model, a deep belief network, a forward neural network, a multilayer neural network, etc. The first part 720 reduces the dimensionality of the input data 710 (e.g., measurement data) to a compressed form (e.g., compressed data 730). Multiple hidden layers (two of which are shown in Figure 7, but any number of hidden layers can be used) can separate the input layer of the machine learning model from the compressed data. During training of the machine learning model 700, the first part 720 can generate one or more functions to fit the input data 710 to a lower-dimensional representation without user guidance. The reduction (e.g., compression, encoding) can be performed in several stages (i.e., first converting the input data 710 to partially compressed data, then further converting it to compressed data 730), or the reduction (e.g., compression, encoding) can be performed in a single stage.

[0203] The second part 740 takes compressed data 730 as input and produces output data 750. During training, model 700 is trained to minimize the difference between input data 710 and output data 750, where output 750 is a reconstruction of input data 710 from compressed data 730. The minimization function used to train model 700 can also have a penalty for the dimensionality of compressed data 730 strengthened to avoid reverting to a function with insufficient compression (e.g., a discriminant function that perfectly reproduces input data 710 but does not compress the data to a reduced number of dimensions). Model 700 can be trained so that the output data 750 of model 700 closely matches the input data 710.

[0204] The functions used by the first part 720 and the second part 740 can be inherently nonlinear. In some applications, all processes of model 700 (i.e., both reduction and reconstruction, both encoding and decoding, etc.) can be used. In other applications, only some capabilities can be used. For example, during training, model 700 can pass input data 710 through the first part 720 to form compressed data 730, then through the second part 740 to determine output data 750, and then compare the output data 750 with the input data 710 to determine that the output data 750 is substantially similar to the input data 710.

[0205] In some embodiments, Model 700 can be used for anomaly trace analysis. Model 700 can be trained on historical data. The historical training data can be limited to successful runs, such as runs where the characteristics of the manufactured product were within a threshold range, or runs where the achieved chamber state was within a threshold range. Model 700 can then be configured to generate compressed data for future success of manufacturing runs. In some embodiments, Model 700 can be trained and used on data associated with one (or a limited number) of sets, such as equipment, policies, and product designs.

[0206] Model 700 may produce output data 750 with some reconstruction errors (e.g., differences from input data 710). In some embodiments, input data 710 that is in some way similar to the training input used to train Model 700 may produce output data 750 with relatively low reconstruction errors. A reconstruction error metric (e.g., sum of errors between data points, squared errors, normalized sums, root mean square errors, etc.) can be generated to represent a metric of the total reconstruction error of the dataset input to Model 700. In some embodiments, the magnitude of the error metric may indicate the similarity between the input data 710 and the data used to train Model 700. In some embodiments, a high reconstruction error (e.g., a large difference exists between the output data 750 and the reconstructed data generated from the compressed data 730) can indicate that the input data 710 is significantly different from the data used to train the model 700. In some embodiments, the model 700 may be strictly trained on "good" data (e.g., data associated with manufacturing runs that meet a performance threshold), and the reconstruction error metric may indicate that the input data 710 is suboptimal. The reconstruction error can be calculated per run (e.g., per manufactured product), per sensor, per tool, per subsystem, etc., and can be visualized in a convenient format to facilitate the implementation of corrective actions.

[0207] Figure 8 is a block diagram showing a computer system 800 according to a particular embodiment. In some embodiments, the computer system 800 can be connected to other computer systems (for example, via a network such as a local area network (LAN), intranet, extranet, or internet). The computer system 800 can operate in a client-server environment with the capacity of a server or client computer, or it can operate as a peer computer in a peer-to-peer or distributed network environment. The computer system 800 can be provided by a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular telephone, web device, server, network router, switch, or bridge, or any device capable of executing (sequentially or otherwise) a set of instructions specifying actions to be taken by such device. Furthermore, the term “computer” includes any group of computers that individually or collectively execute one or more sets of instructions in order to perform any one or more of the methods described herein.

[0208] In a further embodiment, the computer system 800 may include a processing device 802, a volatile memory 804 (e.g., random access memory (RAM)), a non-volatile memory 806 (e.g., read-only memory (ROM) or electrically erasable programmable ROM (EEPROM)), and a data storage device 818, which can communicate with each other via a bus 808.

[0209] The processing device 802 can be provided by one or more processors, such as a general-purpose processor (e.g., a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor that implements other types of instruction sets, or a microprocessor that implements a combination of instruction set types) or a specialized processor (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0210] The computer system 800 may further include a network interface device 822 (for example, connected to network 874). The computer system 800 may also include a video display unit 810 (for example, an LCD), an alphanumeric input device 812 (for example, a keyboard), a cursor control device 814 (for example, a mouse), and a signal generation device 820.

[0211] In some embodiments, the data storage device 818 may include a non-temporary computer-readable storage medium 824 (e.g., a non-temporary machine-readable medium) capable of storing instructions 826 that encode one or more of the methods or functions described herein, the instructions 826 of which encode components of Figure 1 (e.g., prediction component 114, corrective action component 122, model 190, etc.) and instructions for carrying out the methods described herein.

[0212] Instruction 826 may also reside fully or partially in volatile memory 804 and / or processing device 802 while it is being executed by computer system 800, so that volatile memory 804 and processing device 802 may also constitute a machine-readable storage medium.

[0213] In the descriptive example, computer-readable storage medium 824 is shown as a single medium, but the term “computer-readable storage medium” includes a single or multiple mediums that store one or more sets of executable instructions (e.g., a centralized or distributed database, and / or associated caches and servers). The term “computer-readable storage medium” also includes any tangible medium capable of storing or encoding a set of instructions that cause a computer to execute one or more of the methods described herein for execution by a computer. The term “computer-readable storage medium” also includes, but is not limited to, solid memory, optical media, and magnetic media.

[0214] The methods, components, and features described herein can be implemented by individual hardware components or integrated into the functionality of other hardware components such as ASICS, FPGAs, DSPs, or similar devices. In addition, the methods, components, and features can be implemented by firmware modules or functional circuits within hardware devices. Furthermore, the methods, components, and features can be implemented in any combination of hardware devices and computer program components, or within computer programs.

[0215] Unless otherwise specifically stated, terms such as “receive,” “execute,” “provide,” “acquire,” “cause,” “access,” “determine,” “add,” “use,” “train,” “reduce,” “generate,” and “correct” refer to actions and processes performed or implemented by a computer system that manipulate and convert data represented as physical (electronic) quantities in computer system registers and memory into other data similarly represented as physical quantities in computer system memory or registers or other such information storage, transmission, or display devices. Furthermore, in this specification, terms such as “first,” “second,” “third,” and “fourth” are meant to be labels for distinguishing different elements and do not necessarily have an orderly meaning according to numerical indication.

[0216] The examples described herein also refer to apparatus for performing the methods described herein. Such apparatus may be specifically constructed to perform the methods described herein, or may include a general-purpose computer system selectively programmed by a computer program stored within the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0217] The methods and examples described herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used in accordance with the teachings described herein, or it may be advantageous to construct more specialized devices to perform each of the methods and / or individual functions, routines, subroutines, or operations described herein. Examples of structures for various such systems are described above.

[0218] The above description is intended to be illustrative, not restrictive. While this disclosure has been described by reference to specific examples and embodiments, it should be understood that this disclosure is not limited to the examples and embodiments described. The scope of this disclosure should be defined with reference to the following claims, along with the entire scope of equivalents given thereto.

Claims

1. A method for determining a malfunction in manufacturing equipment, Receiving sensor data during the first manufacturing procedure of the manufacturing chamber, The process involves generating summary data based on the aforementioned sensor data, Identifying the steady-state portion of the sensor data and generating a first portion of the summary data based on the steady-state portion, The generation of summary data includes identifying a temporary portion of the sensor data and generating a second portion of the summary data based at least partially on one or more differences between the temporary portion and predicted data related to the temporary portion, Receiving reference data, wherein the reference data includes data associated with the sensor data. The summary data and the reference data are provided to the comparison model, The comparison model receives similarity scores associated with the summary data and the reference data, To implement corrective measures taking into consideration the aforementioned similarity score. A method that includes this.

2. The method according to claim 1, wherein generating a first portion of the summary data based on the steady-state portion includes calculating statistical data from the steady-state portion.

3. The method according to claim 1, wherein the predicted data associated with the temporary portion includes the output of a digital twin model of one or more components of the manufacturing chamber.

4. The method according to claim 1, wherein the comparison model includes a dynamic time warping model.

5. The method according to claim 4, wherein the similarity score is based on the sum of the number of matching pairs of points between the summary data and the reference data, determined by the dynamic time warping model.

6. The method according to claim 1, wherein the value of the summary data is within the control range.

7. The aforementioned corrective measures are, Sending an alert to the user, Schedule preventive maintenance, or The method according to claim 1, comprising one or more of the following: scheduling corrective maintenance.

8. A system for determining a fault in manufacturing equipment, comprising a memory and a processing device coupled to the memory, wherein the processing device is Receiving sensor data during the first manufacturing procedure of the manufacturing chamber, The process involves generating summary data based on the aforementioned sensor data, Identifying the steady-state portion of the sensor data and generating a first portion of the summary data based on the steady-state portion, The generation of summary data includes identifying a temporary portion of the sensor data and generating a second portion of the summary data based at least partially on one or more differences between the temporary portion and predicted data related to the temporary portion, Receiving reference data, wherein the reference data includes data associated with the sensor data. The summary data and the reference data are provided to the comparison model, The comparison model receives similarity scores associated with the summary data and the reference data, A system configured to take corrective action in consideration of the aforementioned similarity score.

9. The system according to claim 8, wherein generating a first portion of the summary data based on the steady-state portion includes calculating statistical data from the steady-state portion.

10. The system according to claim 9, wherein the predicted data associated with the temporary portion includes the output of a digital twin model of one or more components of the manufacturing chamber.

11. The system according to claim 8, wherein the comparison model includes a dynamic time warping model.

12. The system according to claim 11, wherein the similarity score is based on the sum of the number of matching pairs ss of points between the summary data and the reference data, determined by the dynamic time warping model.

13. The system according to claim 8, wherein the value of the summary data is within the control range.

14. The aforementioned corrective measures are, Sending an alert to the user, Schedule preventive maintenance, or The system according to claim 8, comprising one or more of the following: scheduling corrective maintenance.

15. A non-temporary machine-readable storage medium that stores instructions for causing a processing device to perform an action when executed, in order to determine a fault in manufacturing equipment, wherein the action is Receiving sensor data during the first manufacturing procedure of the manufacturing chamber, The process involves generating summary data based on the aforementioned sensor data, Identifying the steady-state portion of the sensor data and generating a first portion of the summary data based on the steady-state portion, The generation of summary data includes identifying a temporary portion of the sensor data and generating a second portion of the summary data based at least partially on one or more differences between the temporary portion and predicted data related to the temporary portion, Receiving reference data, wherein the reference data includes data associated with the sensor data. The summary data and the reference data are provided to the comparison model, The comparison model receives similarity scores associated with the summary data and the reference data, A non-temporary machine-readable storage medium, which includes taking corrective action in consideration of the similarity score.

16. The non-temporary machine-readable storage medium according to claim 15, wherein generating a first portion of the summary data based on the steady-state portion includes calculating statistical data from the steady-state portion.

17. The non-temporary machine-readable storage medium according to claim 15, wherein the value of the summary data is within the control range.

Images