Method for equipment matching by diagnosis for manufacturing equipment

Abstract

The method includes receiving trace sensor data associated with a first manufacturing process of a manufacturing chamber. The method further includes processing the trace sensor data with a processing device to generate summary data associated with the trace sensor data. The method further includes generating a quality index score based on the summary data. The method further includes providing an alert to a user based on the quality index score. The alert includes an indication that the manufacturing chamber performance does not meet a first threshold.

Description

[Technical field]

[0001] The present disclosure relates to diagnostic methods for determining faults in manufacturing equipment and, in some embodiments, to an inter-machine matching method for determining faults in manufacturing equipment. [Background technology]

[0002] A product can be created by running one or more manufacturing processes using manufacturing equipment. For example, a semiconductor manufacturing equipment can be used to create a substrate through a semiconductor manufacturing process. A product should be created that has specific properties that are suitable for a target application. Understanding and controlling the properties in the manufacturing chamber aids in the consistent creation of the product. Summary of the Invention

[0003] The following is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an exhaustive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure, nor to delineate the scope of any particular embodiments of the disclosure or the scope of the claims. The sole purpose of this summary is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description presented later.

[0004] In one aspect of the disclosure, a method includes receiving trace sensor data associated with a first manufacturing process of a manufacturing chamber. The method further includes processing the trace sensor data with a processing device to generate summary data associated with the trace sensor data. The method further includes generating a quality index score based on the summary data. The method further includes providing an alert to a user based on the quality index score. The alert includes an indication that the manufacturing chamber performance does not meet a first threshold.

[0005] In another aspect of the disclosure, a system is enabled that includes a memory and a processing device coupled to the memory. The processing device is configured to receive trace sensor data associated with a first manufacturing process of a manufacturing chamber. The processing device is further configured to process the trace sensor data to generate summary data associated with the trace sensor data. The processing device is further configured to receive one or more criteria for classifying the summary data to classify the one or more sensor data as outlier sensor data. The processing device is further configured to generate a quality index score based on the outlier summary data. The processing device is further configured to provide an alert to a user based on the quality index score. The alert includes an indication that the manufacturing chamber performance does not meet a first threshold.

[0006] In another aspect of the disclosure, a non-transitory machine-readable storage medium is enabled. The medium stores instructions that, when executed, cause a processing device to perform operations. The operations include receiving trace sensor data associated with a first manufacturing process of a manufacturing chamber. The operations further include processing the trace sensor data to generate summary data associated with the trace sensor data. The operations further include generating a quality index score based on the summary data. The operations further include providing an alert to a user based on the quality index score. The alert includes an indication that the manufacturing chamber performance does not meet a first threshold.

[0007] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example system architecture, according to some embodiments. [Diagram 2] 1 is a cross-sectional view of a fabrication chamber according to some embodiments. [Diagram 3]FIG. 1 is a diagram of a dataset generator for creating datasets for training, testing, and validating models, according to some embodiments. [Figure 4A] FIG. 1 is a block diagram illustrating a system for generating output data according to some embodiments. [Figure 4B] 1 is a flow diagram of a method for performing device-to-device matching analysis using trace sensor data, according to some embodiments. [Figure 4C] 1 is a flow diagram of a method for generating summary data according to some embodiments. [Figure 5A] 1 is a flow diagram of a method for generating a dataset for a model for generating predictive data, according to some embodiments. [Figure 5B] 1 is a flow diagram of a method for utilizing one or more models to perform corrective actions, according to some embodiments. [Figure 5C] 1 is a flow diagram illustrating a method for performing a training operation of a device-to-device matching analysis, according to some embodiments. [Figure 5D] 1 is a flow diagram illustrating a method for performing inference operations for device-to-device matching analysis, according to some embodiments. [Figure 5E] 5 is a flow diagram of a method 500E for performing corrective action based on a metric indicative of performance of a processing chamber, according to some embodiments. [Figure 5F] 1 is a flow diagram of a method for performing inlier analysis, according to some embodiments. [Figure 5G] 1 is a flow diagram of a method for performing trace sensor data analysis, according to some embodiments. [Figure 5H] 1 is a flow diagram of a method for performing outlier clustering analysis according to some embodiments. [Figure 5I] 1 is a flow diagram of a method for performing inlier drift detection according to some embodiments. [Figure 6A] FIG. 2 illustrates a visual representation of splitting a trace into transient and steady-state portions in accordance with some embodiments. [Figure 6B] FIG. 1 illustrates a visual representation of a digital twin output used in a summary data generation operation, according to some embodiments. [Figure 6C] FIG. 1 illustrates a visual representation of a clustering operation result, according to some embodiments. [Figure 6D] FIG. 13 illustrates the results of an elastic dissimilarity model applied to a reference time trace and a similar target time trace. [Figure 6E] FIG. 13 shows the results of an elastic dissimilarity model applied to a reference time trace and a dissimilarity target time trace. [Figure 6F] FIG. 1 illustrates an exemplary visual representation of the results of trace data analysis, according to some embodiments. [Figure 6G] FIG. 1 illustrates an exemplary visual representation of the results of trace data analysis, according to some embodiments. [Figure 7] FIG. 1 is an illustration of the operation of a model capable of reducing the dimensionality of input data, according to some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating a computer system according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Described herein are techniques related to a tool-to-tool matching (TTTM) diagnostic method that can be used to diagnose problems and / or take corrective action in manufacturing equipment. Manufacturing equipment can be used to create products such as substrates (e.g., wafers, semiconductors, displays, photovoltaic devices, etc.). Manufacturing equipment (e.g., manufacturing tools) often include a manufacturing chamber that isolates the substrates being processed from the environment. The properties of the substrates being created should meet target property values ​​to facilitate performance, functionality, etc. Manufacturing parameters are selected to create substrates that meet the target property values. Manufacturing parameters can include process parameters such as temperature, gas flow rates, plasma conditions, pressure, and hardware parameters such as component use, component settings, component locations, etc. A manufacturing system controls such parameters by specifying set points for the property values ​​and receiving data from one or more sensors located in the manufacturing chamber. The manufacturing system adjusts the set points of various components until the sensor readings match (within a tolerance threshold range) the set points.

[0010] A manufacturing system may exhibit or exhibit characteristics that result in suboptimal performance, for example, due to drift, aging, or failure of components, improperly performed maintenance, changes in target process parameters, etc. Sensor data collected during the manufacturing process can convey information indicative of drifting, aging, malfunctioning, or failed subsystems and / or components. A manufacturing system may include numerous sensors, and in some systems, may include tens or even hundreds to thousands of sensors. In some systems, sensors may collect data over time (e.g., trace data) during a manufacturing procedure. A single time trace (e.g., associated with one sensor and one processing run, or one manufactured product) may include hundreds of data points, thousands of data points, or more.

[0011] Ensuring that conditions are consistent, e.g., consistent run-to-run, chamber-to-chamber, etc., within a single chamber, can increase the consistency, performance, quality, etc., of the products made. Tool-to-Tool Matching (TTTM) is a process targeted at ensuring consistency. In some cases, sensor data is used as a measure of process conditions. In conventional systems, it can be inconvenient to utilize sensor data for diagnostics and corrective actions. In some systems, sensor data can be used to identify the root cause of suboptimal performance of a tool. Suboptimal performance may be detected by performing measurements on a finished product, e.g., via random quality control, sparse sampling, etc. In response to a measurement of suboptimal performance (e.g., a product outside of manufacturing tolerances), the tool's sensor data can be investigated. The sensor data can be utilized to identify tool subsystems, components, etc. that have aged, drifted, failed, malfunctioned, etc. that contribute to the tool's suboptimal performance. Identifying the sensor data of interest can include comparing trace data from many different process operations and many different sensors, and can include comparisons to past data as well as current data. When such a large amount of sensor data is available, isolating the sensor data that is indicative of corrective action can become very cumbersome. This analysis system can be wasteful in terms of time spent, computer processing time and associated energy costs, as well as the risk of discovering some information but missing other indicative sensor data by stopping the very cumbersome analysis after discovering multiple components to perform maintenance, replacement, etc.

[0012] In some systems, the investigation into the sensor data can be triggered by suboptimal performance, e.g., one or more products having characteristics outside of manufacturing specifications have been produced. In some cases, only a portion of the products undergo performance measurements, e.g., metrology measurements, to ascertain production quality. Performance measurements (e.g., metrology) can be costly, e.g., can take a significant amount of time to produce. Manufacturing equipment may continue to be used to produce products while metrology is being performed on previous products and / or previous products are in a queue for metrology measurements. If manufacturing equipment degrades (e.g., components age, drift, fail, etc., causing the equipment to perform suboptimally), metrology measurements of the products may not be performed until multiple products have been processed using the suboptimal equipment. As a result, suboptimal products may be produced. Such systems are wasteful in terms of time wasted processing suboptimal products, energy spent, materials spent, etc.

[0013] In conventional systems, metrology-based defect detection results from the difficulty of isolating useful sensor data (e.g., identifying data indicative of corrective actions to be taken in conjunction with manufacturing equipment) from a large volume of sensor data. In some cases, sensor data may be observably affected before metrology is observably affected by drift, aging, or faulty components. In such cases, sensor data may be used to schedule corrective actions to occur concurrently with planned downtime (e.g., preventive maintenance operations) before metrology is affected, reducing costly unscheduled downtime for manufacturing systems. If sensor data cannot be reliably used, corrective actions may be performed in response to suboptimal metrology data, resulting in unscheduled downtime. In some embodiments, unscheduled downtime may incur additional costs, such as expedited shipments for replacement components.

[0014] The methods and devices of the present disclosure address at least one or more of the above-mentioned deficiencies in conventional solutions. In some embodiments, summary data is enabled and generated that allows for fast processing times and communication bandwidth, reduced complexity of analysis of the full trace data, etc. Generating the summary data can include application of statistical methods, machine learning methods, digital twin methods, etc. to the trace data (e.g., trace sensor data).

[0015] One or more metrics (e.g., standards of measurement) may be utilized to indicate the health of a system of manufacturing equipment. In some embodiments, summary data may be utilized to generate one or more metric values. The metric values ​​may be used as an indication that further investigation associated with a chamber, process, product, component, etc. should be performed. In some embodiments, a metric value (e.g., an index, a quality score, a quality index score, etc.) may be calculated that indicates the absence of an observable problem. In some embodiments, a tool-to-tool matching (TTTM) index may be generated to enable efficient isolation of a problematic system, tool, product, component, etc. Metric values ​​and indexes may be aggregated in various ways to aid in drill-down analysis, e.g., the process of increasing understanding of a tool from identifying that a problem exists to identifying that corrective action should be taken. For example, a metric related to a large number (e.g., more than a historical number) of outliers in the sensor data (e.g., reflected by summary data) may be utilized. In some embodiments, a metric value that meets a threshold may trigger further analysis, including the generation of another metric value associated with a different metric. In some embodiments, a set of metrics may be utilized as part of a drill-down analysis. For example, a first metric value (e.g., an indicator) may indicate that a manufacturing chamber is performing suboptimally, which may aggregate further indicators indicating that the performance is due to one subsystem, which may generate further indicators indicating the processing operation where the performance degradation occurred, and so on.

[0016] In some embodiments, data (e.g., summary data) may be provided to one or more models (e.g., machine learning models) to identify root causes, subsystems, corrective actions to be taken, etc. In some embodiments, data at every stage of drill-down analysis may be visualized and displayed via a graphical interface.

[0017] In some embodiments, an analysis may be performed utilizing the trace sensor data (e.g., in response to one or more metric values ​​or indicators indicative of suboptimal performance). The trace data may be provided to a machine learning model. The machine learning model may be trained to identify root causes, problematic subsystems, corrective actions to be taken, etc. The trace data analysis may be displayed via a graphical interface.

[0018] Aspects of the present disclosure provide technical advantages over conventional solutions. The present disclosure results in a more efficient substrate manufacturing process with minimized waste. If a problem occurs in the manufacturing chamber, the method of the present disclosure may be able to perform corrective actions before processing of the next product begins, before a suboptimal product is submitted for quality measurement, before quality measurement is completed, etc. In this manner, wasted material, manufacturing chamber time, and energy supplied to the manufacturing process may be minimized. Product quality may be increased if the root causes of product quality variations and chamber drift can be identified and corrected. The efficiency of making products of acceptable quality may also be increased if process parameters can be more precisely adjusted to improve material costs, energy, and time. Aging components may be identified and flagged for replacement or maintenance to reduce unscheduled downtime, costs associated with expedited shipment of replacement parts, etc.

[0019] In some embodiments, the present disclosure describes a method that includes receiving trace sensor data associated with a first manufacturing process of a manufacturing chamber. The method further includes processing the trace sensor data by a processing device to generate summary data associated with the trace sensor data. The method further includes generating a quality index score based on the summary data. The method further includes providing an alert to a user based on the quality index score. The alert includes an indication that the manufacturing chamber performance does not meet a first threshold.

[0020] In some embodiments, the present disclosure describes a method that includes receiving first data associated with measurements obtained by a sensor during a first manufacturing procedure of a manufacturing chamber. The method further includes receiving second data, the second data including reference data associated with the first data. The method further includes providing the first data and the second data to a comparison model. The method further includes receiving, from the comparison model, a similarity score associated with the first data and the second data. The method further includes performing a corrective action in consideration of the similarity score.

[0021] In some embodiments, the present disclosure describes a method that includes receiving trace sensor data associated with a first manufacturing process of a processing chamber. The method further includes processing the trace sensor data using one or more trained machine learning models that generate a representation of the trace sensor data, and then generating reconstructed sensor data based on the representation of the trace sensor data. The one or more trained machine learning models output the reconstructed sensor data. The method further includes comparing the trace sensor data to the reconstructed sensor data. The method further includes determining one or more differences between the reconstructed sensor data and the trace sensor data based on the comparison. The method further includes determining whether to recommend a corrective action associated with the processing chamber based on the one or more differences between the trace sensor data and the reconstructed sensor data.

[0022] 1 is a block diagram illustrating an example system 100 (example system architecture) according to some embodiments. System 100 includes client devices 120, manufacturing equipment 124, sensors 126, measurement equipment 128, a prediction server 112, and a data store 140. Prediction server 112 may be part of a prediction system 110. Prediction system 110 may further include server machines 170 and 180.

[0023] The sensors 126 can provide sensor data 142 associated with the manufacturing equipment 124 (e.g., associated with the manufacturing equipment 124 producing a corresponding product, such as a substrate). The sensor data 142 can be used for equipment health and / or product health (e.g., product quality). The manufacturing equipment 124 can produce the product according to a recipe or run over a period of time. In some embodiments, the sensor data 142 can include one or more values ​​of temperature (e.g., heater temperature), spacing (SP), pressure, high frequency radio frequency (HFRF), radio frequency (RF) match voltage, RF match current, RF match capacitor position, electrostatic chuck (ESC) voltage, actuator position, current, flow rate, power, voltage, etc. The sensor data 142 can be associated with or indicative of manufacturing parameters, such as hardware parameters (e.g., settings or components, e.g., size, type, etc.) of the manufacturing equipment 124 or process parameters of the manufacturing equipment 124. Alternatively or in addition, data associated with some hardware parameters can be stored as manufacturing parameters 150, which can include past manufacturing parameters 152 and current manufacturing parameters 154. The manufacturing parameters 150 can indicate input settings for the manufacturing devices (e.g., heater power, gas flow rates, etc.). When the manufacturing equipment 124 is performing a manufacturing process, it can provide sensor data 142 and / or manufacturing parameters 150 (e.g., can be equipment readings generated during processing of a product). The sensor data 142 can vary from product to product (e.g., from substrate to substrate). The substrate can have characteristic values ​​(e.g., film thickness, film strain, etc.) measured by the metrology equipment 128. The metrology data 160 can be a type of data stored in the data store 140.

[0024] In some embodiments, the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150 may be processed (e.g., by the client device 120 and / or the prediction server 112). The processing of the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150 may include generating features. In some embodiments, these features are patterns (e.g., slope, width, height, peaks, etc.) in the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150, or combinations of values ​​(e.g., power derived from voltage and current, etc.) from the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150. The sensor data 142 may include features, which may be used by the prediction component 114 to perform signal processing and / or to obtain prediction data 168 for taking corrective actions.

[0025] Each instance (e.g., set) of sensor data 142 may correspond to a product (e.g., board), a set of manufacturing equipment, a type of board produced by the manufacturing equipment, etc. Each instance of metrology data 160 and manufacturing parameters 150 may similarly correspond to a product, a set of manufacturing equipment, a type of board produced by the manufacturing equipment, etc. The data store may further store information relating sets of different data types, e.g., information indicating a set of sensor data, a set of metrology data, and a set of manufacturing parameters are all associated with the same product, manufacturing equipment, type of board, etc.

[0026] In some embodiments, data associated with the processing of one or more products can be used to generate summary data 162. Summary data 162 can include data that characterizes other data. For example, summary data 162 can be generated from trace sensor data 142. Trace sensor data 142 can include large amounts of data (e.g., data from hundreds of sensors in a tool each obtaining hundreds or thousands of measurements per product). Summary data 162 can be designed to be less cumbersome to manipulate than trace sensor data 142 and to present information (e.g., information indicative of a chamber failure). In some embodiments, data (e.g., trace sensor data 142) can be provided to a processing device (e.g., prediction server 112, client device 120, etc.) to generate summary data 162. Summary data 162 can include, for example, metadata (e.g., tool ID, recipe name, product ID, product information, etc.), contextual data (e.g., sensor ID, step number, timestamp, subsystem, etc.), and / or basic statistics (e.g., mean, maximum, minimum, quartiles, kurtosis, control limits, etc.). In some embodiments, generating the summary data 162 can include using a digital twin model of one or more components of the manufacturing equipment 124. As used herein, a digital twin is a digital replica of a physical asset, such as a manufactured part, a chamber part, or a process chamber. The digital twin includes properties of the physical asset, including, but not limited to, coordinate axis dimensions, weighting properties, material properties (e.g., density, surface roughness), electrical properties (e.g., conductivity), optical properties (e.g., reflectivity), and the like. In some embodiments, the digital twin model can be used to generate a prediction of the time behavior of the data, e.g., the sensor data 142. The summary data 162 can include an indication of how the characteristics of the measured data differ from the prediction of the digital twin model (e.g., overshoot, rise time, settling time, steady state value error, and the like).In some embodiments, the summary data 162 may be generated based on a portion of the input data, e.g., only a steady state portion of the summary data 162, only a transient portion, etc. The summary data 162 is described in more detail in connection with FIG. 5B.

[0027] In some embodiments, the prediction system 110 may generate the predicted data 168 using machine learning, such as supervised machine learning (e.g., a machine learning model may be configured to create labels associated with input data, such as measurement predictions, performance predictions, etc.). In some embodiments, the prediction system 110 may generate the predicted data 168 using unsupervised machine learning (e.g., a machine learning model may be trained with unlabeled data, such as a model configured to perform clustering, dimensionality reduction, etc.). In some embodiments, the prediction system 110 may generate the predicted data 168 using semi-supervised learning (e.g., a machine learning model may be trained using both labeled and unlabeled input datasets).

[0028] The client devices 120, manufacturing equipment 124, sensors 126, measurement equipment 128, prediction server 112, data store 140, server machine 170, and / or server machine 180 can be coupled to each other via network 130 to generate predictive data 168 and perform corrective actions.

[0029] In some embodiments, the network 130 is a public network that provides the client device 120 with access to the prediction server 112, the data store 140, and other publicly available computing devices. In some embodiments, the network 130 is a private network that provides the client device 120 with access to the manufacturing equipment 124, the sensors 126, the measurement equipment 128, the data store 140, and other privately available computing devices. In some embodiments, the functions of one or more of the server machine 170, the server machine 180, and / or the prediction server 112 may be performed by a virtual machine, for example by utilizing a cloud-based service. The network 130 may provide access to such a virtual machine. The network 130 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof.

[0030] The client device 120 may include computing devices such as a personal computer (PC), a laptop, a mobile phone, a smartphone, a tablet computer, a netbook computer, a network-connected television ("smart TV"), a network-connected media player (e.g., a Blu-ray player), a set-top box, an over-the-top (OTT) streaming device, an operator box, etc. The client device 120 may include a corrective action component 122. The corrective action component 122 may receive user input of instructions (e.g., via a graphical user interface (GUI) displayed via the client device 120) associated with the manufacturing equipment 124. In some embodiments, the corrective action component 122 transmits the instructions to the prediction system 110, receives output (e.g., prediction data 168) from the prediction system 110, determines a corrective action based on the output, and implements the corrective action. In some embodiments, the corrective action component 122 obtains (e.g., from the data store 140, etc.) sensor data 142 associated with the manufacturing equipment 124 (e.g., current sensor data 146) and provides the sensor data 142 associated with the manufacturing equipment 124 (e.g., current sensor data 146) to the predictive system 110. In some embodiments, the corrective action component 122 stores the sensor data 142 in the data store 140, and the predictive server 112 retrieves the sensor data 142 from the data store 140. In some embodiments, the predictive server 112 can store the output of the trained model 190 (e.g., predicted data 168) in the data store 140, and the client device 120 can retrieve the output from the data store 140. In some embodiments, the corrective action component 122 receives corrective action instructions from the predictive system 110 and implements the corrective action. Each client device 120 may include an operating system that enables a user to one or more of create, view, or edit data (e.g., instructions associated with the manufacturing equipment 124, corrective actions associated with the manufacturing equipment 124, etc.).

[0031] In some embodiments, the metrology data 160 corresponds to historical characteristic data of a product (e.g., produced using manufacturing parameters associated with historical sensor data 144 and historical manufacturing parameters 152), and the predictive data 168 is associated with predicted characteristic data (e.g., predicted data of a product to be or have been produced in conditions recorded by current sensor data 146 and / or current manufacturing parameters 154). In some embodiments, the predictive data 168 is predicted metrology data (e.g., virtual metrology data) of a product to be or have been produced in accordance with conditions recorded as current sensor data 146 and / or current manufacturing parameters 154. In some embodiments, the predictive data 168 is an indication of anomalies (e.g., an abnormal product, an abnormal component, an abnormal manufacturing equipment 124, an abnormal energy usage, etc.) and may be an indication of one or more causes of those anomalies. In some embodiments, the predictive data 168 is an indication of a change or drift over time in some component, such as manufacturing equipment 124, a sensor 126, a metrology device 128, etc. In some embodiments, the predictive data 168 is an indication of the end of life of a component, such as manufacturing equipment 124 , a sensor 126 , a measurement device 128 , or the like.

[0032] Running a manufacturing process that results in a defective product can be costly in terms of time, energy, product, components, manufacturing equipment 124, costs of identifying the defects and disposing of the defective product, etc. By inputting sensor data 142 (e.g., manufacturing parameters that are being or should be used to manufacture the product) into the predictive system 110, receiving an output of predictive data 168, and performing corrective actions based on the predictive data 168, the system 100 can have the technical advantage of avoiding the costs associated with creating, identifying, and disposing of defective products.

[0033] Running a manufacturing process that results in a failure of a component of the manufacturing equipment 124 can be costly in terms of downtime, product damage, equipment damage, expedited ordering of replacement components, etc. By inputting sensor data 142 (e.g., indicative of manufacturing parameters that are being or should be used to manufacture a product) into the predictive system 110, receiving an output of predictive data 168, and performing corrective actions (e.g., predicted operational maintenance such as replacing, treating, cleaning, etc. of components) based on the predictive data 168, the system 100 can have the technical advantage of avoiding the costs of one or more of unexpected component failures, unscheduled downtime, lost production rate, unexpected equipment failures, product waste, etc. Monitoring the performance of components, e.g., manufacturing equipment 124, sensors 126, metrology equipment 128, etc., over time can provide an indication of deteriorating components.

[0034] The manufacturing parameters may be sub-optimal for producing the product, which may result in costly consequences such as increased resource (e.g., energy, coolant, gas, etc.) consumption, increased amount of time to produce the product, increased component failures, increased amount of defective products, etc. By inputting the sensor data 142 into the trained model 190, receiving the output of the prediction data 168, and performing corrective actions to update the manufacturing parameters (e.g., set optimal manufacturing parameters) (e.g., based on the prediction data 168), the system 100 may have the technical advantage of using optimal manufacturing parameters (e.g., hardware parameters, process parameters, optimal design) and / or healthy equipment to avoid the costly consequences of sub-optimal manufacturing parameters.

[0035] The corrective action may be associated with one or more of computational process control (CPC), statistical process control (SPC) (e.g., SPC on electronic components to determine the process being controlled, SPC to predict the useful life of a component, SPC to compare with 3σ graphs, etc.), advanced process control (APC), model-based process control, preventive operational maintenance, design optimization, manufacturing parameter updates, manufacturing recipe updates, feedback control, machine learning corrections, and the like.

[0036] In some embodiments, the corrective action includes providing an alert (e.g., an alert to stop or not perform a manufacturing process on additional substrates if the predictive data 168 indicates a predicted anomaly, such as an anomaly in a product, component, or manufacturing equipment 124). In some embodiments, the corrective action includes providing feedback control (e.g., modifying a manufacturing parameter in response to the predictive data 168 indicating a predicted anomaly). In some embodiments, performing the corrective action includes updating one or more manufacturing parameters.

[0037] The manufacturing parameters may include hardware parameters (e.g., information indicative of components contained within the manufacturing equipment, an indication of recently replaced components, an indication of firmware versions or updates, etc.) and / or process parameters (e.g., temperature, pressure, flow rate, current and / or voltage, gas flow rate, ramp rate, etc.). In some embodiments, the corrective action includes performing preventive operational maintenance (e.g., replacing, treating, cleaning components of the manufacturing equipment 124, etc.). In some embodiments, the corrective action includes performing design optimization (e.g., updating manufacturing parameters for an optimized product, updating the manufacturing process, updating the manufacturing equipment 124, etc.). In some embodiments, the corrective action includes updating a recipe (e.g., changing timing of instructions to put the manufacturing equipment 124 into idle mode, sleep mode, warm-up mode, etc., adjusting set points for temperature, gas flow rate, plasma generation, etc.).

[0038] Prediction server 112, server machine 170, and server machine 180 may each include one or more computing devices such as a rack mount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a graphics processing unit (GPU), an accelerator application specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc.

[0039] The prediction server 112 can include a prediction component 114. In some embodiments, the prediction component 114 can receive current sensor data 146 and / or current manufacturing parameters 154 (e.g., received from client device 120, retrieved from data store 140) and generate output (e.g., predicted data 168) for performing corrective actions associated with the manufacturing equipment 124 based on the current data. In some embodiments, the prediction component 114 can use one or more trained models 190 to determine the output for performing corrective actions based on the current data.

[0040] In some embodiments, the model 190 can include a trained physics-based digital twin model. The physics-based model can be capable of solving simultaneous equations that describe physical phenomena that may occur in the fabrication chamber, such as equations governing heat flow, energy balance, gas conduction, mass balance, fluid dynamics, and current flow. In some embodiments, the physics-based model performs calculations of gas conduction in the fabrication chamber. The fabrication parameters 150 can be provided to the trained physics-based model. The trained physics-based model can provide as output modeled characteristic values ​​indicative of conditions in the fabrication chamber corresponding to sensors 126 disposed in the fabrication chamber (e.g., fabrication equipment 124). The output of the physics-based model can be stored in the data store 140.

[0041] The prediction component 114 of the prediction server 112 can receive summary data 162 generated from data collected by the sensors 126. The prediction component 114 can utilize the summary data 162 to generate prediction data 168. The prediction data 168 can include indications of failures, aging, drift, etc. in the manufacturing equipment 124, the sensors 126, etc. In some embodiments, the summary data can be utilized to generate indicators, metrics indicative of the health and / or performance of, for example, a tool, a chamber, a set of manufacturing equipment 124, etc. The indicators can be used for corrective action, can be displayed to a user, for example, on a GUI, further analysis can be performed, and so on. The indicators can indicate that further investigation of the cause of the failure, aging, drift, etc. should be performed. In some embodiments, the indicators can be generated by providing the summary data to a trained machine learning model (e.g., model 190). In some embodiments, the prediction component 114 can utilize summary data associated with indicators indicative of further investigation to be performed (e.g., summary data associated with irregular index values) to isolate subsystems related to the performance of the tool. In some embodiments, the summary data associated with an indicator indicating further investigation to be performed is used by the predictive component 114 to perform an analysis indicating unexpected drift in the sensor data. In some embodiments, the trace data associated with an indicator indicating further investigation to be performed can be utilized to isolate subsystems that contribute to the performance of the tool. Further discussion of methods related to generation of summary data, generation of indicators, and drill-down analysis is provided in conjunction with FIGS. 5A-5G.

[0042] The historical sensor data 144 and historical manufacturing parameters 152 can be used to train one or more models 190, including physics-based models, digital twin models, machine learning models, etc. In some embodiments, the digital twin models can be used to generate summary data. The digital twin models can be generated from physical principles. In some embodiments, the generated digital twin models may not capture some complex parts of the manufacturing system, such as parts, which may be manufactured slightly differently (e.g., within manufacturing tolerances), parts may age or drift, and so on. To increase accuracy, the historical data can be used to refine the physics-based or digital twin models. The historical data can also be used to train one or more machine learning models. In some embodiments, the machine learning models can be utilized to generate indicators, generate visual representations of the data, generate predictions of metrology data of the completed product, etc.

[0043] In some embodiments, the sensor data 142 may include data collected from the sensors 126 during a manufacturing run that produced an acceptable product (e.g., as measured by the metrology equipment 128). A manufacturing run that produces an acceptable product may be referred to as a golden run. The sensor data associated with such a manufacturing run may be stored as part of the historical sensor data 144 in the data store 140. The prediction component 114 of the prediction server 112 may compare the golden run sensor data, the current sensor data 146, and the expected sensor data (e.g., output by a trained physics-based model) to determine whether a component failure, drift, etc. is occurring. In some embodiments, some or all of these operations may instead be performed by different devices, e.g., the client device 120, the server machine 170, the server machine 180, etc.

[0044] In some embodiments, the predictive component 114 can determine which components contribute to the difference between expected sensor data (e.g., historical sensor data, sensor data from different tools, sensor data reflecting manufacturing parameters, etc.), current sensor data 146, and / or golden run sensor data. This can be done, for example, by examining historical data (e.g., by machine learning models, statistical models, etc.) of systems performing as intended and systems exhibiting suboptimal performance. Labeling with the sensor trace data an indication of the cause of the fault (e.g., the subsystem responsible for the suboptimal performance) can enable the processing device to link the sensor difference (e.g., a particular pattern or feature in the data from a particular sensor) to the fault. The processing logic can be configured to link a particular feature or pattern in the sensor data to a particular type of fault, aging, drift, or failure of a particular component, etc. Sensors can be assigned metric values ​​indicative of performance, for example, a metric value that aggregates the likelihood that a sensor indicates a fault over multiple operations or runs. The sensor data can be grouped into subsystems. The sensor metric values ​​or indicators can be aggregated into subsystem metric values ​​or indicators. The aggregated subsystem values ​​(eg, metric values, indices, etc.) may be used, such as in identifying subsystem performance.

[0045] The machine learning model (e.g., one or more of the models 190) can be trained using historical sensor data 144, historical manufacturing parameters 152, metrology data 160 corresponding to the run in which the historical sensor data 144 was measured, etc. One type of machine learning model that can be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network. An artificial neural network generally includes a feature representation component with a classification or regression layer that maps features to a target output space. Convolutional neural networks (CNNs), for example, host multiple layers of convolutional filters. Pooling can be performed to address nonlinearities at lower layers, and multi-layer perceptrons are typically added on top to map upper layer features extracted by the convolutional layers to decisions (e.g., classification outputs). Deep learning is a class of machine learning algorithms that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks include a hierarchical structure of multiple layers, with different layers learning different levels of representations corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. In an image recognition application, for example, the raw input may be: a matrix of pixels, a first representation layer may abstract the pixels and encode edges, a second layer may compose and encode the arrangement of edges, a third layer may encode higher level shapes (e.g., leaves, lips, gums, etc.), and a fourth layer may recognize the role of scanning. In particular, the deep learning process can learn which features naturally fall into which levels best. The "deep" in "deep learning" refers to the number of layers through which data is transformed. More precisely, deep learning systems have substantial contribution allocation path (CAP) depth. A CAP is a chain of transformations from input to output.CAP describes the potentially causal connections between inputs and outputs. For forward neural networks, the CAP depth can be the depth of the network, which can be the number of hidden layers plus one. For recurrent neural networks, where a signal can propagate through a layer more than once, the CAP depth is potentially unlimited.

[0046] The use of physics-based digital twin models in association with manufacturing equipment has significant technical advantages over operating the manufacturing equipment without such models. Multiple components (sensors, hardware, processing, etc.) can be tied together in a closed control loop. If there is a problem with any of the equipment in the closed control loop, other components can adjust set points, etc., and readings can return to target values, but the actual conditions in the chamber may have changed. This can result in less consistent substrate fabrication, lost fabrication rates, energy, time, and material inefficiencies, premature component failures, and increased maintenance costs. In some embodiments, multiple data streams indicative of the same characteristic can be monitored to provide additional evidence of the root cause of changes in sensor data, chamber conditions, etc. In some embodiments, a sensor indicative of chamber pressure is monitored. Sensors can include sensors that measure chamber pressure, actuator positions of actuators that affect chamber pressure, RF match voltage, RF match current, RF match capacitor position, etc. Some of these characteristics can be measured by more than one sensor. For example, a manufacturing chamber can include multiple pressure sensors, some of which are included in the closed control loop and some of which are free to provide measurements of chamber conditions. The physics-based models can allow for slight changes, modifications, drift, etc. in sensor readings, sensor performance, etc. to be captured and analyzed.

[0047] In some embodiments, the predictive component 114 receives the current sensor data 146 and / or the current manufacturing parameters 154, performs signal processing to decompose the current data into a set of current data, provides the set of current data as input to the trained model 190, and obtains an output indicative of predicted data 168 from the trained model 190. In some embodiments, the predicted data 168 is indicative of the metrology data 160 (e.g., a prediction of substrate quality). In some embodiments, the predicted data 168 is indicative of the health of the component. In some embodiments, the predicted data 168 is indicative of the performance of the component.

[0048] In some embodiments, the various models discussed in connection with the model 190 (e.g., physics-based digital twin models, predictive machine learning models, etc.) can be combined into one model (e.g., an ensemble model) or can be separate models. The predictive component 114 can receive current sensor data 146 and current manufacturing parameters 154, provide the data to the trained model 190, and receive information indicating how much some components in the manufacturing chamber have drifted from their previous performance. Data can be passed between some separate models included in the model 190 and the predictive component 114. In some embodiments, some or all of these operations can instead be performed by different devices, such as the client device 120, the server machine 170, the server machine 180, etc. Those skilled in the art will appreciate that variations in data flow, which components perform which processes, which data is provided to which models, etc. are within the scope of this disclosure.

[0049] The data store 140 can be a memory (e.g., random access memory), a drive (e.g., hard drive, flash drive), a database system, or another type of component or device capable of storing data. The data store 140 can include multiple storage components (e.g., multiple drivers or multiple databases) that can span multiple computing devices (e.g., multiple server computers). The data store 140 can store sensor data 142, manufacturing parameters 150, metrology data 160, summary data 162, and forecast data 168. The sensor data 142 can include historical sensor data 144 and current sensor data 146. The sensor data can include pre-processed data such as sensor data time traces for the duration of the manufacturing process, associations of data with physical sensors, average and composite data, and data indicative of sensor performance over time (i.e., many manufacturing processes). The manufacturing parameters 150 and metrology data 160 can include similar characteristics. The historical sensor data 144 and historical manufacturing parameters 152 can be historical data (e.g., at least a portion for training the model 190). The current sensor data 146 may be the current data (e.g., at least a portion of the historical data to be input into the learning model 190) for which predictive data 168 (e.g., to perform corrective actions) should be generated. The summary data 162 may include the processed sensor data 142. The summary data may include information indicative of corrective actions to be performed in connection with the manufacturing equipment 124. The summary data may be less intensive to manipulate than the trace sensor data.

[0050] In some embodiments, the prediction system 110 further includes a server machine 170 and a server machine 180. The server machine 170 includes a dataset generator 172 capable of generating datasets (e.g., a set of data inputs and a set of target outputs) for training, validating, and / or testing the physics-based digital twin model, and the model 190, which in some embodiments includes one or more machine learning models. Some operations of the dataset generator 172 are described in more detail below with respect to FIGS. 3, 4, and 5A. In some embodiments, the dataset generator 172 can divide the historical data (e.g., the historical sensor data 144, the historical manufacturing parameters 152, the summary data 162 stored in the data store 140) into a training set (e.g., 60 percent of the historical data), a validation set (e.g., 20 percent of the historical data), and a test set (e.g., 20 percent of the historical data). In some embodiments, the prediction system 110 (e.g., via the prediction component 114) generates a plurality of sets of features. For example, the first set of features may correspond to a first set of types of sensor data (e.g., from a first set of sensors, a first combination of values ​​from the first set of sensors, a first pattern of values ​​from the first set of sensors) corresponding to each of the datasets (e.g., a training set, a validation set, and a test set), and the second set of features may correspond to a second set of types of sensor data (e.g., from a second set of sensors different from the first set of sensors, a second combination of values ​​different from the first combination, a second pattern different from the first pattern) corresponding to each of the datasets.

[0051] The server machine 180 includes a training engine 182, an authentication engine 184, a selection engine 185, and / or a test engine 186. The engines (e.g., the training engine 182, the authentication engine 184, the selection engine 185, and the test engine 186) can refer to hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, a processing device, etc.), software (e.g., instructions executed on a processing device, a general-purpose computer system, or a dedicated machine), firmware, microcode, or a combination thereof. The training engine 182 may be capable of training a model 190 using one or more sets of features associated with a training set from the dataset generator 172. The training engine 182 can generate multiple trained models 190, each trained model 190 corresponding to a distinct set of features of the training set (e.g., sensor data from a distinct set of sensors). For example, a first trained model may be trained using all features (e.g., X1-X5), a second trained model may be trained using a first subset of features (e.g., X1, X2, X4), and a third trained model may be trained using a second subset of features (e.g., X1, X3, X4, and X5), where the second subset of features may partially overlap with the first subset of features. The dataset generator 172 may receive the output of the trained models (e.g., physics-based digital twin models, machine learning models), collect that data into training, validation, and testing datasets, and use these datasets to train a second model (e.g., a machine learning model configured to output predictive data, corrective actions, etc.).

[0052] The authentication engine 184 may be capable of authenticating the trained models 190 using a corresponding set of features of the authentication set from the dataset generator 172. For example, a first trained model 190 trained using a first set of features of the training set may be authenticated using a first set of features of the authentication set. The authentication engine 184 may determine the accuracy of each of the trained models 190 based on the corresponding set of features of the authentication set. The authentication engine 184 may discard trained models 190 that have an accuracy that does not meet a threshold accuracy. In some embodiments, the selection engine 185 may be capable of selecting one or more trained models 190 that have an accuracy that meets the threshold accuracy. In some embodiments, the selection engine 185 may be capable of selecting the trained model 190 with the highest accuracy of the trained models 190.

[0053] The testing engine 186 may be capable of testing the trained model 190 using a corresponding set of test set features from the dataset generator 172. For example, a first trained model 190 trained using a first set of training set features may be tested using a first set of test set features. The testing engine 186 may determine the trained model 190 that has the highest accuracy among all of the trained models based on the test set.

[0054] In the case of a machine learning model, the model 190 can refer to a model artifact created by the training engine 182 using a training set that includes data inputs and corresponding target outputs (correct responses for each training input). Patterns in the dataset that map the data inputs to the target outputs (correct responses) can be discovered, and the machine learning model 190 is provided with a mapping that captures these patterns. The machine learning model 190 can use one or more of a support vector machine (SVM), a radial basis function (RBF), clustering, supervised machine learning, semi-supervised machine learning, unsupervised machine learning, a k-nearest neighbor algorithm (k-NN), linear regression, random forests, neural networks (e.g., artificial neural networks), and the like.

[0055] The prediction component 114 can provide the current sensor data 146 and / or the summary data 162 to the model 190, and can execute the model 190 on the inputs to obtain one or more outputs. The prediction component 114 can be able to determine (e.g., extract) prediction data 168 from the output of the model 190, and can determine (e.g., extract) confidence data from the output indicating a confidence level that the prediction data 168 is an accurate predictor of the process associated with the current sensor data 146 and / or input data for a product made or to be made using the manufacturing equipment 124 with the current manufacturing parameters 154. The prediction component 114 or the corrective action component 122 can use the confidence data to determine whether to trigger a corrective action associated with the manufacturing equipment 124 based on the prediction data 168.

[0056] The confidence data may include or indicate a confidence level that the prediction data 168 is an accurate prediction for a product or component associated with at least a portion of the input data. In one example, the confidence level is a real number between 0 and 1, inclusive, where 0 indicates no confidence that the prediction data 168 is an accurate prediction for a product processed according to the input data or component health state of the component of the manufacturing equipment 124, and 1 indicates absolute confidence that the prediction data 168 accurately predicts a characteristic of a product processed according to the input data or component health state of the component of the manufacturing equipment 124. In response to confidence data indicating a confidence level below a threshold level for a predetermined number of cases (e.g., a percentage of cases, a frequency of cases, a total number of cases, etc.), the prediction component 114 may cause the trained model 190 to be retrained (e.g., based on current sensor data 146, current manufacturing parameters 154, etc.).

[0057] For purposes of explanation and not limitation, aspects of the disclosure describe using historical data (e.g., historical sensor data 144, historical production parameters 152) to train one or more machine learning models 190 and inputting current data (e.g., current sensor data 146, current production parameters 154, etc.) into the one or more trained machine learning models to determine the predicted data 168. In other embodiments, a heuristic or rule-based model is used to determine the predicted data 168 (e.g., without using a trained machine learning model). The predictive component 114 can monitor the historical sensor data 144, the historical production parameters 152, the summary data 162, and the metrology data 160. The heuristic or rule-based model can monitor or otherwise use any of the information described with respect to the data input 310 of FIG. 3.

[0058] In some embodiments, the functionality of client device 120, prediction server 112, server machine 170, and server machine 180 may be provided by fewer machines. For example, in some embodiments, server machines 170 and 180 may be combined into a single machine, and in some other embodiments, server machine 170, server machine 180, and prediction server 112 may be combined into a single machine. In some embodiments, client device 120 and prediction server 112 may be combined into a single machine.

[0059] In general, functions described in one embodiment as being performed by client device 120, prediction server 112, server machine 170, and server machine 180 may also be performed by prediction server 112 in other embodiments, where appropriate. In addition, functions attributed to particular components may also be performed by different or multiple components operating together. For example, in some embodiments, prediction server 112 may determine corrective actions based on prediction data 168. In another example, client device 120 may determine prediction data 168 based on output from a trained machine learning model, a physics-based model, or the like.

[0060] Additionally, the functionality of a particular component may be performed by different or multiple components working together. One or more of prediction server 112, server machine 170, or server machine 180 may be accessed as a service offered to other systems or devices via an appropriate application programming interface (API).

[0061] In embodiments, a "user" may be represented as a single individual. However, other embodiments of the present disclosure encompass a "user" that is an entity managed by multiple users and / or automated sources. For example, a set of individual users that are aggregated as a group of administrators may be considered a "user."

[0062] Embodiments of the present disclosure may be applied to data quality assessment, feature enhancement, model evaluation, virtual metrology (VM), predictive maintenance (PdM), marginal optimization, and the like.

[0063] 2 illustrates a cross-sectional view of a fabrication chamber 200 (e.g., a semiconductor wafer fabrication chamber) according to some aspects of the disclosure. The fabrication chamber 200 can be one or more of an etch chamber (e.g., a plasma etch chamber), a deposition chamber (including atomic layer deposition, chemical vapor deposition, physical vapor deposition, or plasma versions thereof), an anneal chamber, etc. For example, the fabrication chamber 200 can be a chamber for a plasma etcher, a plasma cleaner, etc. Example chamber parts can include a substrate support assembly 204, an electrostatic chuck, a ring (e.g., a process kit ring), chamber walls, a base, a showerhead 206, a gas distribution plate, a liner, a liner kit, a shield, a plasma screen, a flow balancer, a cooling base, a chamber viewport, a chamber lid, a nozzle, etc.

[0064] In one embodiment, the fabrication chamber 200 includes a chamber body 208 and a showerhead 206 that enclose an interior volume 210. In some chambers, the showerhead 206 can be replaced with a lid and nozzle. The chamber body 208 can be constructed from aluminum, stainless steel, or other suitable materials. The chamber body 208 generally includes a sidewall 212 and a bottom 214.

[0065] An exhaust outlet 216 may be defined in the chamber body 208, and the interior volume 210 may be coupled to a pumping system 218. The pumping system 218 may include one or more pumps and valves utilized to evacuate and regulate the pressure of the interior volume 210 of the fabrication chamber 200. Actuators may be located at or near the exhaust outlet 216 for controlling the gas flow rate out of the chamber and / or the pressure within the chamber.

[0066] The showerhead 206 can be supported on a sidewall 212 of the chamber body 208 or on the top of the chamber body. The showerhead 206 (or in some embodiments, a lid) can be opened to allow access to the interior volume 210 of the fabrication chamber 200 and can provide a seal for the fabrication chamber 200 when closed. A gas panel 220 can be coupled to the fabrication chamber 200 to provide process or cleaning gases to the interior volume 210 through the showerhead 206 (or lid and nozzles). Multiple gas delivery holes can be included throughout the showerhead 206. Examples of process gases that can be used to process substrates in the fabrication chamber 200 include halogen-containing gases such as C2F6, SF6, SiCl4, HBr, NF3, CF4, CHF3, F2, Cl2, CCl4, BCl3, and SiF4, among others, as well as other gases such as O2 or N2O. Examples of carrier gases include N2, He, Ar, and other gases that are inert to the process gases (e.g., non-reactive gases).

[0067] The substrate support assembly 204 is disposed within the interior volume 210 of the fabrication chamber 200 below the showerhead 206. In some embodiments, the substrate support assembly 204 includes a susceptor 222 and a shaft 224. The substrate support assembly 204 supports a substrate during processing. In some embodiments, one or more heaters 226 and reflectors 228 are also disposed within the fabrication chamber 200.

[0068] Sensors are used to measure properties within the fabrication chamber 200. Sensors associated with the fabrication chamber 200 may include temperature sensors, pressure sensors (in some embodiments, there may be at least two pressure sensors), flow sensors, etc. In some embodiments, a physics-based digital twin model of the fabrication chamber 200 can be utilized to identify faulty components and take corrective action.

[0069] In some embodiments, the chamber 200 can include multiple sensors that collect time trace data. The time trace data can be transmitted to a data storage device (e.g., data store 140 of FIG. 1). Summary data can be generated from the time trace sensor data. Processing of the summary data and / or the time trace data can generate data indicative of problematic (e.g., faults, drift, aging, etc.) chambers, subsystems, components, etc. The data processing can be performed by one or more models, including physics-based models, machine learning models, digital twin models, etc. The summary data can be used to generate an index. The value of the index can be indicative of the performance of a tool, component, fleet, etc.

[0070] FIG. 3 is a diagram of a dataset generator 372 (e.g., dataset generator 172 of FIG. 1 ) that creates datasets for training, testing, validation, etc. of a model (e.g., model 190 of FIG. 1 ) according to some embodiments. The dataset generator 372 may be part of the server machine 170 of FIG. 1 . In some embodiments, the model 190 may include multiple machine learning models, physics-based models, digital twin models, etc. Each model may have its own dataset generator, or multiple models may share a dataset generator. The operation of one dataset generator (associated with the model used to map sensor data to performance data) is described in detail in connection with FIG. 3 . Any of the other models may also be associated with a similar dataset generation procedure. For other models, the input and output data may differ from the input and output data described in connection with FIG. 3 .

[0071] The system 300, including a dataset generator 372 (e.g., dataset generator 172 of FIG. 1 ), creates a dataset for a machine learning model (e.g., model 190 of FIG. 1 ). The dataset generator 372 can create a dataset (e.g., data input 310) using historical manufacturing parameters 352 (e.g., historical manufacturing parameters 152 of FIG. 1 ), historical sensor data, summary sensor data, etc. The system 300 can be used to generate datasets for training, testing, and validating machine learning models. In some embodiments, the machine learning model is provided with performance data 322 as a target output 320. The performance data can be indicative of characteristics of a finished product, e.g., metrology data of a processed substrate. In some embodiments, the performance data can be indicative of a condition of one or more components of manufacturing equipment, e.g., failure, aging, drift, recommended maintenance, etc. In some embodiments, the dataset generator 372 can be configured to generate a dataset for a physics-based model of a component of manufacturing equipment. The output of the physics-based model can be refined by training the model using sensor data as a target output. In some embodiments, sensor data associated with a successful process (e.g., golden run data from a process that produced a product within manufacturing tolerances, golden run data from a recently qualified chamber, etc.) can be utilized as input 310 or target output 320.

[0072] In some embodiments, the dataset generator 372 generates a dataset (e.g., a training set, a validation set, a test set) that includes one or more data inputs 310 (e.g., training inputs, validation inputs, test inputs) and can include one or more target outputs 320 corresponding to the data inputs 310. The dataset can also include mapping data that maps the data inputs 310 to the target outputs 320. The data inputs 310 can also be referred to as “features,” “attributes,” or “information.” In some embodiments, the dataset generator 372 can provide a dataset to the training engine 182, validation engine 184, or test engine 186 that uses the dataset to train, certify, or test one or more models 190 (e.g., one of the models included in the model 190, ensemble model 190, etc.). In some embodiments, the model 190 can include a physics-based digital twin model and a model used to analyze the output of the physics-based model. In some embodiments, the dataset generator 372 can provide inputs to the physics-based model as training inputs (e.g., manufacturing parameters) and can provide target output data associated with the outputs of the second model (e.g., component contribution data from a machine learning model), and these associated datasets can be used to train the model 190. Some embodiments of generating a training set can be further described with respect to FIG. 5A.

[0073] In some embodiments, the dataset generator 372 generates the data input 310 but does not generate the target output 320 to provide a training dataset for an unsupervised machine learning model, such as a clustering model, an autoencoder, a dimensionality reduction model, etc. The operation of such models is further described in connection with FIG. 7. In some embodiments, the dataset generator 372 generates the data input 310 and the target output 320 (e.g., for training an unsupervised or semi-supervised model). In some embodiments, the data input 310 can include one or more sets of data. As an example, the system 300 can produce a set of sensor data, such sensor data can include one or more of sensor data from one or more types of sensors, a combination of sensor data from one or more types of sensors, patterns from sensor data from one or more types of sensors, manufacturing parameters from one or more manufacturing parameters, a combination of some manufacturing parameter data and some sensor data, summary data, etc.

[0074] In some embodiments, the dataset generator 372 can generate a first data input corresponding to a first set of historical sensor data 344A and / or historical manufacturing parameters 352A for training, validating, and / or testing a first machine learning model, and the dataset generator 372 can generate a second data input corresponding to a second set of historical sensor data 344B and / or historical manufacturing parameters 352B for training, validating, and / or testing a second machine learning model.

[0075] The data inputs 310 and target outputs 320 for training, validating, and / or testing a machine learning model may include information for a particular manufacturing chamber (e.g., a particular piece of substrate manufacturing equipment). For example, the historical manufacturing parameters 352, the historical sensor data 344, the summary data, and the performance data 322 may be associated with the same manufacturing chamber.

[0076] In some embodiments, the information used to train the machine learning model may be from a particular type of manufacturing equipment of a manufacturing facility having particular characteristics (e.g., manufacturing equipment 124 in FIG. 1 ), allowing the trained model to determine outcomes for a particular group of manufacturing equipment 124 based on inputs for current sensor data (e.g., current sensor data 146) associated with one or more components that share the characteristics of the particular group of manufacturing equipment 124. In some embodiments, the information used to train the model may be for components from more than one manufacturing facility, allowing the trained machine learning model to determine outcomes for components based on inputs from one manufacturing facility.

[0077] In some embodiments, after generating a dataset and using the dataset to train, validate, or test the machine learning model 190, the model 190 may be further trained, validated, or tested, or tuned (e.g., adjusting weights or parameters associated with the input data of the model 190, such as parameters in a physics-based model or connection weights in a neural network).

[0078] 4A is a block diagram illustrating a system 400 for generating output data (e.g., predicted data 168 of FIG. 1 ), according to some embodiments. In some embodiments, system 400 can be used in conjunction with a machine learning model (e.g., model 190 of FIG. 1 ) to determine correlations between sensors and performance of a tool, chamber, or set of manufacturing equipment, which can inform the execution of corrective actions (e.g., corrective actions can be executed in light of simulated sensor data). In some embodiments, a system similar to system 400 can be used in conjunction with a physics-based model to determine corrective actions associated with manufacturing equipment.

[0079] At block 410, the system 400 (e.g., a component of the predictive system 110 of FIG. 1 ) performs data partitioning (e.g., via the dataset generator 172 of the server machine 170 of FIG. 1 ) of the historical data 464 (e.g., the historical performance data for the model 190, the historical manufacturing parameters 152, and the historical sensor data 144 of FIG. 1 ) to generate a training set 402, a validation set 404, and a test set 406. For example, the training set may be 60% of the historical data, the validation set may be 20% of the historical data, and the test set may be 20% of the historical data.

[0080] The generation of the training set 402, the validation set 404, and the test set 406 can be tailored to a particular application. For example, the training set can be 60% of the historical data, the validation set can be 20% of the historical data, and the test set can be 20% of the historical data. The system 400 can generate multiple sets of features for each of the training set, the validation set, and the test set. For example, if the historical data 464 includes sensor data from 20 sensors (e.g., sensor 126 of FIG. 1 ) and features derived from 10 manufacturing parameters (e.g., manufacturing parameters corresponding to the sensor data from the 20 sensors), the sensor data can be divided into a first set of features including sensors 1-10 and a second set of features including sensors 11-20. The manufacturing parameters can also be divided into multiple sets, e.g., a first set of manufacturing parameters including parameters 1-5 and a second set of manufacturing parameters including parameters 6-10. Either the target inputs, the target outputs, or both can be divided into multiple sets, or none can be divided. Multiple machine learning models, physics-based models, etc. can be trained on different sets of data.

[0081] At block 412, the system 400 performs model training using the training set 402 (e.g., via the training engine 182 of FIG. 1). Training of machine learning models and / or physics-based models (e.g., digital twins) can be accomplished in a supervised learning manner, which involves feeding a training dataset with labeled inputs through the model, observing its outputs, defining an error (by measuring the difference between the output and the label values), and adjusting the weights of the model such that the error is minimized using techniques such as deep gradient descent and backpropagation. In many applications, repeating this process across many labeled inputs in the training dataset results in a model that can produce correct outputs when presented with inputs different from those present in the training dataset.

[0082] For each training data item in the training dataset, the training data item can be input to a model (e.g., a machine learning model). The model can then process the input training data items (e.g., process recipes from past runs) to generate an output. The output can include, for example, predicted performance data for the processed product. This output can be compared to the label of the training data item (e.g., actual performance metrics measured).

[0083] Processing logic may then compare the generated output (e.g., predicted performance) contained in the training data item with the label (e.g., actual performance measurement). Processing logic determines an error (i.e., classification error) based on the difference between the output and the label. Processing logic adjusts one or more weights and / or values ​​of the model based on this error.

[0084] When training a neural network, an error term or Δ may be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters (weights for one or more inputs of the node) for one or more of its nodes. The parameters may be updated in a backpropagation fashion, such that the nodes in the highest layer are updated first, followed by the nodes in the next layer, and so on. The artificial neural network includes multiple layers of "neurons", each layer receiving as input values ​​from neurons in the previous layer. The parameters for each neuron include a weight associated with the value received from each of the neurons in the previous layer. Adjusting the parameters may thus include adjusting the weights assigned to each of the inputs to one or more neurons in one or more layers in the artificial neural network.

[0085] The system 400 can train multiple models using multiple sets of features from the training set 402 (e.g., a first set of features from the training set 402, a second set of features from the training set 402, etc.). For example, the system 400 can train models to generate a first trained model using a first set of features in the training set (e.g., sensor data from sensors 1-10) and generate a second trained model using a second set of features in the training set (e.g., sensor data from sensors 11-20). In some embodiments, the first trained model and the second trained model can be combined to generate a third trained model (e.g., which may be a better predictor than the first or second trained models themselves). In some embodiments, the sets of features used in comparing models may overlap (e.g., the first set of features may be sensor data from sensors 1-15 and the second set of features may be from sensors 5-20). In some embodiments, hundreds of models can be generated, including models with various permutations of features and combinations of models. In some embodiments, hundreds or more models can be stacked together to generate one ensemble model that has greater predictive power than any individual model alone.

[0086] At block 414, the system 400 performs model validation using the validation set 404 (e.g., via validation engine 184 of FIG. 1). The system 400 can validate each of the trained models using a corresponding set of features in the validation set 404. For example, the system 400 can validate a first trained model using a first set of features in the validation set (e.g., sensor data from sensors 1-10) and validate a second trained model using a second set of features in the validation set (e.g., sensor data from sensors 11-20). In some embodiments, the system 400 can validate hundreds of models (e.g., models with various permutations of features, combinations of models, etc.) generated at block 412. At block 414, the system 400 can determine the accuracy of each of the one or more trained models (e.g., via model validation) and can determine whether one or more of the trained models have an accuracy that meets a threshold accuracy. In response to determining that none of the trained models have an accuracy that meets the threshold accuracy, flow returns to block 412, where the system 400 performs model training using a different set of features from the training set. In response to determining that one or more of the trained models have an accuracy that meets the threshold accuracy, flow proceeds to block 416. The system 400 can discard trained models that have an accuracy below the threshold accuracy (e.g., based on a validation set).

[0087] At block 416, the system 400 performs model selection (e.g., via the selection engine 185 of FIG. 1 ) to determine which of the one or more trained models that meet the threshold accuracy has the highest accuracy (e.g., the selected model 408 based on the validation of block 414). In response to determining that two or more of the trained models that meet the threshold accuracy have the same accuracy, flow may return to block 412, where the system 400 performs model training using a further refined training set corresponding to a further refined set of features to determine the trained model having the highest accuracy.

[0088] At block 418, the system 400 performs model testing (e.g., via the test engine 186 of FIG. 1) using the test set 406 to test the selected model 408. The system 400 can test the first trained model using a first set of features in the test set (e.g., sensor data from sensors 1-10) to determine a first trained model (e.g., based on the first set of features of the test set 406) that meets a threshold accuracy. In response to the selected model 408 not meeting the threshold accuracy (e.g., the selected model 408 overfits the training set 402 and / or the validation set 404 and cannot be applied to other data sets, such as the test set 406), flow proceeds to block 412, where the system 400 performs model training (e.g., retraining) using a different training set corresponding to a different set of features (e.g., sensor data from a different sensor). In response to determining that the selected model 408 has an accuracy that meets the threshold accuracy based on the test set 406, flow proceeds to block 420. At least in block 412, the model can learn patterns in past data to make predictions, and in block 418, the system 400 can apply the model to the remaining data (e.g., test set 406) to test these predictions.

[0089] At block 420, the system 400 receives current sensor data 446 (e.g., current sensor data 146 of FIG. 1 ) using the trained model (e.g., selected model 408) and determines (e.g., extracts) predicted performance data 468 (e.g., predicted data 168 of FIG. 1 ) from the output of the trained model. Corrective actions associated with the manufacturing equipment 124 of FIG. 1 can be performed in light of the predicted performance data 468. In some embodiments, the current sensor data 446 can correspond to the same type of features in the historical manufacturing parameter data. In some embodiments, the current sensor data 446 corresponds to the same type of features as a subset of the type of features in the historical manufacturing parameter data used to train the selected model 408.

[0090] In some embodiments, current data is received. The current data may include current performance data 460 (e.g., measurement data 160 of FIG. 1). The model 408 is retrained based on the current data. In some embodiments, a new model is trained based on the current sensor data 446 and the current performance data 460.

[0091] In some embodiments, one or more of operations 410-420 may be performed in various orders and / or with other operations not presented and described herein. In some embodiments, one or more of operations 410-420 may not be performed. For example, in some embodiments, one or more of data partitioning of block 410, model validation of block 414, model selection of block 416, or model testing of block 418 may not be performed.

[0092] A system similar to the system 400 of FIG. 4A can be configured to train, certify, test, and use a physics-based digital twin model. The physics-based model is configured to accept manufacturing parameters (e.g., set points provided to manufacturing equipment) as inputs and provide simulated sensor data (e.g., predicted sensor data) as outputs. Other models related to this physics-based model can follow similar data flows of training, certification, testing, and use. In some embodiments, the model can receive simulated sensor data and measured sensor data and can be configured to output a list of components predicted to contribute to the difference between the two data sets. The model can provide an estimate of how much each component contributes to those variations. For example, the model can provide an estimate of power delivery to the components. Sensor data describing the power delivery can be provided to the physics-based model, or the output of the physics-based model and the sensor data can be provided to another model. The difference between the predicted data and the measured data can indicate components that are aging, faulty, drifting, etc. The partition, training, validation, selection, testing, and use blocks of system 400 may be similarly performed using data of different data types to train additional models. Retraining may also be performed using current simulated sensor data and current measured sensor data. In some embodiments, golden run data may also be provided for partition, training, validation, selection, testing, and use of machine learning or physics-based models.

[0093] 4B is a flow diagram of a method 430 for performing a TTTM analysis using trace sensor data, according to some embodiments. In some embodiments, the operations of method 430 can be performed after various machine learning models, statistical models, digital twin models, etc. to be used as part of the TTTM analysis module have been trained.

[0094] At block 431, a configuration operation is performed. In some embodiments, the configuration operation is performed by a user. In some embodiments, a user may interact with a graphical user interface to configure the TTTM system. In some embodiments, the TTTM system may also be configured using other methods, such as text entry, command line control, etc. In some embodiments, the configuration settings (e.g., a document or file containing data indicative of the configuration settings) may be referenced by a processing device that performs the TTTM analysis iteratively throughout the analysis.

[0095] The configuration settings may include labeling the information and prescriptions, including dates, recipe labels, product labels, equipment and / or sensor labels, etc. The configuration settings may include instructions for utilizing data collected by one or more tools, one or more sensors in association with one or more processing runs or products, etc. The configuration settings may include instructions to the processing device to perform (or not perform) some type of analysis. The configuration settings may indicate which metrics to utilize, what type of summary data to generate, which model (e.g., machine learning model) should be used in the analysis, which features of the TTTM software module should be utilized in the analysis (e.g., whether inlier drift detection should be performed or time trace analysis should be performed), etc. The configuration settings may include instructions for conditional thresholds for performing further analysis, such as control limits for classifying data points as outliers, a threshold number of outliers for triggering drill-down analysis, etc. The configuration settings may include instructions for metric value aggregation and / or indicator aggregation to be performed by the TTTM module. In some embodiments, a user may be able to store one or more "favorite" configuration settings for use in future TTTM analyses. In some embodiments, the configuration settings may include options for visualization, including the type of plot to generate (e.g., box plot, histogram, scatter plot, etc.), the number of plots to generate, threshold metric or index values ​​for triggering generation of plots, design elements of the generated plots, whether best fits or trend lines are calculated and / or displayed on the plots, etc.

[0096] After the configuration settings are determined (or default settings, favorites, etc. are selected), an automated processing procedure 432 is executed. In some embodiments, the TTTM module (guided by the configuration settings) independently executes a number of operations to generate an analysis of the sensor data. The operation of the automated processing procedure 432 is discussed in more detail in connection with Figures 5B-5I, with a brief summary of these operations included herein.

[0097] At block 433, a data collection operation is performed by the processing device. Data is retrieved for analysis by the processing device according to configuration settings. The retrieved data may include trace sensor data, pre-processed sensor data, manufacturing data (e.g., hardware or processing parameters, recipe information, etc.), summary data, etc.

[0098] At block 434, data processing operations are performed by the processing device. In some embodiments, the data processing is performed according to the configuration settings. The data processing may include generating metric values, indicators, aggregated metrics and indicators, trace analysis, drift analysis, and the like. The data processing may include providing the data (e.g., the data collected at block 433) to one or more models to generate outputs. The data processing models may include machine learning models, digital twin models, statistical models, and the like. The data processing operations are discussed in more detail in connection with FIG. 5D.

[0099] At block 435, the processing logic performs a visualization operation. The visualization operation may be performed according to configuration settings. In some embodiments, the generation of the visual representation may be triggered by a condition being met during data processing, e.g., a metric value exceeding a threshold, a certain portion of data points being classified as outliers, etc. In some embodiments, the visual representation may be generated at multiple stages of analysis, e.g., TTTM index values ​​may be plotted, and one or more aggregations of TTTM index values ​​may also be plotted. The visualization may be performed using any convenient plotting scheme, e.g., box plots, histograms, scatter plots, pie charts, etc. The visual representation is discussed in more detail in connection with Figures 6A-6G.

[0100] At block 436, processing logic performs analysis and reporting operations. In some embodiments, the operations of block 436 are provided to a user. In some embodiments, the operations of block 436 may be triggered automatically (e.g., a case evaluation procedure to evaluate the performance of the TTTM module may occur periodically). In some embodiments, the operations of block 436 may be triggered by a user. In some embodiments, the results of the operations of block 436 may inform configuration settings for future analysis (as shown in FIG. 4B by the arrow back to block 431).

[0101] 4C is a flow diagram illustrating a method 450 for generating summary data, according to some embodiments. The generation of summary data can begin with the collection of trace data. As an example, the method 450 operates on all traces from a single run and a single sensor. In other embodiments, data from multiple sensors, multiple runs, etc. can also be used to generate the summary data. In some embodiments, the trace data can be stored separated into multiple process operations, and the data separated by operations can be utilized to generate the summary data. In some embodiments, pre-processing can be performed on this data, such as, for example, smoothing, interpolation, normalization, etc. Pre-processing can be performed at various stages of analysis.

[0102] At block 451, all trace data from a single run is split into multiple processing operations by processing logic. In some embodiments, all trace data and the data split into separate processing operations (e.g., steps of a processing recipe) may be provided to processing logic to perform basic summary operations. The data may be split into multiple processing operations based on timestamps of data points, behavior of the trace data, behavior of reference trace data, etc. The data split into multiple processing operations may be provided to a processing device and further split into temporary and steady state portions. In some embodiments, various operations may be performed in a different order than shown in FIG. 4C, and this may be within the scope of the present disclosure.

[0103] At block 452, the trace data divided into multiple processing operations is further divided into a transient portion and a steady-state portion. A visual representation of the division of the transient portion is shown in FIG. 6A. In some embodiments, a process parameter such as temperature or voltage may have an operational set point. If the characteristic value changes rapidly, it may take an amount of time to reach such a set point. Periods of rapid change in the characteristic value may be isolated as a transient portion of the trace data, and periods where the characteristic value is more stable may be isolated as a steady-state portion of the trace data. In some embodiments, a processing recipe may require the characteristic (e.g., temperature) to change (e.g., increase) in a controlled manner over a period of time. Such periods that are neither steady state nor transient may also be isolated and analyzed. In some embodiments, the slope data may be analyzed as steady-state data. In some embodiments, the slope data may be analyzed as transient data. In some embodiments, the slope data may be analyzed in a manner different from steady-state or transient data. In some embodiments, features from the transient data analysis and / or the steady-state analysis may be utilized to analyze the slope data.

[0104] The transient and steady state portions of the trace data may be separated by the processing device. In some embodiments, the separation of the transient and steady state portions may be performed according to configuration settings. Identification of the transient portions of the trace data may include determining the slope of the trace data, one or more statistical measures of the data (e.g., standard deviation, range, etc.) based on prior knowledge (e.g., time to steady state estimate derived from golden run data), etc. In some embodiments, a sliding window is utilized to separate the transient and steady state portions of the trace data. A window that is smaller (e.g., containing fewer data points) than the trace data associated with the operation may be analyzed. In some embodiments, a point within the window may be determined to be part of the steady state portion or part of the transient portion. In some embodiments, a point within the window may be determined to be part of the transient portion if the standard deviation of the points within the window is greater than a threshold. The window may then be moved to include a different subset of data points of the processing operation trace data. The size of the window, the metric (e.g., slope, standard deviation, quartile value, range, etc.) used to determine whether a portion of the trace data is transient, the threshold at which the determination of transience is made, etc. may all be performed according to configuration settings. In some embodiments, the boundary between the portions designated as steady state and the portions designated as transient may be subjected to further processing, such as modifying a window size, modifying a decision metric, modifying a threshold metric value, etc. The data divided into the transient and steady state (and slope or other type) portions is provided to processing logic to generate summary data. In some embodiments, the steady state and transient portions are provided to processing logic to perform basic data summary operations (block 454). In some embodiments, the transient portions are provided to processing logic to perform digital twin summary operations (block 453).

[0105] At block 453, processing logic performs a digital twin summary operation. In some embodiments, the digital twin summary operation is performed in relation to the temporal portion of the trace data. In some embodiments, a digital twin model of the physical asset is generated. The physical asset may include one or more components of a manufacturing system. The performance of the physical asset may be modeled by the digital twin, and the modeled behavior may be compared to behavior recorded by one or more sensors associated with the manufacturing system. A visual representation of the digital twin data for use in generating the summary data may be seen in FIG. 6B.

[0106] Various characteristics of the temporal portion can be compared to the digital twin output to generate summary data. In some systems, the characteristic value may exceed the set point (e.g., overshoot) before settling to the set point value. Characteristics of the overshoot region of the trace data (e.g., the value of the overshoot, the length of time of the overshoot, the time when the maximum or minimum value of the overshoot is reached, etc.) can be used to generate the summary data. In some embodiments, the characteristic value can exhibit a rise time to reach the set point. The rise time characteristics of the sensor data can be compared to the digital twin data to generate summary data, e.g., the time required for the characteristic value to reach 50% of the indicated value, 90% of the indicated value, etc. In some embodiments, it may take some time for the characteristic value to settle to the set point (e.g., after an overshoot). This settling time can be used to generate summary data (e.g., the time until the value remains within 5% of the set point, within 2% of the set point, etc.). In some embodiments, the final characteristic value reached may differ from the set point. Based on this difference, the summary data can be generated. In some embodiments, the digital twin model can model the characteristics of the closed loop control system. Characteristics such as gain, phase, and bandwidth of the closed loop control system may also be monitored and used to generate summary data. In some embodiments, the characteristic value may fluctuate (e.g., as the set point is approached, the value may go above and below the set point several times before stabilizing). The summary data may include detection of such fluctuations and characteristics of the fluctuations such as frequency, amplitude, etc. In some embodiments, maximum, minimum, slope, average, or other metrics may be included in the summary data associated with the temporal portion.

[0107] At block 454, a basic summary operation is performed by the processing logic. In some embodiments, several portions are provided to the basic summary operation logic (e.g., both transient and steady state data can be provided to the basic summary operation logic). In some embodiments, the basic summary data can include metadata, such as data associated with the processing run. The metadata can include processing run start time, run ID, tool ID, tool name, lot name, lot ID, processing recipe name, product ID, product information (e.g., product design), product number, product count (e.g., number of products since last maintenance event), etc.

[0108] In some embodiments, the basic summary data may include context data, e.g., data associated with a sensor, a process operation, etc. The context data may include a channel or sensor name or ID, a process operation number (e.g., step number), a timestamp of the process operation, an operation name or ID, an operation type (e.g., an etch operation, a deposition operation, etc.), a manufacturing subsystem name or ID, etc.

[0109] In some embodiments, the basic summary data may include statistical information associated with the trace data. The statistical information may include data mean, median, minimum, maximum, range, quartiles and other percentile information, standard deviation, skew, kurtosis, a flag indicating whether the data is within a control range, and the like. The basic summary data may be generated according to configuration settings. At block 455, the summary data is stored for future use, for example, by the TTTM module. The summary data may be stored in a data store, for example, data store 140 of FIG. 1.

[0110] 5A-5I are flow diagrams of methods 500A-I associated with generating data to trigger corrective actions, according to certain embodiments. Methods 500A-I can be performed by processing logic, which can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, processing device, etc.), software (e.g., instructions executed on a processing device, general-purpose computer system, or dedicated machine), firmware, microcode, or combinations thereof. In some embodiments, methods 500A-I can be performed in part by prediction system 110. Method 500A can be performed in part by prediction system 110 (e.g., server machine 170 and dataset generator 172 of FIG. 1, dataset generator 372 of FIG. 3). Prediction system 110 can use method 500A to generate a dataset for at least one of training, validating, or testing a machine learning model, according to embodiments of the present disclosure. Methods 500B-500I can be performed by prediction server 112 (e.g., prediction component 114). In some embodiments, a non-transitory storage medium stores instructions that, when executed by a processing device (e.g., prediction system 110, server machine 180, prediction server 112, etc.), cause the processing device to perform one or more of methods 500A-I.

[0111] For ease of explanation, methods 500A-I are shown and described as a series of operations. However, operations in accordance with the present disclosure may occur in various orders and / or simultaneously, with other operations not shown and described herein. Moreover, not all illustrated operations may be performed to implement methods 500A-I in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that methods 500A-I may alternatively be represented via a state diagram or events as a series of interrelated states.

[0112] FIG. 5A is a flow diagram of a method 500A for generating a dataset for a model for generating predicted data (eg, predicted data 168 of FIG. 1) according to certain embodiments.

[0113] Referring to FIG. 5A, in some embodiments, at block 501, processing logic performing method 500A initializes a training set T to an empty set.

[0114] At block 502, processing logic generates a first data input (e.g., a first training input, a first validation input) that may include one or more of sensor data (e.g., historical sensor data 144 of FIG. 1 , historical sensor data 344 of FIG. 3 ), production parameters (e.g., historical production parameters 152 of FIG. 1 ), etc. In some embodiments, the first data input may include a first set of features for a type of data, and the second data input may include a second set of features for a type of data (e.g., as described with respect to FIG. 4A ).

[0115] In some embodiments, at block 503, the processing logic generates a first target output for one or more of the data inputs (e.g., the first data input). In some embodiments, the first target output is performance data (e.g., metrology data of processed products, faulty components, etc.). In some embodiments, the input data can be in the form of sensor data and the target output can be a list of likely faulty components, as in the case of machine learning models configured to run in conjunction with a physics-based digital twin model. In some embodiments, no target output is generated (e.g., an unsupervised machine learning model can group or find correlations in the input data rather than requiring a target output to be provided).

[0116] At block 504, processing logic may generate mapping data indicative of an input-output mapping. The input-output mapping (or mapping data) may reference data inputs (e.g., one or more of the data inputs described herein), target outputs for the data inputs, and associations between the data inputs and the target outputs. In some embodiments, block 504 may not be performed, such as in connection with machine learning models where no target output is provided.

[0117] At block 505, processing logic, in some embodiments, adds the mapping data generated at block 504 to the dataset T.

[0118] At block 506, processing logic branches based on whether the dataset T is sufficient for at least one of training, validating, and / or testing the model 190. If the dataset T is sufficient, execution proceeds to block 507; if not, execution also returns to block 502. It should be noted that while in some embodiments the dataset T may be determined to be sufficient based solely on the number of inputs in the dataset, and in some embodiments the number of inputs mapped to outputs, in some other embodiments the dataset T may be determined to be sufficient based on one or more other criteria in addition to or instead of the number of inputs (e.g., a measure of diversity of the data examples, accuracy, etc.).

[0119] At block 507, the processing logic provides (e.g., to the server machine 180) a dataset T for training, certifying, and / or testing the machine learning model 190. In some embodiments, the dataset T is a training set and is provided to the training engine 182 of the server machine 180 to perform training. In some embodiments, the dataset T is a certification set and is provided to the certification engine 184 of the server machine 180 to perform certification. In some embodiments, the dataset T is a test set and is provided to the testing engine 186 of the server machine 180 to perform testing. For example, in the case of a neural network, the input values ​​of a given input-output mapping (e.g., the numerical values ​​associated with the data inputs 310) are input to the neural network, and the output values ​​of the input-output mapping (e.g., the numerical values ​​associated with the target outputs 320) are stored in the output nodes of the neural network. The weights of the connections in the neural network are then adjusted according to a learning algorithm (e.g., backpropagation, etc.), and the procedure is repeated for other input-output mappings in the dataset T. After block 507, the model (e.g., model 190) may be subjected to at least one of training using the training engine 182 of the server machine 180, certification using the certification engine 184 of the server machine 180, or testing using the testing engine 186 of the server machine 180. The trained model may be implemented by the prediction component 114 (of the prediction server 112) to generate prediction data 168 for performing corrective actions associated with the signal processing or manufacturing equipment 124.

[0120] 5B is a flow diagram of a method 500B for utilizing one or more models to determine corrective actions to be taken, according to some embodiments. The operations of FIG. 5B may be included in the automated processing operations described in connection with FIG. 4B.

[0121] Referring to FIG. 5B, at block 510, the processing logic performs preparatory operations associated with generating tool-to-tool matching (TTTM) information indicative of corrective actions. The preparatory operations may be triggered by manual input (e.g., upon recognition of suboptimal performance, in response to suboptimal metrology data of one or more finished products, etc.), at scheduled timing intervals, or the like. The TTTM analysis preparation may include data collection logic. Data (e.g., sensor trace data) may be retrieved from a data store. The data collection may be performed according to configuration settings. The collected data may include the date the process was run, tool ID of the manufacturing equipment, recipe name, data indicative of key operations, sensors, or parameters, etc. The configuration settings may include instructions for which of these types of data to collect and procedures for preprocessing (e.g., data cleansing, resampling, averaging, smoothing, etc.). The processing logic may perform data preprocessing as part of the preparatory operations.

[0122] The preparation operation may further include the processing logic generating summary data associated with, for example, a manufacturing process. The data summarization operation is discussed in detail in connection with FIG. 4C. The data summarization may be performed according to configuration settings. In some embodiments, the summary data is generated from the trace sensor data. The trace sensor data may be separated into multiple portions, and summary data is extracted separately from each portion. For example, the trace sensor data may be separated into one or more temporary portions and one or more steady-state portions. In some embodiments, a sliding window is used to provide statistics of data values ​​of a portion of the trace. If a particular statistical marker (e.g., standard deviation, variance, etc.) of a point within the window is greater than (or less than) a threshold, the point within the window may be determined to be included in the steady-state portion of the trace data, the temporary portion of the trace data, etc. Other metrics (e.g., slope of the data, time threshold, etc.) may also be used to specify the steady-state and temporary portions of the trace data. In some embodiments, the steady-state and temporary data are processed differently by the summarization logic. The configuration settings may include adjustments regarding how the steady-state and temporary portions are identified, such as window size, values ​​or identification information of a metric that determines which portion the window belongs to, etc. In some embodiments, different data is included in the summary data for different portions of the trace data. In some embodiments, a physics-based (e.g., digital twin) model of the component is used to generate a prediction of behavior. In some embodiments, a difference between the digital twin prediction and the measured sensor data is utilized to generate the summary data. For example, the digital twin model may predict that the voltage supplied to a component (e.g., an electrostatic chuck) may have a particular rise time (e.g., time to reach 90% of a target value), characteristic overshoot (e.g., percent of peak voltage relative to a target value), and settling time (e.g., time to drift to reach a steady value).The summarization logic may include recording the difference between the measured temporal portions of the sensor data and the predictions of such metrics by the digital twin model.

[0123] Summary data can include metadata, context data, statistics, and digital twin-assisted summary data. Metadata can include data describing the manufacturing process (e.g., the state in which a semiconductor wafer was processed). Metadata can include run start time, recipe ID, tool ID associated with manufacturing equipment, tool name, lot name and / or ID, product ID, product information, product number, number of products made by the tool, etc. Context data can include data describing the trace data such as sensor ID, recipe operation number, timestamp data, subsystem, operation type, etc. Statistical data can include mean, minimum and maximum, percentile (e.g., quartile) values, variance, skew, kurtosis, range, interquartile range, lower and upper control limits (e.g., calculated from other statistics such as three standard deviations of the mean or some multiple of the interquartile range outside the percentile values), etc. Digital twin assisted summary data can include overshoot (e.g., the percentage difference between a target value of a characteristic and the peak value of the characteristic), rise time (e.g., the time for a characteristic to adjust to 90% of the target value), settling time (e.g., the time for a measured value to fall within some threshold range of a target value), steady state error (e.g., the percentage difference between the final measured value and the target value), closed loop control system parameters such as gain margin (e.g., the gain difference between no gain and the system gain at a frequency where the phase offset between the input and output is 180°), phase margin (e.g., the phase offset difference between the system phase and 180° at a frequency where there is no gain), and bandwidth, fluctuation detection, fluctuation frequency, fluctuation amplitude, etc.

[0124] At block 511, processing logic identifies one or more existing models that fit the data to be analyzed. The existing models may be models that have been trained using historical data. The suitability of the existing models may be determined based on being trained on the same set of manufacturing equipment, the same type of manufacturing equipment, for the same or similar process, etc. If there is no existing matching model, flow proceeds to block 512. If there is an existing matching model, flow proceeds to block 513.

[0125] At block 512, processing logic performs training operations associated with performing TTTM analysis. Training operations are discussed in more detail in connection with FIG. 5C. The training operations may include retrieving data associated with one or more manufacturing processes, such as from data store 140 of FIG. 1. The training operations may include retrieving and / or generating golden run sensor data, retrieving summary data, retrieving trace data, etc. The training operations may be performed according to configuration settings.

[0126] At block 513, processing logic performs inference operations associated with the TTTM analysis. Inference operations are discussed in more detail in connection with FIG. 5D. The inference may include analysis using one or more models. The models may include physics-based models, digital twin models, machine learning models, etc. The inference may be performed according to configuration settings. The inference operation may be performed in response to a previous inference operation, e.g., performing further drill-down analysis in response to an output of the previous operation satisfying one or more criteria. Results of the inference may include corrective actions, system, or component instructions for further follow-up, scheduled maintenance, etc. The inference operation may include visualization of the results.

[0127] At block 514, processing logic performs a case evaluation operation associated with the TTTM analysis. The case evaluation procedure can be used to generate information indicative of the performance of the TTTM analysis procedure. The performance of various models and algorithms can be evaluated. The case evaluation operation can include comparing predicted performance metrics to measured performance metrics. In some embodiments, the case evaluation can be triggered by a user. In some embodiments, the case evaluation can be triggered by a result, such as a result of the TTTM analysis outside a threshold range, a result of a measurement outside a threshold range of similarity to the TTTM analysis prediction, etc. In some embodiments, the case evaluation can be performed according to a case evaluation schedule (e.g., every n process times, a selected frequency, etc.). Data associated with the manufacturing process (e.g., sensor data, summary data, etc.) can be provided to the processing logic along with the output of the TTTM analysis (e.g., one or more components with predicted failures, subsystems with predicted drift or aging, predicted metrology data of the finished product, visualization of the health of the components or subsystems, etc.). Measured performance data associated with the TTTM analysis output (e.g., predicted maintenance results of faulty components, measured measurements of a product, etc.) may also be provided to processing logic that performs the case evaluation.

[0128] The case evaluation operation may include and / or may be directed to determining whether a TTTM module (e.g., one or more models included in the TTTM analysis) should be retrained. The processing logic may evaluate whether to perform retraining based on one or more of a number of metrics. In some embodiments, retraining may be performed based on a particular portion of the predictions, e.g., a portion of the predictions, not meeting an accuracy threshold over a period of time. In some embodiments, retraining may be performed based on multiple consecutive products not meeting an accuracy threshold. In some embodiments, retraining may be performed based on known changes to processing conditions, such as recipe updates, incoming material updates, performed chamber maintenance, etc. The processing logic may perform the evaluation using data from cases with sufficiently accurate TTTM predictions and cases with unsatisfactory TTTM predictions. The processing logic may perform the evaluation using both positive cases (cases where the TTTM analysis predicted failures, aging, drift, etc., e.g., true positives and false positives) and negative cases (cases where the TTTM analysis did not predict failures, e.g., true negatives and false negatives).

[0129] The processing logic may further evaluate the false cases (e.g., false positives, false negatives). The false cases may be evaluated for root causes, e.g., for possible modifications to the TTTM model to reduce the false cases. In some embodiments, the effect of adjusting weights of operations, sensors, etc. may be evaluated. In some embodiments, the effect of adjusting control limits may be evaluated. In some embodiments, the effect of removing sensors, operations, etc. from future TTTM modeling may be evaluated. The processing logic may store the results of the case evaluation. The processing logic may generate recommendations, e.g., to update or retrain all or a portion of the TTTM analysis module.

[0130] At block 515, the processing logic performs retraining of the TTTM module. The retraining may be performed according to the recommendations generated during the case evaluation at block 514. For example, the processing logic may generate a recommendation to retrain the sensor weights during the case evaluation operation. During the retraining operation, the processing logic may implement the recommendation by providing additional data for training one or more models utilized for the TTTM analysis. In some embodiments, the retraining may include model validation and / or testing. In some embodiments, the retraining operation may include training a new model using different sensor data (e.g., a different set of sensors as described in connection with FIG. 4A). In some embodiments (e.g., for recommendations generated during the case evaluation operation), a retraining operation may not be performed. After retraining, the flow may return to the inference stage (block 513) for the TTTM analysis of additional products, processes, manufacturing equipment, etc.

[0131] 5C is a flow diagram illustrating a method 500C of performing training operations of a TTTM analysis method, according to some embodiments. These operations may be directed to preparing the TTTM analysis for inference as opposed to training a particular machine learning model, physics-based model, etc. At block 521, data is retrieved for use in the TTTM analysis. The retrieved data may include trace data, pre-processed trace data, summarized data, etc. The retrieved data may include data identified, generated, stored, etc. in preparation operations for the TTTM analysis. The data may be retrieved according to configuration settings, such as settings indicating which analysis techniques, which trained models, etc., should be used in the TTTM analysis.

[0132] At block 522, processing logic determines whether one or more golden data sets are to be used in the TTTM analysis. Golden data can include trace data associated with a successfully manufactured product, properly functioning equipment, etc., cleaned or pre-processed trace data, summary data, etc. The use of one or more golden data sets can be based on configuration settings, etc., and one or more models are to be used as part of the TTTM analysis, etc., based on the configuration settings. If a golden data set is not to be used, the flow proceeds to block 526. If one or more golden data sets are to be used, the flow proceeds to block 523. At block 523, processing logic determines whether one or more golden data sets are available (e.g., data designated as golden data is stored in data store 140). If golden data is available (e.g., golden data for all sensors, processes, systems, etc. for which golden data is to be used), the flow proceeds to block 525, where one or more golden data sets are retrieved. If a golden data set is not available (e.g., one or more golden data sets to be used in the TTTM analysis are not available), the flow proceeds to block 524.

[0133] At block 524, processing logic generates one or more golden data sets. Processing logic may select data (e.g., from data store 140) to act as the one or more golden data sets. Data may be selected as golden data based on one or more indications that the data corresponds to acceptable processing parameters (e.g., processing parameters that facilitate the creation of a product within manufacturing tolerances). In some embodiments, a set of tools, chambers, and / or manufacturing equipment that have been recently qualified (e.g., recently maintained, recently installed, etc.) may be designated as a golden tool. Data from a process performed by the golden tool may be designated as golden data. In some embodiments, data indicative of the creation of an acceptable product may be associated with sensor data (e.g., metrology data, classification or quality data, etc.). Sensor data associated with the creation of an acceptable product (including pre-processed sensor data, summarized sensor data, etc.) may be designated as golden run data. In some embodiments, one or more sensors may have associated golden run data. In some embodiments, one or more processing operations may have associated golden run data.

[0134] At block 526, the processing logic defines control limits. The control limits may be set according to configuration settings. In some embodiments, the control limits may be established by user input (e.g., an acceptable processing range may be selected by a user). In some embodiments, the control limits may be calculated by the processing logic (e.g., three standard deviations from the mean). In some embodiments, a composite system may also be used (e.g., three standard deviations from the mean, but within user-defined limits).

[0135] 5D is a flow diagram illustrating a method 500D of performing inference operations of the TTTM analysis method, according to some embodiments. At block 530, one or more models to be used in the TTTM analysis are imported. These may include trained machine learning models, physics-based models, digital twin models, statistical models, etc. The models may be imported according to configuration settings.

[0136] At block 531, the processing logic performs an outlier detection operation. The summary data may be compared to control limits to determine whether a data point (e.g., summary data associated with a set of trace data) is an outlier. Each sensor in each processing operation (e.g., recipe step) may be considered individually. A data point may be considered an outlier if any one summary data is outside the control limits, if multiple summary data are outside the control limits, if the overall weight of the summary data (e.g., summaries with greater predictive power are given a greater weight) is outside the control limits, etc. In some embodiments, the individual outlier signals may be aggregated to generate an index (e.g., a TTTM index) indicative of the performance and / or quality of a chamber, tool, process, facility, product, set of manufacturing equipment, etc.

[0137] In some embodiments, the generation of the TTTM index may depend on the weights of the sensors and process operations, as well as the control limits. Some sensor-operation combinations may be found to have higher predictive power than other combinations. For example, for a semiconductor substrate processing operation utilizing plasma, a sensor indicating a parameter of the plasma may have a high predictive power of chamber performance. For a processing operation in which plasma processing is not used, the same sensor may not provide information that is easily correlated with tool performance. A weighting factor may indicate the importance of the contribution of the associated data to performance.

[0138] In some embodiments, data points classified as high outliers (e.g., above the upper control limit), low outliers (e.g., below the lower control limit), and inliers (e.g., between the upper and lower control limits) may be treated differently. In some embodiments, one or more weighting factors may be applied to the sensor data, summary data, etc. The weighting factors (e.g., weights) may indicate the impact of the property values ​​measured in the process chamber on performance. In some embodiments, weights may be applied to sensors (e.g., sensors closer to the substrate may produce data that correlates more closely to performance). In some embodiments, weights may be applied to process operations (e.g., sensor data during an etch operation may correlate more closely to the performance of an etch recipe than operations where no etching occurs, such as warm-up or cool-down operations). In some embodiments, weights may be applied to process runs (e.g., process runs with known problems may be given a lower weight). In some embodiments, weighting factors may be applied to types of data (e.g., types of sensors such as temperature sensors, types of summary data such as averages may be weighted more heavily than standard deviations, summary data from temporary portions may be weighted differently than steady state portions of trace data, etc.), In some embodiments, weighting factors may be applied to combinations of conditions such as a particular sensor during a particular operation (e.g., a sensor that measures the voltage applied to generate a plasma may produce results that are more highly correlated with performance of a processing operation involving a plasma, but may not be as correlated with performance in operations without a plasma).

[0139] In some embodiments, the weights can be used to generate the TTTM index. In some embodiments, the weights can be selected such that the sum of the weights from all sensors for each processing operation (e.g., processing step) can be normalized (e.g., equal to 1). Outlier sensors for an operation contribute zero to the TTTM index value, while inliers can contribute to the weight. If all sensors are inliers, the TTTM index can be 1 (or can be scaled to a different value). A TTTM index value lower than 1 (or a scaled value) can indicate that further analysis is recommended for that manufacturing process, operation, tool, equipment, etc.

[0140] In some embodiments, the weights can be assigned (or the weights can be assisted in the assignment of weights) by a weight discovery model. In some embodiments, the weight discovery model can include a machine learning model. The machine learning model can be trained using data associated with the sensors (e.g., summary data) as training inputs and performance data (e.g., measurement data) as target outputs. Training the machine learning model adjusts the weights connecting the input data to the target output data. The machine learning model can output weights connecting the operation / step combinations to performance. The weights can be used in TTTM analysis, such as as an output by the user-tuned machine learning model. In some embodiments, a partial least squares model can be used to generate the weight function. In some embodiments, a stacked partial least squares model can be used (e.g., multiple partial least squares models arranged as an ensemble model).

[0141] At block 532, processing logic performs a sensor contribution calculation operation. In some embodiments, the performance of the operations of blocks 532-535 may depend on the value of the TTTM index. For example, the processing logic may be configured to perform further analysis (e.g., drill-down analysis to explore root causes) if the TTTM index meets a threshold (e.g., falls below a threshold established in the configuration settings). At block 532, sensor contribution information may be generated that indicates "how much" one or more data points are outliers. The previous operations include labeling if the data point is within the control limits. The sensor contribution value indicates how much the outlier is outside the control limits. An exemplary sensor contribution metric is the severity distance. The severity distance is a measure of how much a data point (e.g., a summary data point) is outside the control limits. The severity distance indicates the relationship between a data point (or an aggregation of data points) and an associated control range. In some embodiments, the severity distance may be generated for one or more sensor / operational data points for a manufacturing run (e.g., a single product). The severity distance can be based on one or more summary data (e.g., one or more types of summary data). In some embodiments, the calculation of the severity distance can include a calculation of how far one or more data points are outside of the control limits. In some embodiments, the calculation of the severity distance can include a calculation of how far a data point is outside of the control limits relative to the size of the control range (e.g., a temperature sensor can be assigned a control range of 20-40 degrees, and a data point at 50 degrees can be calculated to be 50% of the size of the control range above the upper control limit). For example, a data point below the lower control limit (e.g., below the control range) can have a severity distance calculated as follows:

[0142]

number

[0143] An aggregated metric value or indicator may be generated based on one or more severity distances. An exemplary aggregated metric is a severity score. The severity score may include an aggregation of severity distances for multiple process runs (e.g., for several manufactured products). The severity score may indicate the likelihood that a particular sensor / operation pair (e.g., a sensor measuring a voltage applied to generate a plasma in a plasma etch processing operation) produces data indicative of a chamber failure, chamber drift, etc., over several runs. Calculation of an aggregated severity distance, such as a severity score, may include a coefficient indicating the frequency of outlier data points. For example, the severity score may be calculated as follows:

[0144]

number

[0145] In some embodiments, the metric values ​​can be further aggregated to aid in the analysis. For example, the processing logic can aggregate severity scores (which in some embodiments correspond to a single sensor and a single operation) across strategy operations to find a severity indicator per sensor (e.g., by summing the product of operation weights and severity scores across the operations of the process). Such analysis can be triggered by a severity score reaching a threshold (e.g., a quality indicator, quality score, severity score, etc. that meets a criterion), a number or portion of severity scores that meet one or more thresholds, etc. The trigger conditions for aggregating data and generating new indicators can be stored as configuration settings. Aggregating severity scores in this manner can provide additional information to problematic or faulty sensors. Summing across sensors can be performed similar processes to determine whether one or more operations of the manufacturing process are performing suboptimally. In some embodiments, the TTTM indices (which in some embodiments are associated with one run or product and one processing operation) can be aggregated to generate a per-run TTTM indices, for example by adding the product of the TTTM indices (e.g., per run, per operation) and the weights of the operations in multiple operations (e.g., warm-up, etch, deposition, and cool-down operations). This can assist in determining whether a particular operation of a manufacturing process is performing suboptimally. A similar procedure can be performed to isolate faults, for example, from a single run (e.g., a single semiconductor wafer). Various aggregations of the indices can be used by processing logic to recommend that components or subsystems be maintained or replaced, can be fed as input to a machine learning model to generate corrective action recommendations, and the like. Corrective actions can include scheduling preventive maintenance, updating processing recipes, sending alerts to users, and the like. In some embodiments, clustering analysis can be performed on the indices and / or metric values. Clustering analysis is discussed in more detail in connection with FIG. 5H.Visualization of the clustering is discussed in more detail in relation to block 535.

[0146] At block 533, the processing logic performs inlier drift detection. A method of inlier drift detection is described in detail in connection with FIG. 5I. TTTM index analysis, severity distance, severity score, etc. can be associated with the outlier data. Inlier data (e.g., data within the control limits) can also provide valuable information for analysis. Inlier analysis can be performed using summary data. In some embodiments, summary data from a run (e.g., one product, such as one semiconductor wafer) can be compared to golden summary data (e.g., summary data records during the creation of a product that meets a performance threshold). In some embodiments, summary data from a series of manufacturing runs (e.g., a day, a week, a month, etc. of processing) can be compared to golden summary data from a series of manufacturing runs.

[0147] The comparison between the golden data and the data to be analyzed can be performed using any technique that compares data sets. In some embodiments, techniques that quantify the similarity of time series can be utilized. One metric that can be used is to pre-process and / or clean the data such that each data point of the data set to be analyzed has a corresponding data point of the golden run data (e.g., data with the same timestamp, data associated with the same processing operation, etc.). The difference between the values ​​of the corresponding data points can then be aggregated (e.g., summing the squares, absolute values, etc.). The metric can be further processed, e.g., a square root is taken from the summed squares, divided by the number of data points to normalize, and so on. The higher the value of the metric, the more dissimilar the two data sets are. In some embodiments, a discrete Fourier transform can be performed on both data sets, and similarly, the corresponding Fourier coefficients can be compared to the above analysis comparing corresponding data points. In some embodiments, an autoregressive model can be utilized to compare the data sets. An autoregressive model is representative of a random process, where the value of a data point of the process depends on the value of the previous data point. An autoregressive model can be used to determine the likelihood that the data set to be analyzed and the golden data set are associated with the same process (eg, processing conditions, chamber quality, etc. are the same between the two sets).

[0148] In some embodiments, an elastic dissimilarity measure may be utilized to determine a similarity value between two data sets. In some embodiments, dynamic time warping may be utilized in the inlier drift analysis. Dynamic time warping works by aligning or warping the data sets in the time domain such that the accumulated cost of alignment is minimized. A cost function may be determined according to configuration settings. The elastic dissimilarity measure may be considered as adjusting the spacing between data points in one data set to better align with the second data set. The elastic dissimilarity measure only adjusts the spacing between data points, not the order of the data points. The elastic dissimilarity measure may return a metric or index value that indicates how much spacing adjustment is performed to bring the two data sets into optimal alignment (e.g., as determined by minimizing a cost function). In some embodiments, after elastic warping, each point of the first data set may be associated with a point of the second data set. For example, the data to be analyzed may be warped into optimal alignment with the golden data. After warping, each point of the data to be analyzed can be assigned a "nearest" point in the golden run dataset (e.g., some of the points in the analyzed set can be assigned the same nearest point in the golden dataset). In some embodiments, each data point of both datasets can be spliced ​​to one or more nearest points in the other dataset. For example, each point of the warped dataset being analyzed can be assigned a nearest point in the golden dataset. This assignment can include some points of the golden dataset being associated with multiple points in the dataset to be analyzed. The assignment can also include some points of the golden dataset not being assigned as nearest neighbors to some points in the analyzed dataset. Each data point in the golden set can then be assigned to a nearest point in the time-warped dataset to be analyzed.If the two datasets are aligned (e.g., the warping cost function is minimized when each of the data points 1...n in the dataset to be analyzed is nearest neighbor to the corresponding point 1...n in the golden dataset), the number of nearest associations may be equal to the number of data points in the set. If the two datasets are significantly out of alignment, one or more points in each dataset may have some points designated as nearest neighbors in the other dataset. In this case, the number of nearest associations may be greater than the number of data points in the set. The increased number of associations may be used as a metric indicating how well warping produces an optimal alignment of the two datasets (in some embodiments, with preprocessing such as normalization, subtraction, etc. of the number of data points). Inlier drift detection and elastic dissimilarity measures are further discussed in connection with Figures 5I and 6D-E.

[0149] In some embodiments, each tool, chamber, set of manufacturing equipment, etc. may operate somewhat differently. Each tool, chamber, set of equipment, etc. may have its own associated golden data set or data sets. Each type of process (e.g., each target product design, each recipe, etc.) may have its own associated golden data set or data sets.

[0150] Inlier drift detection operations can be used in taking corrective actions. In some embodiments, an inlier drift detection metric above a certain threshold (in some embodiments, by configuration setting) can trigger corrective actions. Corrective actions can include providing an alert to a user, scheduling maintenance, updating a policy, etc.

[0151] At block 534, additional features of the TTTM module may be utilized by the processing logic for further analysis. Use of the additional features may be subject to configuration settings. In some embodiments, values ​​of various metrics may be used in determining whether additional features should be utilized. The additional features may assist in drill-down analysis to isolate causes of chamber failures, drift, unexpected results, poor product quality, etc. The additional features may be used to determine corrective actions to be taken. The additional features may include providing data associated with the manufacturing process to one or more models (e.g., machine learning, physics-based, etc.) to provide additional analysis.

[0152] In some embodiments, additional features of the TTTM module may include trace data analysis, e.g., a trace data exploration analysis module. Trace data analysis may include the use of a trained model (e.g., a machine learning model). In some embodiments, trace data analysis may be performed by providing trace data to a trained machine learning model and receiving output from the machine learning model indicating corrective actions, equipment components to be maintained or replaced, faults in subsystems of manufacturing equipment, etc. In some embodiments, an autoencoder model may be used to perform the trace data analysis. Autoencoder operation is discussed in more detail in connection with FIG. 7. An example of a visual representation associated with the results of trace data analysis is discussed in more detail in connection with FIGS. 6F-6G.

[0153] Trace data analysis can include data preparation operations, such as data cleaning, reformatting, normalization, interpolation, etc. The trace data can then be fed to a trained model. The trained model can create output indications of sensors, subsystems, products, etc. that are not performing in an expected manner. In some embodiments, trace data associated with many (e.g., all) sensors associated with one product processed by manufacturing equipment can be fed to the trained model. A set of data associated with a set of products can be fed to the trained model. The model can generate data indicative of sub-optimal predicted product performance, measurements, characteristics, etc. based on the trace data analysis of the trace sensor data.

[0154] At block 535, the processing logic performs a result visualization operation. Metrics, analyses, etc. performed or generated by the TTTM analysis process may be incorporated into one or more visualization procedures, and in some embodiments, the processing logic may perform the visualization operation according to configuration settings. Indicators, metrics, etc. may be visualized in any convenient or suitable manner, such as scatter plots, histograms, box plots, line graphs, etc.

[0155] In some embodiments, the data may be clustered for visualization. In some embodiments, severity distance may be utilized for clustering and visualization of outliers. Data from multiple runs and multiple sensors may be utilized for clustering and visualization. In some cases, the majority of the sensor data is inlier data (e.g., within the control limits). The data may be preprocessed to remove redundant data (e.g., all inlier data may be removed). In some embodiments, dimensionality reduction may be applied to simplify the remaining outlier data. Dimensionality reduction may aid in visualization of the data, for example, reducing the data to two or three dimensions for more convenient analysis. In some embodiments, nonlinear dimensionality reduction models may be utilized. Nonlinear dimensionality reduction models may capture correlations that linear dimensionality reduction models (e.g., principal component analysis) do not capture. Dimensionality reduction models may convert affinities or similarities between data points into probabilities that the data points are located near each other in a reduced dimensional space (e.g., by Euclidean distance, Minkowski distance, correlation distance, or another distance function). In some embodiments, the dimensionality reduction model may include a t-distribution type probabilistic nearest neighbor embedding algorithm. In some embodiments, parameters that control the operation of the dimensionality reduction model may be stored as configuration settings, such as perplexity (a metric that determines the number of nearest neighbors expected in a cluster), distance metrics in high dimensional spaces, etc.

[0156] In some embodiments, the dimensionality reduced data can be clustered using a clustering model. In some embodiments, a clustering model can be selected that self-selects the number of clusters in the dataset. In some embodiments, density-based spatial clustering of applications with noise clustering algorithms can be utilized. Parameters that determine the details of the operation of the clustering model can be stored as configuration settings (e.g., metrics affecting clustering distance, cluster size, distance metric, etc.).

[0157] In some embodiments, data points (e.g., clustered and dimension-reduced outlier points) can have contributions to a ranked severity score (e.g., severity distances of sensors that contribute to the severity score associated with the outlier point). The sensors that contribute most to a cluster (e.g., sensors represented in multiple data points in a cluster, sensors that contribute to one cluster but not another cluster, etc.) can be used to label the cluster. In some embodiments, sensors and associated data can be assigned to subsystems of the manufacturing equipment (e.g., heater power subsystem, chamber temperature subsystem, chamber pressure subsystem, etc.). Clusters can be labeled by the subsystem that contributes most severely to the outlier condition of the point associated with the cluster. In this manner, clusters can be grouped by root cause (e.g., subsystem failure). Outlier clustering can be used in taking corrective actions, for example, to provide alerts to users, schedule maintenance, adjust process strategies, etc.

[0158] The processing logic may generate a visual representation related to the inlier drift detection. The results of the elastic dissimilarity scaling model may be visualized. The changes may be represented by a scatter plot, a bar graph, a line graph, etc., the association of data points may be plotted on a plot showing the relationship between dataset 1 and dataset 2, or other methods of visualizing the inlier drift detection data may be used.

[0159] The processing logic can generate a visual representation associated with the anomalous trace detection. In some embodiments, sensors can be grouped into subsystems, and metrics describing deviations from expected sensor results can be aggregated into subsystem metrics. Subsystem analysis metrics per subsystem, per run (produced product) can be displayed, for example, as a heat map.

[0160] In some embodiments, the visual representation may be provided to the user as part of a graphical user interface. In some embodiments, additional data may be made available through user interaction with the graphical user interface. In some embodiments, additional analysis may be performed based on the user's interaction with the graphical user interface. For example, a heat map displaying the results of an anomalous trace detection analysis may be based only on a single metric (e.g., per subsystem, per run), but upon selecting a data point in the visual representation, more data may be displayed (details of the difference between the analyzed trace data and the expected trace data, details of the output of the machine learning model, details of recommended corrective actions, etc.). In another example, a visualization tool may display representations of multiple TTTM metrics for various tools, products, etc. A user may select a particular representation (e.g., one displaced from a group of other points) and additional analysis (e.g., inlier drift detection, anomalous trace detection, etc.) may be performed on the data associated with that data point.

[0161] 5E is a flow diagram of a method 500E of performing corrective action based on a metric indicative of performance of a processing chamber, according to some embodiments. At block 540, processing logic receives trace sensor data associated with a first manufacturing process of a manufacturing chamber. The trace sensor data may have been collected by the manufacturing chamber during processing of a product, such as a semiconductor wafer.

[0162] At block 542, the processing logic processes the trace sensor data to generate summary data associated with the trace sensor data. In some embodiments, generating the summary data includes identifying at least one steady state portion and at least one transient portion of the trace sensor data. The transient portion includes a set of trace data associated with a time period during which the characteristic value in the chamber changes (e.g., the variance is above a threshold). The steady state portion includes a set of trace data associated with a time period during which the characteristic value measured by the sensor does not change (e.g., the variance is below a threshold). In some embodiments, a sliding window is utilized to distinguish the steady state portion from the transient portion. In some embodiments, determining whether a set of data points (e.g., points within a window) of the trace sensor data belongs to the transient portion or the steady state portion can include determining whether a statistical measure (e.g., standard deviation) of the set of data points meets a threshold. The transient portion can include a set of points whose standard deviation is above a threshold, and the steady state portion can include a set of points whose standard deviation is below a threshold. Determining whether a set of points belongs to the transient portion or the steady state portion can include consideration of a statistical measure of adjacent windows of data points (e.g., overlapping sets). In some embodiments, a processing recipe includes multiple processing operations. In some embodiments, trace data from a sensor of a process operation (e.g., one or more process operations of a process recipe) can include a temporal head (e.g., when conditions in a fabrication chamber are adjusted to a set point), a steady state portion (e.g., when conditions are maintained at the set point), and a temporal tail.

[0163] In some embodiments, summary data is generated separately for each portion (e.g., each steady-state portion, each temporary portion, etc.). The summary data for the steady-state portion may include a statistical description of the data, e.g., mean, maximum, minimum, range, standard deviation, kurtosis, etc. The summary data for the temporary portion may include a statistical description. The summary data for the temporary portion may further include a metric generated considering the output of the digital twin model. A digital twin model of one or more components of the manufacturing chamber associated with one or more sensors may be generated (e.g., the sensor may measure the power supplied to the component, and a digital twin model of the component and the power supply for the component may be constructed). The digital twin model may receive information indicative of a processing strategy (e.g., set points, nearby environmental characteristic values, etc.) as input and produce time trace data as output that predicts the sensor readings. In some embodiments, the output of the digital twin model (e.g., simulated sensor data) may be compared to the measured sensor data. In some embodiments, the summary data for the temporary portion may include a difference (e.g., a difference greater than a threshold) between the features of the simulated sensor data and the features of the measured sensor data. The transient data characteristics may include metrics such as overshoot to a target or final value, rise time, settling time, parameters of the detected variations, the difference between the set point and the final value, and the like.

[0164] At block 544, a quality index score is generated based on the summary data. The quality index score can be associated with a particular run (e.g., product made by manufacturing equipment), process operation, sensor, or some combination of these features (e.g., a quality index score can be associated with one sensor and one operation across multiple runs, a quality index score can be associated with a group of sensors such as all temperature sensors, and so on). In some embodiments, quality indexes are generated that indicate the quality of the runs and processing operations. In some embodiments, the generated quality indexes can be used to implement measures to increase the consistency of processing parameters, such as run-to-run, chamber-to-chamber, etc., e.g., tool-to-tool matching (TTTM).

[0165] In some embodiments, a quality index is generated taking into account one or more classification conditions. In some embodiments, data (e.g., summary data) may be classified as inlier data or outlier data. In some embodiments, data may be classified as inlier data, low outlier data (e.g., below the lower control limit), or high outlier data (e.g., above the upper control limit). Classification may be performed by comparing the data value to the upper and lower control limits, for example, to determine if the data value is within a control range, to determine if the data value meets a threshold, etc. Control limits may be assigned by a user, and may be statistical in nature (e.g., three standard deviations away from the mean of the historical data), etc. In some embodiments, inliers do not contribute to the quality index score, and the quality index may be based only on outliers.

[0166] In some embodiments, the quality index is generated taking into account multiple weighting factors. The weighting factors can represent the contribution of the data to performance, e.g., to a characteristic of the finished product. For example, in some processes, data from one sensor may correlate more strongly to performance than data from another sensor. For such processes, data from a first sensor that correlates less strongly to product performance may be weighted less in the quality index score calculation than data from a second sensor that correlates more strongly to product performance. The weighting factors can be associated with sensors, process operations, sensor / operation pairs (e.g., a sensor measuring the voltage supplied to the plasma generating device may be weighted more heavily during plasma operation and less or zero weight during plasma-free operation), features of the data (e.g., one aspect of the summary data may correlate more to product performance than another aspect, a portion of the trace data may correlate more to product performance than another portion, etc.). In some embodiments, the weighting factors can be extracted from a trained machine learning model. The machine learning model can be trained to correlate process data (e.g., sensor data, summary sensor data, etc.) to performance data (e.g., metrology data). The machine learning model can be configured to map one or more relationships between the sensor data and the measurement data. The weighting coefficients can be extracted from the machine learning model (e.g., a stacked partial least squares model), can be determined taking into account factors extracted from the machine learning model, can be calculated based on data including those extracted from the machine learning model (e.g., in addition to process knowledge), and so on.

[0167] In some embodiments, the quality index score can be generated as an aggregation of the calculated metrics. For example, the TTTM index score can be generated by aggregating data associated with many sensors (e.g., via a weighted sum) to calculate a quality index score indicative of the quality of one operation of one processing run (e.g., one product). Other aggregations of indices can also be applicable to the TTTM process, such as aggregating many runs to indicate the quality of the process over time, aggregating across operations and sensors to provide a summary quality value of the processing run, etc. The processing logic can generate many indices to provide (e.g., to a user, further processing logic, etc.) information associated with an index value that can indicate a problem with the manufacturing equipment (e.g., all index values ​​that meet a threshold, a certain percentage or number of index values ​​that are most likely associated with a problem or failure, etc.). An index value indicative of a problem or failure can be used to initiate further analysis, such as a drill-down analysis, to determine root causes, corrective actions to be taken, etc.

[0168] In some embodiments, the quality index score can include information associated with how far the data point is outside the control range, e.g., the severity of the outlier. In some embodiments, the data point can be assigned a severity distance. Calculating the severity distance (e.g., severity distance score) can include calculating how far the data point is outside the control range, e.g., as a fraction of the size of the control range (e.g., an outlier below the lower control range can have a severity distance calculated using the ratio between how far the data point is below the lower control limit and the difference between the upper and lower control limits). The control limits and control ranges can be associated with sensors, process operations, chambers, product designs, etc. The severity distance calculation can include weighting factors. The severity distance can be associated with sensors, process operations, and processing runs (e.g., products).

[0169] In some embodiments, the severity distances can be aggregated (e.g., into a severity score). The aggregated severity distance score can be used in drill-down analysis. As an example, generating a severity score (e.g., aggregated severity distance) can include adding the severity distances for multiple runs (e.g., to generate a severity score associated with a sensor and process operation). In some embodiments, the severity score can include factors related to how many points are outliers, such as the frequency (e.g., percent) of outlier points per sensor and process operation.

[0170] In some embodiments, one or more metrics indicative of quality (e.g., TTTM index, severity score, etc.) may be provided to the processing logic for visualization. In some embodiments, the data points (e.g., severity distance) may be provided to a dimensionality reduction model. In some embodiments, the dimensionality reduction model may be a non-linear model such as a t-distributed probabilistic nearest neighbor embedding model. In some embodiments, the data points (e.g., reduced severity distance) may be provided to a clustering model. In some embodiments, the clustering model may be a non-linear clustering model such as density-based spatial clustering of applications with noise models. In some embodiments, the data points of the clusters may be assigned to subsystems (e.g., based on the top contributions of the data points to the severity score). In some embodiments, each cluster may be assigned a root cause of a subsystem of the manufacturing equipment.

[0171] At block 546, the processing logic provides an alert to a user based on the quality index score, the alert including an indication that the manufacturing chamber performance does not meet the first threshold. In some embodiments, a metric (such as a TTTM index score) may be provided to the user. In some embodiments, a visual representation of the metric (e.g., a bar graph, a histogram, a box plot, etc.) may be provided to the user. In some embodiments, the visual representation may be used by the user to initiate, inform, or continue a drill-down analysis, for example to identify a root cause of a failure or a corrective action to be taken.

[0172] FIG. 5F is a flow diagram of a method 500F of performing inlier analysis, according to some embodiments. In one embodiment, method 500F is performed after method 500E is performed on data in which no outliers were identified. At block 550, processing logic receives first data associated with measurements obtained by a sensor during a first manufacturing procedure of a manufacturing chamber. In some embodiments, the data may include summary data. The data may include inlier data points, where an inlier data point is a point that meets a criterion (e.g., a threshold). At block 552, processing logic receives second data. The second data includes reference data associated with the first data. The reference data may be data associated with a process that produces an acceptable product, a process in which an acceptable condition is achieved, or the like. The reference data may be referred to as golden data. The reference data may be associated with the same process, sensor, chamber, equipment, or the like as the first data.

[0173] At block 554, the first data and the reference data (e.g., golden run data) are provided to the comparison model. In some embodiments, the comparison model is an elastic discrepancy model, e.g., a model that manipulates the space between the data points of the two data sets without changing their order to achieve the closest possible match. In some embodiments, the comparison model can be a dynamic time warping model. In some embodiments, data (e.g., summary data) over multiple processing runs (e.g., days, weeks, or more product production) can be provided to the comparison model. A metric can be provided (e.g., by an elastic discrepancy model) that indicates how different the measured data is from the golden data. For example, the model can indicate the best match between each data point of both data sets and the data points of the other set. Two identical data sets can have the same number of matched pairs if there are points in one data set. Two sets that are not well matched can have a larger number of pairs. The metric can relate to the number of matched pairs between the two sets, e.g., the ratio of data points to matched pairs.

[0174] At block 556, processing logic receives similarity scores associated with the first data and the reference data from the comparison model. In some embodiments, the similarity scores relate to a plurality of match pairs between the two data sets. At block 558, processing logic performs a corrective action taking into account the similarity scores. The corrective action may include sending an alert to a user. The alert may include a visual representation, e.g., a visual representation of the plurality of similarity scores, a visual representation of a closest point match between the two sets of data, etc. The corrective action may include scheduling preventative or corrective maintenance. The corrective action may include updating a process recipe.

[0175] 5G is a flow diagram of a method 500G of performing trace sensor data analysis, according to some embodiments. At block 560, processing logic receives trace sensor data associated with a first manufacturing process in a processing chamber. In some embodiments, the processing logic can provide the trace data in response to a value of one or more metrics satisfying a threshold. For example, a TTTM index can satisfy a threshold. In response to the index satisfying the threshold (e.g., satisfying a criterion), the trace data associated with the TTTM index (e.g., trace data from a sensor having a high severity score) can be further analyzed. In some embodiments, the processing logic performs pre-processing on the trace data. Pre-processing can include smoothing, cleaning (e.g., resizing, removing empty data, etc.), interpolation, etc.

[0176] At block 562, the processing logic processes the trace sensor data using one or more trained machine learning models. The trained machine learning models are trained to generate a representation (e.g., a dimensionality-reduced representation) of the trace sensor data. The one or more trained machine learning models are trained to reconstruct the sensor data based on the representation of the trace sensor data. The one or more trained machine learning models output the reconstructed data. In some embodiments, the trace sensor data may be provided to an autoencoder model. In some embodiments, the trace sensor data may be provided to a trained encoder model to generate a representation of the trace sensor data. The representation may then be provided to a trained decoder model to generate the reconstructed sensor data. The autoencoder model may be an encoder-decoder model. In some embodiments, the machine learning model may be trained by providing data (e.g., preprocessed sensor trace data) to the model. The model may be configured to generate a representation (e.g., a dimensionality-reduced representation) of the input data. The model may be configured to generate the reconstructed data from the representation of the input data. The model may then be trained by minimizing the difference between the input data and the reconstructed data (according to configuration settings, e.g., to avoid making changes to the trace sensor data to generate the representation).

[0177] At block 564, processing logic compares the trace sensor data to the reconstructed sensor data. In some embodiments, the machine learning model has been trained using data that meets a threshold, e.g., data within control limits, data with a specified portion within the control limits, data associated with a process that produced a product that meets a performance threshold, golden run data, etc. The trace sensor data provided to the trained machine learning model may or may not be associated with irregular process runs, e.g., process runs with data outside the control limits.

[0178] At block 566, the processing logic determines one or more differences between the reconstructed sensor data and the trace sensor data. The differences may be determined based on the comparison performed in connection with block 564. In some embodiments, the trace sensor data may differ in some way from the training data, e.g., the trace sensor data may be associated with a processing run that did not produce an acceptable product, the trace sensor data may be outside one or more control limits, and so on. The differences between the training set and the trace sensor data may cause the trace sensor data and the reconstructed data to be significantly different. The reconstruction error may be used to determine whether the input data falls significantly outside the range of the training data. Because the machine learning model was trained using data from a healthy tool (e.g., a tool that has a relatively new part and produces products that meet quality standards), the machine learning model produces an accurate reproduction of the trace data when the trace data is from a healthy tool. However, when the reconstructed data differs from the trace data, this is an indication that the trace data deviates in some way from that of a healthy tool. In some embodiments, the reconstruction error (e.g., the error for each data point of the trace data) may be aggregated, and a metric of the error associated with, for example, a sensor and a processing run (e.g., one product) may be generated.

[0179] At block 568, the processing logic determines whether to recommend a corrective action associated with the process chamber. The determination may be based on one or more differences between the trace sensor data and the reconstructed sensor data. The determination may be based on an aggregated error metric. In some embodiments, the sensor may be associated with a subsystem. The error metric may be further aggregated to the subsystem (e.g., as a root mean square error). In some embodiments, a visual representation may be generated. The visual representation may be organized by run, subsystem, sensor, etc.

[0180] FIG. 5H is a flow diagram of a method 500H for performing an outlier clustering analysis, according to some embodiments. Outlier clustering trigger and processing details may be subject to configuration settings. In some embodiments, outlier clustering may be triggered by multiple outlier points in a previous analysis meeting a threshold. At block 570, sensor contribution values ​​are extracted by processing logic. Calculation of sensor contributions is discussed in more detail in connection with block 532 of FIG. 5D. In some embodiments, each sensor contribution metric (e.g., severity metric, severity distance, severity score, etc.) utilized for clustering may be associated with a single run and sensor. In some embodiments, clustering methods may be used to analyze multiple data points (e.g., from multiple sensors across multiple processing runs). Because a large number of sensors are associated with a tool, this data may occupy a high-dimensional space. In some embodiments, the majority of the data is inlier data, e.g., data within the control limits. In some embodiments, an inlier sensor contribution value may be defined as 0. In some embodiments, only data points having a sensor contribution value that meets a threshold may be utilized for clustering analysis.

[0181] At block 572, data preprocessing for clustering analysis is performed. Data preprocessing can include, for example, cleaning the dataset to include only runs with sensors that exhibit outlier behavior. Other preprocessing can also be performed, including normalization, etc. At block 574, a dimensionality reduction operation is performed by the processing device. In some embodiments, a high-dimensional dataset can be reduced to a more easily visualized space for increased analytical power, such as to accommodate a clustering model. In some embodiments, the dataset can be reduced to a two-dimensional dataset. In some embodiments, the algorithm will likely place data points that are close to each other in the high-dimensional space close to each other in the low-dimensional space. Closeness can be determined by Euclidean distance, cosine distance, or other distance metric. In some embodiments, a non-linear dimensionality reduction model can be used. In some embodiments, a t-distribution type probabilistic nearest neighbor embedding method can be used for dimensionality reduction. Dimensionality reduction can be performed according to configuration settings, such as distance metric, expected number of nearest neighbors, etc.

[0182] At block 576, a clustering operation is performed by the processing device. The clustering operation may be performed on the reconstructed data in the low-dimensional space. In some embodiments, a clustering model that self-selects the number of clusters in the data may be utilized. In some embodiments, a density-based spatial clustering of application by noise method may be utilized for the clustering operation. The clustering may identify a number of sets of data points that are represented near each other in the low-dimensional space, which may indicate a high probability that the data points are represented near each other in the high-dimensional space. Such data points are likely to be related to the same root cause. The clustering operation may be performed according to configuration settings. The configuration settings may include a distance metric, a minimum number of samples in a cluster, a maximum distance between two samples to be considered neighbors, etc.

[0183] At block 578, the processing logic ranks the contributions of the sensor systems to one or more clusters, according to some embodiments. The data points of a cluster may be analyzed by examining the sensor contribution values ​​that contribute to the data points. In some embodiments, the sensors associated with the data points may be ordered by sensor contribution value (e.g., severity distance). In some embodiments, the sensors with the highest contributions (e.g., the three sensors with the highest contribution values) may be utilized to assign first-order subsystems of the manufacturing system to data points, clusters, etc. In some embodiments, the processing logic may visualize the clustering results. Visualization of outlier clustering was discussed in more detail in connection with block 535 of FIG. 5D and FIG. 6C.

[0184] 5I is a flow diagram of a method 500I for performing inlier drift detection according to some embodiments. Inlier drift detection can be triggered and / or performed according to configuration settings. Some operations of the TTTM analysis module utilize outlier data points (e.g., TTTM index, sensor contribution / severity metrics, etc.). In some embodiments, analysis of inlier data points (e.g., data points within control limits) can be used to determine system faults, drift, corrective actions to be taken, etc.

[0185] At block 580, reference data for inlier drift detection is established. In some embodiments, inlier drift detection includes comparing data (e.g., data collected over a period of time) to the reference data. In some embodiments, a golden data set is provided or selected. The golden data set may include data from a processing run associated with a product manufactured within specifications. The golden data set may include data from a processing run performed immediately after successful installation or preventive maintenance of the tool.

[0186] At block 582, a warp path search is performed. In some embodiments, inlier drift detection involves the use of an elastic dissimilarity model. Such a model alters the spacing between data points of the datasets (but does not change the order of the data points) to generate an optimal match between the two sets, measured, for example, by adding the differences between the data values ​​(or an interpolation of the data values) of the two sets. In some embodiments, a dynamic time warping algorithm is applied to the reference dataset and the dataset to be analyzed. In some embodiments, the dataset includes summary data. In some embodiments, the dataset includes summary data collected over a period of time, for example over many processing runs. The warp path defines how to stretch one dataset (e.g., the data to be analyzed) to match the pattern of the other dataset (e.g., golden data).

[0187] At block 584, a warp point count is determined. The warp point count is a metric to indicate how much a dataset must be warped to provide the best match with another dataset. Other metrics can also be used to indicate the degree of warping, such as an aggregate measure of how much one set of data points must be shifted. The warp point count is a measure of how many match pairs exist for comparison points between two datasets. In some embodiments, each point of the target dataset (e.g., the dataset being analyzed) is matched to one "nearest" point of the reference dataset (e.g., the golden dataset). In some embodiments, each point of the reference dataset is matched to one nearest point of the target dataset. If the two datasets are well aligned, each data point of one set can be matched to one other data point of the other set, and the number of matches will be the same as the number of data points. If the two datasets are not well aligned, some data points of one or both sets may be matched to multiple points of the other set, and the number of matches may be greater than the number of data points. The matching point count can be extended to situations where the reference data and the target data have different numbers of data points. In some embodiments, since each data point has at least one matching point, the matching point count can eliminate the number of data points (e.g., the number of guaranteed matches) leaving only the number of additional matches due to warping of one data set, e.g., the warp point count. At block 586, a warp ratio calculation is performed by the processing logic. The warp ratio is a comparison that indicates how much the data set is warped overall (e.g., corrected for the size of the data set). In some embodiments, the warp ratio can be calculated as follows:

[0188]

number

[0189] 6A-6G show example visual representations associated with aspects of TTTM analysis, according to some embodiments. FIG. 6A shows a visual representation 600A of an operation of splitting a trace into a transient portion and a steady-state portion, according to some embodiments. Visual representation 600A includes a time trace 602. Time trace 602 may be associated with one sensor and one processing operation (e.g., the full runtime trace has already been split to generate time trace 602, such as by operation splitter 451 of FIG. 4C). A window 604 is utilized to determine whether a set of data points (e.g., data points within the window) belong to the transient or steady-state portion of the data. A metric value of the points within the window may be used to classify the points within the window as transient or steady-state. In some embodiments, the points in the window are determined to be of the transient portion if the standard deviation of the points within the window is above a threshold.

[0190] In some embodiments, a window 604 is moved through the time trace to classify the data points of the trace as transient or steady state, as indicated by the arrow and dashed window to the right of trace 602. In some embodiments, the time trace 602 can be separated into two or more portions. In some embodiments, the time trace 602 can be separated into a transient head portion 606, a steady state portion 607, and a transient tail portion 608.

[0191] 6B illustrates a visual representation 600B of a digital twin output used in summary data generation operations, according to some embodiments. One or more components of manufacturing equipment can be modeled by a digital twin. The digital twin includes a digital representation of one or more physical assets. The digital twin can be used to model system response (e.g., control tuning to meet set points). Various features of the digital twin response can be compared to sensor data to generate summary data.

[0192] Visual representation 600B includes a time trace 610. In some embodiments, the time trace 610 can be associated with a temporal portion of a processing operation. The digital twin model can include a target set point 611 associated with a characteristic value measured by a sensor in the physical manufacturing system. A feature of the digital twin time trace includes an overshoot 612. The time trace 610 may exceed the set point (or, in some embodiments, fall below the set point) before settling at the set point value. One or more of the values ​​of the overshoot 612 (or a normalized or otherwise processed variant) or the peak time 613 can be utilized to generate summary data. In some embodiments, the rise time 614 can be used to generate summary data. In some embodiments, the rise time 614 can be measured only when the time trace 610 meets the set point 611. In some embodiments, the rise time 614 can be measured only when the time trace 610 reaches a threshold value (e.g., 80% of the difference between the set point and a start value, as shown in visual representation 600B).

[0193] In some embodiments, the duration of a fluctuation, such as fluctuation 615, can be used to generate summary data. Characteristics such as frequency, amplitude, etc. of the fluctuation can be used to generate summary data. Settling time 617 (e.g., the time that time trace 610 remains within error window 616 of set point 611) can be used to generate summary data. In some embodiments, set point error (e.g., the difference between the final value of time trace 610 and set point 611) can be used to generate summary data.

[0194] FIG. 6C illustrates a visual representation of a clustering operation result such as that discussed in connection with FIG. 5H, according to some embodiments. Outlier data points (e.g., data points that exhibit unusual or suboptimal performance) can be used in the clustering operation. The clustering operation can be directed to identifying the root cause (e.g., subsystem) that is responsible for the outlier data (e.g., suboptimal performance of manufacturing equipment). In some embodiments, the high-dimensional sensor data is transformed into a low-dimensional space, for example, by a dimensionality reduction model. In some embodiments, the high-dimensional space is reduced to a two-dimensional space. In some embodiments, a dimensionality reduction algorithm brings points that were close to each other in the high-dimensional space (e.g., by Euclidean distance, cosine distance, etc.) closer to each other in the low-dimensional space. In some embodiments, data points with the same (or related) root cause (e.g., subsystem that made the data point an outlier) can be represented close to each other in the low-dimensional space. In some embodiments, the sensor contribution to the data points can be used to assign subsystems to clusters.

[0195] Visual representation 600C shows outlier points in a low-dimensional space. A clustering algorithm (e.g., density-based spatial clustering application) can be used to separate the data points into groups that are likely to be related to the same root cause. Exemplary visual representation 600C is separated into four clusters 640, 641, 642, and 643. Each cluster's top contributing sensor (e.g., the sensor with the highest severity metric) can be used to label each cluster with a manufacturing subsystem. For example, cluster 640 can be seen to be associated with a heater power subsystem, cluster 642 can be seen to be associated with an electrostatic chuck temperature, cluster 643 can be seen to be associated with a gas flow rate for some part of the manufacturing system, and so on.

[0196] 6D-6E show visual representations of the results of an inlier drift detection elastic dissimilarity model, according to some embodiments. FIG. 6D shows the results of an elastic dissimilarity model 600D applied to a target time trace 650 and a reference time trace 651. FIG. 6D (and FIG. 6E) show original sequential data, e.g., summary data collected over multiple processing runs, and the results of warping (e.g., applying an elastic dissimilarity model) are shown by dashed lines indicating matching point pairs. The result 600D is generated from two traces that are fairly similar. As highlighted by the circles, only some points in either trace are matched to multiple points in the other trace, and in each case the number of matching points is small. The target trace 650 and the reference trace 651 are similar in time, as shown by the dashed lines that are fairly ordered (e.g., mostly parallel).

[0197] FIG. 6E shows the results of an elastic dissimilarity model 600E applied to a target time trace 660 and a reference time trace 661 that are more dissimilar than those shown in FIG. 6D. Many metrics can be used to quantify the degree of warping of the target trace 660, for example to trigger corrective action. The result 600E includes several data points that are aligned with many points in the other trace. One way to indicate how much warping is associated with a pair of time traces includes comparing the number of matches that should exist in a well-matched pair of traces to the number of matches that exist in the analysis. Other methods include characterizing the aggregate displacement of the matches in time (e.g., how far the dashed line in FIG. 6E is stretched from left to right), an indication of the slope of the alignment line, etc.

[0198] 6F-6G show example visual representations 600F and 600G of the results of trace data analysis, according to some embodiments. FIG. 6F includes a heat map 670 and a legend 671. The reconstruction error of the autoencoder model may be utilized to generate the heat map 670. In some embodiments, the reconstruction error (e.g., relating to the difference between the input data and the reconstructed data at each timestamp of the time trace) may be aggregated into a single error number per trace. In some embodiments, the reconstruction error may be further aggregated by sensor subsystem. In some embodiments, a single value may be generated for one processing run and one fabrication subsystem. The heat map 670 may generate multiple subsystems (e.g., subsystems A, B, C, and D, such as the vacuum subsystem, temperature subsystem, chucking subsystem, etc.) and multiple processing runs (e.g., fabricated substrates). The legend 671 may visually indicate subsystem / run pairs with high reconstruction error, e.g., subsystem / run pairs that are likely to have been performed suboptimally. The example visual representation 600F shows a particularly high reconstruction error associated with run number 5, subsystem B. In some embodiments, visual representation 600F may be generated as a corrective action and may be considered, for example, an alert provided to a user. In some embodiments, a visual representation such as visual representation 600F may be used in taking corrective action, for example visual representation 600F may indicate that maintenance to subsystem B is appropriate.

[0199] FIG. 6G illustrates an example visual representation 600G of output from an autoencoder model associated with trace data analysis, according to some embodiments. Visual representation 600G includes input data pattern 680 and reconstructed data pattern 681. In some embodiments, visual representation 600G may be generated (or displayed to a user) in response to detecting a high reconstruction error, e.g., in response to detecting a high reconstruction error in a heatmap visual representation, such as FIG. 6F. In some embodiments, a user may select to see more detail in a portion of the visual representation associated with the time trace analysis (e.g., by clicking on a portion of heatmap 670 associated with Subsystem B and Processing Run 5) to display a more detailed view (e.g., visual representation 600G). In some embodiments, suboptimal performance may be associated with a high reconstruction error, a large difference between input trace data pattern 680 and reconstructed data pattern 681, e.g., as shown in visual representation 600G.

[0200] FIG. 7 is a diagram of the operation of a model 700 (e.g., a machine learning model), such as an autoencoder model, that can reduce the dimensionality of input data, in accordance with some embodiments.

[0201] The input data 710 of the model 700 is data associated with the fabrication of a substrate. In some embodiments, the input data 710 includes one or more of metrology data, manufacturing parameters, sensor data, or a combination thereof. The input data 710 can be pre-processed data. In some embodiments, the input data 710 is metrology data associated with the substrate. The metrology data can be of any (or many) types including thickness, in-plane displacement, chemical properties, electronic properties, optical properties, etc. In some embodiments, the input data 710 includes trace sensor data and / or pre-processed versions of trace sensor data associated with the fabrication of a product.

[0202] The model 700 includes a first portion 720 (e.g., an encoder) and a second portion 740 (e.g., a decoder). In some embodiments, the model is one or more of an autoencoder, a neural network model, a convolutional neural network model, a deep belief network, a forward neural network, a multi-layer neural network, and the like. The first portion 720 performs dimensionality reduction of the input data 710 (e.g., measurement data) into a compressed form (e.g., compressed data 730). A number of hidden layers (two of which are shown in FIG. 7, but any number of hidden layers can be used) can separate the input layer of the machine learning model from the compressed data. During training of the machine learning model 700, the first portion 720 can generate one or more functions to fit the input data 710 into a lower dimensional representation without guidance from a user. The reduction (e.g., compression, encoding) can be performed over several stages (i.e., first converting the input data 710 into partially compressed data and then further converting it into compressed data 730), or the reduction (e.g., compression, encoding) can be performed in a single stage.

[0203] The second portion 740 takes the compressed data 730 as input and produces output data 750. During training, the model 700 is trained to minimize the difference between the input data 710 and the output data 750, with the output 750 being a reconstruction of the input data 710 from the compressed data 730. The minimization function used to train the model 700 may also enforce a penalty on the dimensionality of the compressed data 730 to avoid reverting to a function with poor compression (e.g., a discriminant function that perfectly reproduces the input data 710 but does not compress the data to a reduced dimensionality). The model 700 may be trained such that the output data 750 of the model 700 closely matches the input data 710.

[0204] The functions utilized by the first portion 720 and the second portion 740 may be non-linear in nature. In some applications, all processes of the model 700 (i.e., both reduction and reconstruction, both encoding and decoding, etc.) may be utilized. In other applications, only some capabilities may be utilized. For example, during training, the model 700 may pass input data 710 through the first portion 720 to form compressed data 730, then pass it through the second portion 740 to determine output data 750, and then compare the output data 750 to the input data 710 to determine that the output data 750 is substantially similar to the input data 710.

[0205] In some embodiments, the model 700 can be utilized for anomaly trace analysis. The model 700 can be trained on past data. The past training data can be limited to successful runs, e.g., runs where the product characteristics produced were within a threshold range, where the chamber conditions achieved were within a threshold range, etc. The model 700 can then be configured to generate condensed data for future successful manufacturing runs. In some embodiments, the model 700 can be trained and utilized on data associated with one (or a limited number) set of equipment, recipes, product designs, etc.

[0206] The model 700 may produce output data 750 that has some reconstruction error (e.g., difference from the input data 710). In some embodiments, input data 710 that is in some way similar to the training inputs used to train the model 700 may produce output data 750 that has a relatively low reconstruction error. A reconstruction error metric (e.g., sum of errors between data points, squared error, normalized sum, root mean square error, etc.) may be generated to represent a metric of the total reconstruction error of the dataset input to the model 700. In some embodiments, the magnitude of the error metric may indicate the similarity between the input data 710 and the data used to train the model 700. In some embodiments, a high reconstruction error (e.g., there is a large difference between the output data 750 (the reconstructed data generated from the compressed data 730)) may indicate that the input data 710 is significantly different from the data used to train the model 700. In some embodiments, the model 700 is trained strictly on “good” data (e.g., data associated with manufacturing runs that meet a performance threshold) and the reconstruction error metric may indicate that the input data 710 is suboptimal. The reconstruction error may be calculated per run (e.g., per product produced), per sensor, per tool, per subsystem, etc., and may be visualized in a convenient format to facilitate taking corrective actions.

[0207] FIG. 8 is a block diagram illustrating a computer system 800 according to certain embodiments. In some embodiments, the computer system 800 can be connected to other computer systems (e.g., via a network such as a local area network (LAN), an intranet, an extranet, or the Internet). The computer system 800 can operate in the capacity of a server or a client computer in a client-server environment, or can operate as a peer computer in a peer-to-peer or distributed network environment. The computer system 800 can be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web appliance, a server, a network router, switch, or bridge, or any device capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by that device. Additionally, the term "computer" is intended to include any group of computers that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methods described herein.

[0208] In a further aspect, the computer system 800 may include a processing device 802, a volatile memory 804 (e.g., random access memory (RAM)), a non-volatile memory 806 (e.g., read only memory (ROM) or electrically erasable programmable ROM (EEPROM)), and a data storage device 818, which may communicate with each other via a bus 808.

[0209] The processing device 802 may be provided by one or more processors, such as a general-purpose processor (e.g., a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a microprocessor implementing other types of instruction sets, or a combination of types of instruction sets, etc.), or a specialized processor (e.g., an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), or a network processor, etc.).

[0210] The computer system 800 may further include a network interface device 822 (e.g., coupled to a network 874). The computer system 800 may also include a video display unit 810 (e.g., an LCD), an alphanumeric input device 812 (e.g., a keyboard), a cursor control device 814 (e.g., a mouse), and a signal generating device 820.

[0211] In some embodiments, the data storage device 818 may include a non-transitory computer-readable storage medium 824 (e.g., a non-transitory machine-readable medium) that may store instructions 826 encoding any one or more of the methods or functions described herein, the instructions 826 including instructions for encoding the components of FIG. 1 (e.g., the prediction component 114, the corrective action component 122, the model 190, etc.) and for performing the methods described herein.

[0212] The instructions 826 may also reside, completely or partially, within the volatile memory 804 and / or the processing device 802 during execution thereof by the computer system 800, and thus the volatile memory 804 and the processing device 802 may also constitute machine-readable storage media.

[0213] While the illustrative example shows computer-readable storage medium 824 as a single medium, the term "computer-readable storage medium" is intended to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of executable instructions. The term "computer-readable storage medium" is also intended to include any tangible medium capable of storing or encoding a set of instructions for execution by a computer that causes a computer to perform any one or more of the methodologies described herein. The term "computer-readable storage medium" is intended to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0214] The methods, components, and features described herein may be implemented by individual hardware components or may be integrated into the functionality of other hardware components, such as ASICS, FPGAs, DSPs, or similar devices. In addition, the methods, components, and features may be implemented by firmware modules or functional circuits within a hardware device. Furthermore, the methods, components, and features may be implemented in any combination of hardware devices and computer program components, or may be implemented within a computer program.

[0215] Unless specifically stated otherwise, terms such as "receive," "perform," "provide," "obtain," "cause," "access," "determine," "add," "use," "train," "reduce," "generate," "correct," and the like refer to actions and processes performed or implemented by a computer system that manipulate and transform data represented as physical (electronic) quantities in computer system registers and memories into other data similarly represented as physical quantities in the computer system memory or registers or other such information storage, transmission, or display devices. Also, in this specification, terms such as "first," "second," "third," "fourth," and the like are meant to be labels for distinguishing different elements and may not have a sequential meaning according to numerical designation.

[0216] The examples described herein also refer to an apparatus for carrying out the methods described herein. The apparatus may be specifically constructed to carry out the methods described herein, or may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0217] The methods and examples described herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform the methods and / or each of the individual functions, routines, subroutines, or operations described herein. Example structures for a variety of these systems have been set forth in the description above.

[0218] The above description is intended to be illustrative and not limiting. Although the present disclosure has been described by reference to particular examples and embodiments, it will be understood that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. receiving trace sensor data associated with a first manufacturing process in a manufacturing chamber; processing the trace sensor data with a processing device to generate summary data associated with the trace sensor data; generating a quality index score based on the summary data; providing an alert to a user based on the quality index score, the alert including an indication that the manufacturing chamber performance does not meet a first threshold. method.

2. generating the summary data, identifying a steady state portion of the trace sensor data; identifying a temporal portion of the trace sensor data; generating first summary data for the steady state portion by calculating statistical metric values ​​from the steady state portion; and generating second summary data for the temporal portion by considering one or more differences between the temporal portion and predicted data associated with the temporal portion.

3. 3. The method of claim 2, wherein the predicted data associated with the temporary portion includes an output of a digital twin model of one or more components of the manufacturing chamber, the digital twin model configured to output predicted performance data for the one or more components.

4. identifying the steady state portion of the trace sensor data includes determining that a standard deviation of a first set of data values ​​of the trace sensor data is below a second threshold; identifying the temporary portion of the trace sensor data includes determining that a standard deviation of a second set of data values ​​of the trace sensor data is greater than a third threshold. The method of claim 2.

5. The first summary data of the steady state portion is an average of the values ​​of the steady state portion; a minimum of said values ​​of said steady state portion; a maximum of said values ​​of said steady state portion; quartile values ​​of said values ​​of said steady state portion; the standard deviation of said values ​​of said steady state portion; or The method of claim 2 , further comprising one or more of: a kurtosis of the value of the steady state portion.

6. the second summary data of the temporary portion being Indication of overshoot of target or final value, Rise time indication, Stabilization time indication, an indication of the deviation between the set point and said final value; or The method of claim 2 , further comprising one or more indications of a variation in the value of the temporary portion.

7. The quality index score is a plurality of weighting factors that determine the importance of a contribution of data generated by a first sensor during a first process operation to the quality index score; and The method of claim 1 , wherein the summary data is generated taking into account a plurality of control limits that are thresholds used to classify data points of the summary data.

8. providing historical metrology data of products produced using the manufacturing chamber and historical sensor data associated with the products as training data for training a machine learning model, wherein the trained machine learning model is configured to map associations between the sensor data and the metrology data. The method according to claim 7.

9. The method of claim 8 , wherein the weighting coefficients are derived from the associations of the trained machine learning models.

10. determining a severity distance score associated with a first sensor, a first manufactured product, and a first process operation in response to generating the quality index score, the severity distance score indicating a relationship between summary data associated with the first sensor and the first process operation and a control range associated with the first sensor and the first process operation; determining a severity score associated with the first sensor, the plurality of manufactured products, and the first process operation, the severity score being determined taking into account the severity distance; The method of claim 1 further comprising:

11. providing the significance distance and a plurality of significance distances to a dimensionality reduction model; receiving a reduced dimensionality representation of the significance distance from the dimensionality reduction model; providing the reduced dimensionality representation to a clustering model; receiving an indication of a plurality of groups from the clustering model; determining a set of sensors that contribute to a group, where determining the sensors of the set includes comparing severity distance values ​​of sensors associated with data points of the group; determining a subsystem of the manufacturing chamber associated with the group based on the set of sensors; The method of claim 10 further comprising:

12. 1. A system comprising a memory and a processing device coupled to the memory, the processing device comprising: receiving trace sensor data associated with a first manufacturing process in a manufacturing chamber; processing the trace sensor data to generate summary data associated with the trace sensor data; receiving one or more criteria to be used to classify the summary data; classifying one or more of the summary data as outlier summary data; generating a quality index score based on the outlier summary data; The system is configured to provide an alert to a user based on the quality index score, the alert including an indication that the manufacturing chamber performance does not meet a first threshold.

13. Generating summary data identifying a steady state portion of the trace sensor data; identifying a temporal portion of the trace sensor data; generating first summary data for the steady state portion by calculating statistical metric values ​​from the steady state portion; and generating second summary data for the temporary portion by accounting for one or more differences between the temporary portion and data generated by a digital twin model of at least one component of the manufacturing chamber.

14. identifying the steady state portion of the trace sensor data includes determining that a standard deviation of a first set of data values ​​of the trace sensor data is below a second threshold; identifying the temporary portion of the trace sensor data includes determining that a standard deviation of a second set of data values ​​of the trace sensor data is greater than a third threshold. The system of claim 13.

15. The second summary data is Indication of overshoot of target or final value, Rise time indication, Stabilization time indication, an indication of the deviation between the set point and said final value; or The system of claim 13 , further comprising one or more indications of a variation in the value of the temporary portion.

16. the processing device comprising: determining a severity distance score associated with a first sensor, a first manufactured product, and a first process operation, the severity distance score indicating a relationship between summary data associated with the first sensor and the first process operation and a control range associated with the first sensor and the first process operation; 13. The system of claim 12, further configured to determine a severity score associated with the first sensor, the plurality of manufactured products, and the first process operation, the severity score being determined taking into account the severity distance.

17. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a processing device to perform an operation, the instructions comprising: receiving trace sensor data associated with a first manufacturing process in a manufacturing chamber; processing the trace sensor data to generate summary data associated with the trace sensor data; generating a quality index score based on the summary data; and providing an alert to a user based on the quality index score, the alert including an indication that the manufacturing chamber performance does not meet a first threshold.

18. Generating summary data identifying a steady state portion of the trace sensor data; identifying a temporal portion of the trace sensor data; generating first summary data for the steady state portion by calculating statistical metric values ​​from the steady state portion; and generating second summary data of the temporary portion by accounting for one or more differences between the temporary portion and data generated by a digital twin model of at least one component of the manufacturing chamber.

19. The operation, 20. The non-transitory machine-readable storage medium of claim 17, further comprising providing historical metrology data of a product produced using the manufacturing chamber and historical sensor data associated with production of the product as training data for training a machine learning model, wherein the trained machine learning model is configured to map one or more relationships between sensor data and metrology data.

20. the quality index score is generated by considering a plurality of weighting factors, the weighting factors determining the importance of a contribution of data generated by a first sensor during a first process operation to the quality index score; the weighting coefficients are derived from the one or more relationships of the trained machine learning model; 20. The non-transitory machine-readable storage medium of claim 19.

Images