Monitoring edge-deployed machine learning models

Abstract

The method includes determining reference distribution data associated with features used to train a machine learning model to generate a trained machine learning model. The method further includes providing the reference distribution data to an edge device associated with the substrate processing equipment. The method further includes receiving current distribution data associated with the features from the edge device in response to use of the trained machine learning model at the edge device. The method further includes performing corrective actions associated with the trained machine learning model based on the current distribution data.

Description

[Technical Field]

[0001] The present invention relates to machine learning models, and more particularly to monitoring edge-deployed machine learning models.

[0002] Products can be produced by performing one or more manufacturing processes using manufacturing equipment. For example, substrate processing equipment can be used to manufacture substrates through substrate processing operations.

[0003] The following is a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This summary is not an exhaustive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure, nor is it intended to delineate the scope or claims of particular embodiments of the disclosure. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] In one aspect of the present disclosure, a method includes determining reference distribution data associated with features used to train a machine learning model and generating a trained machine learning model. The method further includes providing the reference distribution data to an edge device associated with the substrate processing equipment. The method further includes receiving current distribution data associated with the features from the edge device in response to using the trained machine learning model at the edge device. The method further includes performing a corrective action associated with the trained machine learning model based on the current distribution data.

[0005] In another aspect of the present disclosure, a method includes receiving, from a server device, reference distribution data associated with features used to train a machine learning model and generating a trained machine learning model. The method further includes using the trained machine learning model based on input data associated with the substrate processing apparatus. The method further includes determining current distribution data associated with the features in response to use of the trained machine learning model. The method further includes providing the current distribution data to the server device and causing a corrective action associated with the trained machine learning model to be performed.

[0006] In another aspect of the present disclosure, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform operations. The operations include determining reference distribution data associated with features used to train a machine learning model and generating a trained machine learning model. The operations further include providing the reference distribution data to an edge device associated with the substrate processing device. The operations further include receiving current distribution data associated with the features from the edge device in response to using the trained machine learning model at the edge device. The operations further include performing corrective actions associated with the trained machine learning model based on the current distribution data. [Brief explanation of the drawings]

[0007] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Figure 1] FIG. 1 is a block diagram illustrating an exemplary system architecture, according to some embodiments. [Figure 2] FIG. 1 is a sequence diagram associated with monitoring an edge-deployed machine learning model, according to some embodiments. [Figure 3] FIG. 1 is a block diagram illustrating monitoring of edge-deployed machine learning models, according to some embodiments. [Figure 4A] ~ [Figure 4B] FIG. 1 is a flow diagram of a method related to monitoring edge-deployed machine learning models, according to some embodiments. [Figure 5] FIG. 1 is a block diagram illustrating a computer system according to some embodiments. Detailed Description

[0008] This specification describes techniques related to monitoring of edge-deployed machine learning models (e.g., private monitoring of edge machine learning models, private and high-performance monitoring of edge-deployed machine learning models, etc.).

[0009] Manufacturing equipment is used to produce products. For example, substrate processing equipment is used to manufacture substrates (wafers, semiconductors, etc.). The substrate processing equipment is controlled (e.g., through selection of manufacturing parameters, etc.) to manufacture substrates that meet a threshold. Traditionally, trial and error has been used to control the substrate processing equipment to manufacture substrates that meet the threshold. This results in wasted material, wasted substrates, wasted energy, wasted time, reduced yield, etc.

[0010] When substrate processing equipment is controlled (e.g., using certain manufacturing parameters) to produce substrates that meet a threshold, changes in the substrate processing equipment (e.g., drift, contamination, wear, part replacement, etc.) may occur that result in the produced substrates not meeting the threshold, resulting in wasted material, wasted substrates, wasted energy, wasted time, reduced yield, etc.

[0011] Machine learning models are used in various process control and predictive functions associated with manufacturing equipment. The machine learning models are trained using data associated with the manufacturing equipment. Substrate processing data is often highly sensitive and is not shared with other devices. Traditionally, machine learning models are trained on a server and then provided for use by the substrate processing equipment via an edge device. In some conventional systems, the substrate processing data used by the trained machine learning models on the edge device is not returned to the server due to privacy concerns. In these conventional systems, the trained machine learning models on the server device and the edge device are not updated based on the substrate processing data and may not be updated to accommodate drift, etc. This can result in substrates that do not meet thresholds, substrate failures, wasted substrates, and reduced yields.

[0012] In some conventional systems, all or a portion of the input data (e.g., sensor data) and output data (e.g., prediction data) of an edge-deployed, trained machine learning model are transmitted to a server (e.g., a server device, a central server device) to monitor the trained machine learning model (e.g., to determine whether there is model drift). Transmitting data from an edge-deployed machine learning model to a server reduces data privacy (e.g., the input and output data may be sensitive data) and raises privacy concerns related to data sharing. Additionally, transmitting data from an edge-deployed machine learning model to a server increases the amount of transmitted data, which may cause bandwidth shortages at the edge device, the server, and / or the entire system.

[0013] The devices, systems, and methods disclosed herein provide monitoring of edge-deployed machine learning models (e.g., private monitoring of edge machine learning models, private and high-performance monitoring of edge-deployed machine learning models).

[0014] The system may include a server device (e.g., a central server device) and one or more edge devices (e.g., edge server devices) each associated with a set of substrate processing equipment. In some embodiments, the server device (e.g., the central server device) determines reference distribution data associated with features used to train a machine learning model and generates a trained machine learning model. In some embodiments, the reference distribution data includes bin ranges of training data associated with the features, and the training data is used to train the machine learning model. In some embodiments, the data may have corresponding data values. In some embodiments, the data is categorized into sets based on the data values. For example, the data may include data values ​​in ranges of 0 to 100. For example, the data may be categorized into bin ranges of 0 to 25, 25 to 50, 50 to 75, and 75 to 100. Each of 0 to 25, 25 to 50, 50 to 75, and 75 to 100 may be a corresponding bin range for sorting the data. In some embodiments, the server device further identifies training data associated with the features. Furthermore, in some embodiments, the server device categorized the training data into bins. Here, determining the reference distribution data includes determining bin ranges for the binned training data.

[0015] The server device provides the reference distribution data to an edge device associated with the substrate processing equipment. In some embodiments, the server device may have greater computing power than the edge device. In some embodiments, some data processed at the edge server may not be provided to the server device (e.g., due to privacy concerns, etc.).

[0016] The server device receives current distribution data associated with the feature from the edge device in response to use of the trained machine learning model at the edge device. In some embodiments, the current distribution data is based on categorizing at least one of the input data or the predicted data into bins (e.g., corresponding to bin ranges) via the edge device. Here, the current distribution data includes count data per bin histogram associated with at least one of the input data or the predicted data. In some embodiments, the data is sorted into sets based on data values. In some embodiments, each set can be placed in a bin corresponding to the set's bin range. For example, the data can include data values ​​ranging from 0 to 100. For example, the data can be sorted into bin ranges of 0 to 25, 25 to 50, 50 to 75, and 75 to 100. Each of 0 to 25, 25 to 50, 50 to 75, and 75 to 100 is a bin range for sorting the data and corresponds to a bin. In some embodiments, the histogram data can be a diagram showing the frequency of a variable in each class interval (e.g., including a rectangle whose area is proportional to the frequency of the variable and whose width is equal to the class interval). In some embodiments, the count per bin histogram data may be graphical data used to represent the frequency distribution of data points for a variable. In some embodiments, the count per bin histogram data may classify data into various bins (e.g., range groups) and count the number of data points that fall into each bin. In some embodiments, the server device further performs a distribution comparison based on the reference distribution data and the current distribution data. In some embodiments, the reference distribution data and the current distribution data are histogram data.

[0017] The server device causes a corrective action associated with the trained machine learning model to be executed based on the current distribution data. In some embodiments, the corrective action includes at least one of providing an alert or retraining the trained machine learning model. In some embodiments, the server device further determines a difference between the current distribution data and the reference distribution data. In some embodiments, the server device further determines that a difference between the current distribution data and the reference distribution data meets a threshold, and execution of the corrective action is responsive to the difference meeting the threshold.

[0018] Each edge device (e.g., an edge server device) can interact with a server device (e.g., a central server device) and a corresponding set of substrate processing equipment. In some embodiments, the edge device receives, from the server device (e.g., the central server device), reference distribution data associated with features used to train the machine learning model and generates a trained machine learning model. In some embodiments, the reference distribution data includes bin ranges of training data associated with the features, and the training data was used to train the machine learning model.

[0019] The edge device uses the trained machine learning model based on input data (e.g., sensor data) associated with the substrate processing equipment. In some embodiments, the input data may be data processed by the trained machine learning model. In some embodiments, the input data may be data provided as input to the trained machine learning model. For example, the input data may be the temperature of a chamber during a processing operation. In some embodiments, the temperature input data may be used to generate a target output (e.g., predicted data) using the trained machine learning model.

[0020] The edge device determines current distribution data associated with the feature in response to using the trained machine learning model. In some embodiments, the current distribution data includes bin ranges for at least one of input data or predicted data associated with the feature, and the input data is used to generate the predicted data via the machine learning model. In some embodiments, the edge device further identifies the input data and predicted data associated with the feature. In some embodiments, the edge device further bins at least one of the input data or predicted data. Determining the reference distribution data includes determining bin ranges for the binned input data and predicted data.

[0021] The edge device provides the current distribution data to the server device, which causes the server device to perform a corrective action associated with the trained machine learning model, in some embodiments, the performing the corrective action includes at least one of providing an alert or retraining the trained machine learning model.

[0022] Aspects of the present disclosure provide technical advantages. The present disclosure avoids privacy concerns associated with data sharing in conventional systems. Additionally, the present disclosure avoids bandwidth shortages resulting from sending large amounts of data to a server device. Additionally, the present disclosure avoids the creation of substrates that do not meet thresholds, misoperated substrates, wasted substrates, wasted material, wasted energy, wasted time, reduced yield, etc.

[0023] Some embodiments of the present disclosure describe edge devices associated with manufacturing systems (e.g., substrate processing equipment, manufacturing equipment, etc.), but the present disclosure can also be used with edge devices associated with other systems (e.g., Internet of Things (IoT), telecommunications, robotics, digital health, etc.).

[0024] FIG. 1 illustrates an exemplary computer system architecture 100 according to aspects of the present disclosure. In some embodiments, the computer system architecture 100 can be included as part of a manufacturing system for processing substrates. The computer system architecture 100 includes a server device 120 (e.g., a central server device), manufacturing equipment 124, sensors 126, edge nodes 170A-Z (hereinafter, “edge nodes 170”), trained machine learning models 190A-Z (hereinafter, “trained machine learning models 190”), metrology equipment 128, and a data store 140. Each edge node 170 can include an edge device 127 (e.g., including an edge component 130 and a trained machine learning model 190), manufacturing equipment 124, and sensors 126. The sensors 126 can provide sensor data associated with substrates processed by the manufacturing equipment 124.

[0025] The manufacturing tool 124 can produce products such as substrates (e.g., electronic devices, wafers, semiconductors) according to a recipe (e.g., continuous operation over a period of time). The manufacturing tool 124 can include a processing chamber. The manufacturing tool 124 can perform substrate processing operations on substrates (e.g., wafers) in the processing chamber. Examples of substrate processing operations include deposition processes that deposit one or more layers of film on the surface of the substrate, etching processes that form patterns on the surface of the substrate, etc. The manufacturing tool 124 can perform each process according to a process recipe. The process recipe defines a specific set of operations to be performed on the substrate during the process and can include one or more settings associated with each operation. For example, a deposition process recipe can include temperature settings for the processing chamber, pressure settings for the processing chamber, flow rates of precursors of materials included in the film to be deposited on the substrate surface, etc.

[0026] In some embodiments, the manufacturing tool 124 includes a sensor 126 configured to generate data associated with a substrate processed in the manufacturing system 100. For example, a processing chamber may include one or more sensors configured to generate spectral or non-spectral data associated with a substrate before, during, and / or after a process (e.g., a deposition process) is performed on the substrate. In some embodiments, the spectral data generated by the sensor 126 may indicate the concentration of one or more materials deposited on the surface of the substrate. The sensor 126 configured to generate spectral data associated with the substrate may include a reflectometry sensor, an ellipsometry sensor, a thermal spectral sensor, a capacitance sensor, etc. The sensor 126 configured to generate non-spectral data associated with the substrate may include a temperature sensor, a pressure sensor, a flow sensor, a voltage sensor, etc.

[0027] In some embodiments, the sensors 126 provide sensor data (e.g., sensor values, features, trace data, etc.) associated with the manufacturing tools 124 (e.g., associated with the manufacturing of a corresponding product, such as a wafer, by the manufacturing tools 124). Sensor data received over a period of time (e.g., corresponding to at least a portion of a recipe or run) may be referred to as trace data received over time from different sensors 126 (e.g., historical trace data, current trace data, etc.). The sensor data may include one or more values ​​such as temperature (e.g., heater temperature), spacing (SP), pressure, high frequency radio frequency (HFRF), electrostatic chuck (ESC) voltage, current, material flow rate, power, voltage, etc. The sensor data may be associated with or indicative of hardware parameters (e.g., size, type, etc.) such as settings or components of the manufacturing tools 124, or manufacturing parameters such as process parameters of the manufacturing tools 124. The sensor data may be provided while the manufacturing tools 124 are performing a manufacturing process (e.g., equipment measurements during product processing). The sensor data may vary from substrate to substrate.

[0028] In some embodiments, the edge device 127 is an edge server. The edge device 127 can receive input data and generate prediction data 160 (e.g., using a machine learning model, an inference engine, a heuristic model, an algorithm, a physics-based engine, etc.). In some embodiments, the edge device 127 can include a trained machine learning model 190.

[0029] In some embodiments, the server device can train the machine learning model to generate a trained machine learning model (e.g., trained machine learning model 190). The trained machine learning model 190 can be deployed at the edge node 170 (e.g., an edge-based model). Rather than running on a remote computing device (e.g., the server device 120), the trained machine learning model 190 can be run on the edge device 127 in conjunction with the manufacturing tool 124 (e.g., a tool and / or substrate processing system, such as a platform, a transfer chamber, a mainframe, a factory interface, and / or a tool cluster). In some embodiments, the trained machine learning model 190 can be an edge-based model running on the edge node 170 rather than running on a remote computing device (e.g., on the server device 120). Training of the machine learning model can be performed remotely (e.g., on the server device 120), and then the trained machine learning model 190 can be transferred to or used on the edge device 127. Retraining or training updates of machine learning models (e.g., trained machine learning model 190) can be performed periodically or continuously on edge device 127, or remotely on server device 120. By shifting the execution and / or training (including retraining) of machine learning models to the edge device system, the delay between the generation of sensor data and decision-making based on the sensor data can be significantly reduced. Furthermore, shifting decision-making to the edge device reduces the amount of data transmitted over the network, improving efficiency, mitigating privacy concerns, and increasing the speed of decision-making.

[0030] The metrology tool 128 can provide metrology data associated with substrates processed by the fabrication tool 124. The metrology data can include film property data (e.g., spatial film properties of the wafer), dimensions (e.g., thickness, height, etc.), dielectric constant, dopant concentration, density, defects, etc. In some embodiments, the metrology data can further include one or more surface profile property data (e.g., etch rate, etch rate uniformity, critical dimensions of one or more features included in the surface of the substrate, uniformity of critical dimensions across the surface of the substrate, edge placement error, etc.). The metrology data can be for finished or semi-finished products. The metrology data can vary from substrate to substrate. The metrology data can be generated using, for example, reflectometry, ellipsometry, transmission electron microscopy (TEM), etc.

[0031] The metrology tool 128 may be included as part of the fabrication tool 124. For example, the metrology tool 128 may be included within or coupled to a processing chamber and configured to generate metrology data for a substrate before, during, and / or after a process (e.g., a deposition process, an etch process, etc.) while the substrate resides within the processing chamber. In some embodiments, the metrology tool 128 may be referred to as an in-situ metrology tool. In some embodiments, the metrology tool 128 may be coupled to other stations on the fabrication tool 124. For example, the metrology tool 128 may be coupled to a transfer chamber, a load lock, or a factory interface. In some embodiments, the metrology tool 128 is separate from the fabrication tool 124 (e.g., after the substrate is processed, the substrate is transferred from the fabrication tool 124 to the metrology tool 128 and metrology data is generated).

[0032] A metrology instrument 128 may be included as part of the edge node 170. For example, the metrology instrument 128 may be included within or coupled to a processing chamber and configured to generate metrology data for a substrate before, during, and / or after a process (e.g., a deposition process, an etch process, etc.) while the substrate resides within the processing chamber.

[0033] Server device 120 and edge device 127 may each include one or more computing devices such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a graphics processing unit (GPU), an accelerator application-specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc. Operations of server device 120, edge device 127, data store 140, etc. may be performed by a cloud computing service, a cloud data storage service, etc.

[0034] The server device 120 may include a server component 115. In some embodiments, the server component 115 may receive current distribution data 164 (e.g., bin counts, histogram data, etc.) for performing a comparison with reference distribution data 162. The server component 115 may determine a difference (e.g., a difference metric, a difference metric value) between the current distribution data 164 and the reference distribution data 162. For example, the server component 115 may compare the difference metric value to a threshold value and determine whether the difference metric value meets the threshold. Additionally, the server component 115 may receive additional data, such as the current distribution data 164 (e.g., received from the edge device 127, retrieved from the data store 140, etc.), measurement data 129, etc., to generate a difference metric (e.g., by comparing distributions).

[0035] In some embodiments, server component 115 can receive training data (e.g., target inputs, target outputs, etc.) for performing bin evaluation (e.g., determining bin ranges based on the training data). Server component 115 can determine bin ranges for the training data. In some embodiments, server component 115 can be divided into multiple components, devices, etc. to perform the functions described as being performed by server device 120. In some embodiments, all of the functions described herein can be performed by a single component / device (e.g., server component 115).

[0036] In some embodiments, the server component 115 can train and / or retrain the trained machine learning model 190.

[0037] An edge node 170 (e.g., edge device 127) may be associated with one or more trained machine learning models (e.g., trained machine learning model 190). In some embodiments, edge node 170 may include trained machine learning model 190. A machine learning model associated with edge node 170 (e.g., edge device 127) may perform many tasks, including process control, classification, performance prediction (e.g., associated with substrate processing equipment), process updates, etc. A trained machine learning model (e.g., trained machine learning model 190) may be trained using data (e.g., training data) associated with manufacturing equipment 124 or products processed by manufacturing equipment 124, such as sensor data (e.g., data collected by sensors 126), manufacturing parameters (e.g., data associated with process control of manufacturing equipment 124), metrology data (e.g., data generated by metrology equipment 128), etc. In some embodiments, edge device 127 may be divided into multiple components, devices, etc. to perform the functions described as being performed by edge device 127. In some embodiments, all of the functionality described herein may be performed by a single component / device (eg, edge device 127).

[0038] One type of machine learning model that can be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network. An artificial neural network typically includes a feature representation component that includes a classifier or recurrent layer that maps features to a desired output space. For example, a convolutional neural network (CNN) includes multiple convolutional filter layers. Pooling is performed in lower layers, allowing for the handling of nonlinearities. A multilayer perceptron is typically added at higher layers to map the top-layer features extracted by the convolutional layers to a decision (e.g., a classification output).

[0039] Recurrent neural networks (RNNs) are another type of machine learning model. Recurrent neural network models are designed to interpret a series of inputs that are intrinsically related to each other, such as time trace data or sequential data. The outputs of the perceptrons in the RNN are fed back as inputs to the perceptrons to generate the next output.

[0040] In some embodiments, the trained machine learning model 190 may be at least one of a linear regression model, a deep learning model, a logistic regression model, a decision tree model, a support vector machine (SVM) algorithm model, a naive Bayes algorithm model, a k-nearest neighbor (KNN) algorithm model, a K-means model, a random forest algorithm model, a dimensionality reduction algorithm model, a gradient boosting algorithm model, an AdaBoosting algorithm model, and the like.

[0041] The server component 115 can compare the distribution data and generate a difference index value based on the comparison. The difference index value can include or indicate a level of change in the current data compared to the reference data. In one example, the difference index value is a real number between 0 and 1, where 0 indicates no difference between the current data and the reference data and 1 indicates the current data is completely different from the reference data. In response to the difference index value indicating a level of difference that exceeds a threshold level (e.g., threshold), the server component 115 can perform corrective action associated with the trained machine learning model 190 (e.g., providing an alert, retraining the trained machine learning model based on sensor data, measurement data, etc.). In some embodiments, the retraining can include generating one or more data sets using historical data and / or synthetic data.

[0042] Data store 140 may be memory (e.g., random access memory), a drive (e.g., a hard drive, a flash drive), a database system, or any other type of component or device capable of storing data. Data store 140 may include multiple storage components (e.g., multiple drives or multiple databases) spanning multiple computing devices (e.g., multiple server computers).

[0043] The data store 140 can store one or more of sensor data 142 (e.g., from sensors 126 associated with manufacturing tools 124), metrology data 129 (e.g., from metrology tools 128), performance data 152 (e.g., from metrology tools 128 used to perform metrology on substrates fabricated via manufacturing tools 124), predicted data 160 (e.g., predicted performance data), reference distribution data 162 (e.g., bin ranges, counts per bin, counts per bin histogram, binned training data, etc.), current distribution data 164 (e.g., counts per bin, counts per bin histogram, binned input data and predicted data, etc.), training data, input data, trained machine learning models (e.g., trained machine learning models 190), etc. In some embodiments, the performance data 152 may be data collected from a metrology tool 128 associated with a substrate.

[0044] In some embodiments, the input data is current sensor data. In some embodiments, the training data is historical sensor data 144 and / or historical performance data 154. In some embodiments, the predicted data 160 is output data of a trained machine learning model (e.g., trained machine learning model 190).

[0045] The sensor data 142 can include historical sensor data 144 and current sensor data 146. The performance data 152 can include historical performance data 154 and current performance data 156. The training data can refer to the sensor data 142 and / or the performance data 152 (e.g., the historical sensor data 144 and the historical performance data 154). For example, the server component 115 of the server device 120 can use the training data (e.g., the sensor data 142 and / or the performance data 152, the historical sensor data 144, and the historical performance data 154) to train a machine learning model (e.g., the trained machine learning model 190).

[0046] The trained machine learning model (e.g., trained machine learning model 190) can be used by the edge component 130 of the edge device 127. The edge component 130 can provide current sensor data 146 (e.g., data from sensors 126 associated with manufacturing equipment 124 in the same edge node 170 as the edge device 127) as input to the trained machine learning model (e.g., trained machine learning model 190) and receive output from the trained machine learning model associated with the predicted data 160. In some embodiments, the trained machine learning model 190 can be part of at least one of the edge node 170, the edge device 127, and / or the edge component 130.

[0047] In some embodiments, server device 120 provides reference distribution data 162 (e.g., to edge device 127, data store 140), and edge device 127 provides current distribution data 164 (e.g., to server device 120, data store 140). Server component 115 and / or edge component 130 can use reference distribution data 162 and / or current distribution data 164 to determine whether to perform corrective action (e.g., retraining a machine learning model in response to drift and / or differences between reference distribution data 162 and current distribution data 164).

[0048] In some embodiments, the data store 140 can store data associated with the processing of substrates in the fabrication tool 124. For example, the data store 140 can store sensor data 142 collected by sensors 126 associated with the fabrication tool 124 before, during, or after substrate processing. The sensor data 142 can represent historical sensor data 144 (e.g., sensor data generated for a previous substrate processed during a previous substrate processing in the fabrication system or processing chamber) and / or current sensor data 146 (e.g., sensor data generated for a current substrate processed during a previous substrate processing in the fabrication system or processing chamber).

[0049] In some embodiments, data store 140 may store prediction data 160 produced by trained machine learning model 190 (e.g., the output of the trained machine learning model).

[0050] In some embodiments, data store 140 may store data associated with substrates processed on fabrication tool 124. For example, data store 140 may store metrology data 129 collected by metrology tool 128 before, during, or after substrate processing. Metrology data 129 may represent historical metrology data (e.g., metrology data generated for substrates previously processed on the fabrication system) and / or current metrology data (e.g., metrology data generated for a substrate currently being processed on the fabrication system).

[0051] In some embodiments, data store 140 can store data associated with processing of substrates in fabrication tool 124. For example, data store 140 can store performance data 152 collected before, during, or after substrate processing, for example, by sensors 126 and / or metrology tools 128. Performance data 152 can represent historical performance data 154 (e.g., sensor data and / or metrology data generated for previous substrates processed during previous substrate processing in the fabrication system or processing chamber) and / or current performance data 156 (e.g., sensor data and / or metrology data generated for current substrates processed during previous substrate processing in the fabrication system or processing chamber).

[0052] The data store 140 can also store distribution data. The distribution data includes bin ratings, bin ranges for training data, input data, and / or predicted data (e.g., output data). In some embodiments, bin ranges for existing training data, input data, and / or predicted data are determined, and the training data, input data, and / or predicted data are stored in bins. The distribution data can be either reference distribution data or current distribution data.

[0053] The reference distribution data may be generated at the server device 120 using training data (e.g., data used to train a trained machine learning model). The current distribution data may be generated at the edge node 170 (e.g., an edge server) using input data and / or predicted data (e.g., output data) associated with the trained machine learning model 190 (e.g., data processed by the trained machine learning model).

[0054] The reference distribution data and / or the current distribution data may be histogram data. In some embodiments, the histogram data may be a chart including rectangles whose area is proportional to the frequency of a variable and whose width is equal to the class interval. In some embodiments, the histogram data may be graphical data used to represent the frequency distribution of data points for a variable. In some embodiments, the histogram data may classify data into various bins (e.g., range groups) and count the number of data points that fall into each bin.

[0055] In some embodiments, the server component 115 can compare distribution data (e.g., current distribution data and reference distribution data) to determine differences (e.g., difference index values) between two or more distribution data sets. In some embodiments, the server component 115 can compare the distribution data of a particular feature with a feature importance, or multiply the difference index value by a feature importance value. In some embodiments, feature importance is a measure of the importance of a feature in training a trained machine learning model (e.g., trained machine learning model 190) and can indicate how important a particular feature is to the accuracy of the trained machine learning model.

[0056] In some embodiments, the feature importance value may include or indicate the level of relevance of a particular feature to the operation of the trained machine learning model 190. In some embodiments, the feature importance value is a real number between 0 and 1, where 0 indicates that the feature is irrelevant or has no impact on the correct operation of the trained machine learning model 190 and 1 indicates that the feature is absolutely relevant or has a direct impact on the correct operation of the trained machine learning model 190.

[0057] In some embodiments, the feature importance values ​​may be determined by user input, a machine learning model, or the like.

[0058] In some embodiments, data store 140 can be configured to store data that is not accessible to users of manufacturing equipment 124 and / or edge node 170. For example, sensor data 142, performance data 152, forecast data 160, baseline distribution data, current distribution data, etc., may not be accessible to users (e.g., operators) of manufacturing equipment 124. In some embodiments, not all data stored in data store 140 may be accessible by users of manufacturing equipment 124 and / or edge node 170. In other or similar embodiments, some of the data stored in data store 140 may not be accessible by users, while other portions of the data stored in data store 140 may be accessible by users. In some embodiments, one or more portions of the data stored in data store 140 may be encrypted using an encryption mechanism unknown to the users (e.g., the data is encrypted using a private encryption key). In other or similar embodiments, data store 140 includes multiple data stores, where user-inaccessible data is stored in one or more first data stores and user-accessible data is stored in one or more second data stores.

[0059] In some embodiments, sensor data 142, historical sensor data 144, current sensor data 146, performance data 152, historical performance data 154, current performance data 156, predicted data 160, reference distribution data 162, and / or current distribution data 164 can be processed by server device 120, training data, input data, and / or edge device 127. Processing the data can include generating features. In some embodiments, a feature is a pattern (e.g., slope, width, height, peak, etc.) within the training data, input data, predicted data 160, reference distribution data 162, and / or current distribution data 164, or a combination of values ​​(e.g., power derived from voltage and current, etc.) from the training data, input data, predicted data 160, reference distribution data 162, and / or current distribution data 164. The data can include features, which can be used by server device 120 to determine the significance of the feature and / or to perform corrective action.

[0060] In some embodiments, a feature may be a type of sensor data (e.g., temperature, pressure, humidity). For example, a feature may be a combination of sensor data (e.g., power calculated from current and voltage). In some embodiments, a feature may be a pattern in the sensor data (e.g., slope, etc.). In some embodiments, a feature may be a type of metrology data (e.g., morphology, size attribute, dimensional attribute, image, scanning electron microscope (SEM) image, energy dispersive X-ray (EDX) image, defect distribution, spatial location, elemental analysis, wafer signature, chip layer, chip layout, gray level, signal-to-noise ratio, spacing, etc.). For example, a feature may be a combination of metrology data. In some embodiments, a feature may be a pattern in the metrology data.

[0061] Server device 120, edge device 127, metering equipment 128, and data store 140 may be interconnected via network 180. In some embodiments, network 180 is a public network that provides server device 120 with access to manufacturing equipment 124, data store 140, and other public computing devices. In some embodiments, network 180 is a private network that provides server device 120 with access to manufacturing equipment 124, metering equipment 128, data store 140, and other private computing devices. Network 180 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof.

[0062] In some embodiments, the server component 115 receives instructions for corrective actions and causes the corrective actions to be performed from the server device 120. The server device 120 may include an operating system that allows a user to create, view, or edit data (e.g., instructions associated with a trained machine learning model, instructions associated with a manufacturing equipment 124, corrective actions associated with a trained machine learning model 190, corrective actions associated with a manufacturing equipment 124, etc.).

[0063] The corrective action may be associated with one or more of computational process control (CPC), statistical process control (SPC) (e.g., SPC of electronic components to determine the process under control, SPC to predict the useful life of a component, SPC for comparison with 3 sigma graphs, etc.), advanced process control (APC), model-based process control, preventative maintenance, design optimization, manufacturing parameter updates, manufacturing recipe updates, equipment constant updates, feedback control, feedforward control, machine learning corrections, etc.

[0064] In some embodiments, the corrective action includes providing an alert (e.g., an alert indicating recommended action such as retraining a trained machine learning model, an alarm to stop or prevent a manufacturing process from running if model drift reaches a threshold). In some embodiments, taking the corrective action may include retraining a machine learning model associated with the manufacturing facility 124 and / or a machine learning model deployed to an edge device (e.g., running on an edge server). In some embodiments, taking the corrective action may include training a new machine learning model associated with the manufacturing facility 124.

[0065] In some embodiments, the functionality of server device 120 and edge device 127 may be provided by fewer machines. In some embodiments, server device 120 and edge device 127 may be combined into a single machine. In some embodiments, the functionality of server device 120, edge device 127, and data store 140 may be performed by a cloud-based service.

[0066] In general, functions described as being performed by server device 120 in some embodiments may also be performed by edge device 127 in other embodiments, where appropriate. In general, functions described as being performed by edge device 127 in some embodiments may also be performed by server device 120 in other embodiments, where appropriate. Furthermore, functions attributed to a particular component may be performed by a different component or multiple components operating in conjunction with one another. For example, in some embodiments, edge device 127 may determine corrective actions based on difference metric values. In other examples, server device 120 may determine corrective actions based on current distribution data 164 of trained machine learning model 190.

[0067] Furthermore, the functionality of a particular component may be performed by different components or multiple components working together. One or more of server device 120 or edge device 127 may be accessed as a service offered to other systems or devices via an appropriate application programming interface (API).

[0068] In some embodiments, the machine learning model (e.g., trained machine learning model 190) can be trained on a server device 120 (e.g., a central server) and deployed by an edge device 127 (e.g., a device associated with an edge tool of a manufacturing system). The server device 120 can have more computing power than the edge device 127 (e.g., the server device 120 can have significant computing power, while the edge device 127 can have relatively limited computing power). A process recipe includes parameters selected to produce process results, for example, to enable processing of substrates characterized by one or more target characteristics. A process recipe can include parameters selected and / or adjusted based on a product design, a target output, metrology of the target substrate, etc. A process recipe can include parameters such as process temperature, process pressure, process gases, radio frequency (RF) radiation characteristics, plasma characteristics, etc.

[0069] During use, the relationship between the output variables and input variables (e.g., input data, output data) of a trained machine learning model (e.g., trained machine learning model 190) may change (drift), causing the trained machine learning model to drift (model drift). For example, a machine learning model is trained using training data with values ​​within a certain range. The trained machine learning model may then be deployed and used to process data outside the range of the training data. The input drift may cause the output (e.g., predicted value) of the machine learning model to drift, resulting in model drift. The predicted value of the trained machine learning model may also drift, indicating that the relationship between the output variables and input variables of the trained machine learning model has drifted.

[0070] In some embodiments, the trained machine learning model may be deployed to edge devices and monitored by a server device (e.g., a central server device). In some embodiments, this configuration may be referred to as distributed monitoring of the machine learning model.

[0071] In embodiments, a "user" may be represented as a single individual. However, in other embodiments of the present disclosure, a "user" is an entity controlled by multiple users and / or automated sources. For example, a collection of individual users consolidated as a group of administrators may be considered a "user."

[0072] 2 is a sequence diagram 200 related to monitoring machine learning models deployed at the edge, according to some embodiments. FIG. 2 illustrates monitoring trained machine learning models deployed at the edge while maintaining privacy and performance. In some embodiments, the system includes an operator, a server device 220 (e.g., server device 120 of FIG. 1 ), a data store 240, a firewall 225 (e.g., deployed by server device 220, deployed by an additional device), and / or an edge node 270 (e.g., edge node 170 or edge device 127 of FIG. 1 ).

[0073] The server device 220 may include one or more of a user interface 224, a training pipeline 226, and / or a monitoring central 228. One or more of the user interface 224, the training pipeline 226, and / or the monitoring central 228 may be part of the server component 115 of FIG.

[0074] The edge node 270 may include one or more of a monitoring edge 272, a monitoring software development kit (SDK) 274, a model server 276, and / or an edge 278 (e.g., an edge node 170, an edge device 127, etc.). One or more of the monitoring edge 272, the monitoring SDK 274, the model server 276, and / or the edge 278 may be part of the edge component 130 of FIG. 1.

[0075] In some embodiments, operations 0.1-0.3 may be part of a training phase (e.g., pre-supervision setup). In operation 0.1, the server device 220 (e.g., via a training pipeline) may determine reference distribution data (e.g., reference distribution data 162 of FIG. 1 ) associated with features used to train the machine learning model and generate a trained machine learning model (e.g., trained machine learning model 190). This may involve, after training the machine learning model (e.g., trained machine learning model 190) using training data including inputs X and outputs Y, the server device 220 calculating a list of bins to describe the distribution of each feature in X and Y (where X may be a list of features (e.g., inputs) and Y may be an independent variable (e.g., output)). Further, the server device may calculate a reference distribution of data X and Y using the bins (e.g., classifying data X and Y into bins, such as drift bins). Furthermore, the server device 220 may calculate feature importance using Shapley Additive Algorithm (SHAP) and / or similar.

[0076] In operation 0.2, server device 220 (e.g., via a training pipeline) may provide the reference distribution data from operation 0.1 to data store 240 (e.g., data store 140 of FIG. 1 ). Additionally, in some embodiments, server device 220 provides feature importance data and drift bins to data store 240.

[0077] In operation 0.3, the server device 220 may send (e.g., via the server component 215) the reference distribution data (e.g., the list of bins calculated in operation 0.1) to each edge node 270 (e.g., the edge node 170 of FIG. 1 ) that runs the trained machine learning model (e.g., the trained machine learning model 190). In some embodiments, each edge node 270 obtains the reference distribution data from the data store 240. Data provided to or received from the edge node 270 may pass through a firewall 225 (e.g., to avoid privacy concerns).

[0078] Operations 1.1-1.3 may be part of a prediction phase (e.g., using the trained machine learning model 190 on an edge device). In operation 1.1, the edge node 270 (e.g., via the edge device 127 of FIG. 1 ) identifies (e.g., receives) sensor data (e.g., data from a sensor associated with a manufacturing equipment). Based on the sensor data, the edge node 270 determines features (e.g., inputs, input data) for making a prediction associated with the substrate processing equipment (e.g., generates prediction data 160, output data).

[0079] In operation 1.2, the edge node 270 provides input data (e.g., sensor data, feature quantities) to a trained machine learning model (e.g., trained machine learning model 190). In some embodiments, the edge 278 provides the input data to a model server to invoke predictions.

[0080] In operation 1.3, the edge node 270 (e.g., via the model server) generates output data associated with the predicted data (e.g., predicted data 160 of FIG. 1, predicted performance data using the features from operation 1.2).

[0081] Operations 2.1-2.9 may be part of an integrated drift monitoring phase. In operation 2.1, the edge node 270 (e.g., via the model server 276) sends feature quantities and predicted values ​​(e.g., predicted data 160 in FIG. 1 , the output of a trained machine learning model) to the monitoring SDK 274 of the edge node 270.

[0082] In operation 2.2, the edge node 270 (e.g., via the monitoring SDK 274) determines current distribution data (e.g., current distribution data 164 of FIG. 1 , calculates bin counts). The edge node 270 can use the bins received in operation 2.3 to generate histogram data (e.g., bin counts, current distribution data) for each feature and prediction (e.g., predicted data 160 of FIG. 1 ). For example, each histogram data can reflect the distribution of the data.

[0083] In operation 2.3, the edge node 270 (e.g., via the monitoring SDK 274) may send the current distribution data (e.g., the current distribution data 164 in FIG. 1 , the histogram data) to a monitoring agent (e.g., the monitoring edge 272 of the edge node 270). The current distribution data (e.g., the histogram data) may be retained over time, even as more prediction data is generated by the trained machine learning model (e.g., the trained machine learning model 190) at the edge node 270.

[0084] In operation 2.4, the edge node 270 (e.g., via each monitoring agent operating on the edge device when a data threshold is reached) may send the current distribution data (e.g., histogram data) to the server device 220 (e.g., the monitoring central 228 on the server device). In some embodiments, the current distribution data is provided from the edge node 270 to the data store 240, and the server device 220 retrieves the current distribution data from the data store 240.

[0085] In operation 2.5, the server device (eg, monitoring central 228) may obtain the reference distribution and feature importance data from data store 240.

[0086] In operation 2.6, the monitoring central 228 can calculate model drift for each feature quantity and predicted value, comparing the current data distribution with the reference distribution data. Common algorithmic methods, such as the Population Stability Index (PSI), can be used to calculate model drift. The server device (e.g., via the monitoring central 228) can also multiply the feature quantity importance values ​​by a drift result (drift index) to determine the impact of each feature quantity's drift on overall model performance (e.g., the drift value of a particular feature quantity may not have the same impact on model performance as other feature quantities).

[0087] In operation 2.7, the server device 220 (eg, via the monitoring central 228) may store the drift results (eg, difference index, drift index value) in the data store 240.

[0088] In operation 2.8, an alert may be sent to the user interface 224 or other notification mechanism in response to one or more drift values ​​(eg, difference index, drift index value) exceeding a threshold.

[0089] In operation 2.9, the operator can acknowledge the alert via the user interface 224.

[0090] In some embodiments, data privacy is maintained by transmitting summary statistics (e.g., bin counts, histogram data, etc.) from the edge node 270 (e.g., edge device) to the server device 220. In some embodiments, features and / or raw data are maintained at the edge node 270 and are not transmitted to the server device 220. In some embodiments, performance is improved by transmitting only small amounts of data (e.g., bin counts, histogram data, etc.) between the server device and the edge device.

[0091] In some embodiments, the monitoring SDK 274 and monitoring agents (e.g., monitoring edge 272) running on the edge node 270 (e.g., edge device) have minimal system overhead. In some embodiments, the monitoring SDK 274 and monitoring agents running on the edge node 270 can bin data and send summary statistics (e.g., histograms, distribution data, etc.). In some embodiments, the monitoring SDK 274 can run in the same memory space as the model server 276, which already contains features and predictions in memory, eliminating the need to send or transmit data outside of the process memory space where the data already resides.

[0092] FIG. 3 illustrates a block diagram 300 related to monitoring edge-deployed machine learning models, according to some embodiments.

[0093] In some embodiments, a server device (e.g., server device 120) determines a reference distribution 363 (e.g., a reference distribution graph) associated with input data (e.g., sensor data) and / or target output data (e.g., performance data, measurement data) used to train a machine learning model. In some embodiments, the reference distribution 363 can correspond to features of the input data and / or target output data. In some embodiments, the server device 220 determines reference distribution data 362 (e.g., bin ranges) based on the reference distribution 363 (e.g., of feature A) and provides the reference distribution data 362 (e.g., bin ranges) to the edge device 327 (e.g., edge device 127 of FIG. 1 ).

[0094] In some embodiments, reference distribution data 362 may be derived from or a subset of reference distribution 363. For example, reference distribution data 362 may include only a portion of reference distribution 363. For example, reference distribution data 362 may include only bin ranges, rather than counts for each bin included in reference distribution 363.

[0095] In some embodiments, current distribution data 364 may be derived from or a subset of current distribution 365. For example, current distribution data 364 may include only a portion of current distribution 365. For example, current distribution data 364 may include only counts per bin and not individual data point values ​​included in current distribution 365.

[0096] In some embodiments, the edge device 327 determines a current distribution 365 (e.g., a current distribution graph) by categorizing input data (e.g., sensor data) provided as input to the edge device's 327 trained machine learning model (e.g., trained machine learning model 190) and associated with features and / or predictions (e.g., predicted data associated with the output of the edge device's 127 trained machine learning model (e.g., trained machine learning model 190) responsive to the input) into corresponding bin ranges provided by the server device 320. For example, the bin ranges may include ranges from 0 to 20, 20 to 40, 40 to 60, 60 to 80, and 80 to 100. The server device 320 may categorize feature A data (e.g., training data) into bins corresponding to these ranges. The server device 320 may transmit the bin ranges to the edge device 327 on which a trained machine learning model corresponding to the reference distribution 363 is deployed.

[0097] In some embodiments, the edge device 327 sorts input data and / or predicted (output) data of the trained machine learning model (e.g., trained machine learning model 190) into bins corresponding to the bin ranges of the reference distribution data 362 to generate current distribution data 364 (e.g., counts per bin histogram). In some embodiments, a count number or counts per bin (e.g., a bin rating) may be determined for each bin. In some embodiments, the edge device 327 transmits the current distribution data 364 (e.g., counts per bin histogram) to the server device 320. In some embodiments, the server device 320 compares the current distribution data 364 with the reference distribution data 362 to determine the difference between the two data sets. In some embodiments, the difference between the current distribution data 364 and the reference distribution data 362 may be expressed as a difference index or a difference index value. For example, the difference index value may be expressed as a decimal value between 0 and 1. In some embodiments, the difference index value may be scaled by multiplying it by the feature importance before comparing the difference index value to a threshold. In some embodiments, in response to the dissimilarity index value and / or the current distribution data 364 meeting a threshold, the server device 320 may cause corrective action to be taken.

[0098] 4A-B are flow diagrams of methods 400A-B related to monitoring edge-deployed machine learning models, according to certain embodiments. Methods 400A-B can relate to training and utilizing machine learning models, statistical models, rule-based models, heuristic models, physics-based models, etc. Methods 400A-B can be performed by processing logic, which can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, processing devices, etc.), software (e.g., instructions executed on a processing device, general-purpose computer system, or dedicated machine), firmware, microcode, or a combination thereof. In some embodiments, a non-transitory machine-readable storage medium stores instructions that, when executed by the processing device, cause the processing device to perform one or more of methods 400A-B. In some embodiments, a system (e.g., server device 120 or edge device 127 of FIG. 1 ) includes a memory and a processing device coupled to the memory that performs one or more of methods 400A-B.

[0099] For ease of explanation, methods 400A-B are shown and described as a series of operations. However, operations according to the present disclosure may be performed in various orders and / or simultaneously, and in parallel with other operations not presented and described herein. Moreover, not all of the illustrated operations may be performed to perform methods 400A-B in accordance with the disclosed subject matter. Furthermore, those skilled in the art will understand or appreciate that methods 400A-B may alternatively be represented as a series of interrelated states via a state diagram or events.

[0100] FIG. 4A is a flow diagram of a method 400A for private monitoring of edge machine learning models, according to some embodiments.

[0101] 4A , in some embodiments, at block 402, processing logic performing method 400A (e.g., server device 120 of FIG. 1 ) determines reference distribution data associated with features used to train the machine learning model and generates a trained machine learning model (e.g., trained machine learning model 190). The trained machine learning model may have been trained using data inputs of historical sensor data and target outputs of historical performance data.

[0102] In some embodiments, the reference distribution data includes bin ranges of training data (e.g., historical sensor data and historical performance data) associated with the features, and the training data is used to train the machine learning model.

[0103] At block 404, processing logic provides the reference distribution data to an edge device associated with the substrate processing equipment.

[0104] At block 406, the processing logic receives current distribution data associated with the features from the edge device in response to using a trained machine learning model at the edge device (e.g., trained machine learning model 190). Using the trained machine learning model includes providing the current sensor data to the trained machine learning model and receiving output from the trained machine learning model that is associated with predicted data.

[0105] In some embodiments, the current distribution data is based on binning at least one of the input data (e.g., current sensor data) or the predicted data (e.g., predicted performance data) via an edge device. In some embodiments, the current distribution data may include per-bin count histogram data associated with at least one of the input data or the predicted data.

[0106] At block 408, processing logic performs corrective actions associated with the trained machine learning model based on the current distribution data.

[0107] In some embodiments, the corrective action may include at least one of providing an alert or retraining a trained machine learning model. In some embodiments, the processing logic further performs a distribution comparison based on reference distribution data and current distribution data. In some embodiments, the reference distribution data and the current distribution data are histogram data.

[0108] In some embodiments, processing logic may further identify training data associated with the feature, hi some embodiments, processing logic may further bin the training data, and determining the reference distribution data includes determining bin ranges for the binned training data.

[0109] In some embodiments, the processing logic may further determine a difference (e.g., a difference index) between the current distribution data and the reference distribution data. In some embodiments, the processing logic may further determine that the difference between the current distribution data and the reference distribution data meets a threshold, and the execution of the corrective action is responsive to the difference meeting the threshold.

[0110] In some embodiments, determining the difference between the current distribution data and the reference distribution data can use common algorithmic techniques such as the Population Stability Index (PSI). In some embodiments, determining the difference between the current distribution data and the reference distribution data can include multiplying the feature importance by the difference. In some embodiments, such multiplication by the feature importance can be weighted before comparing the difference index value to a threshold. In some embodiments, such weighting of the difference index value ensures that the model is not retrained unnecessarily (e.g., when a feature has low importance).

[0111] FIG. 4B is a flow diagram of a method 400B for private monitoring of edge machine learning models, according to some embodiments.

[0112] Referring to FIG. 4B , in some embodiments, at block 412, processing logic (e.g., edge device 127 of FIG. 1 ) performs method 400B to receive, from a server device (e.g., server device 120 of FIG. 1 ), reference distribution data associated with features used to train the machine learning model to generate a trained machine learning model (e.g., trained with data inputs including historical sensor data and target outputs including historical performance data).

[0113] In some embodiments, the reference distribution data includes bin ranges of training data (e.g., historical sensor data and / or historical performance data) associated with the features, the training data being used to train the machine learning model.

[0114] At block 414, processing logic uses the trained machine learning model based on input data (e.g., current sensor data) associated with the substrate processing equipment.

[0115] At block 416, processing logic determines current distribution data associated with the feature in response to using the trained machine learning model.

[0116] In some embodiments, the current distribution data may include bin ranges for at least one of input data (e.g., current sensor data) or predicted data (e.g., predicted performance data) associated with the feature, where the input data is used to generate the predicted data via the machine learning model. In some embodiments, the processing logic may further identify the input data and predicted data associated with the feature. In some embodiments, the processing logic may further bin at least one of the input data or the predicted data, where determining the reference distribution data includes determining bin ranges for the binned input data and predicted data.

[0117] At block 418, the processing logic provides the current distribution data to a server device to perform corrective actions associated with the trained machine learning model.

[0118] In some embodiments, taking corrective action may include at least one of providing a warning or retraining a trained machine learning model.

[0119] 5 is a block diagram illustrating a computer system 500 according to certain embodiments. In some embodiments, the computer system 500 is one or more of a server device 120, an edge device 127, and / or a similar device.

[0120] In some embodiments, computer system 500 is connected to other computer systems (e.g., via a network such as a local area network (LAN), an intranet, an extranet, or the Internet). In some embodiments, computer system 500 operates as a server or a client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. In some embodiments, computer system 500 is provided by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch, a bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be performed by the device. Furthermore, the term "computer" includes a collection of computers that individually or collectively execute a set of instructions (or instructions) to perform one or more of the methodologies described herein.

[0121] In a further aspect, the computer system 500 includes a processing device 502, a volatile memory 504 (e.g., random access memory (RAM)), a non-volatile memory 506 (e.g., read-only memory (ROM) or electrically erasable programmable ROM (EEPROM)), and a data storage device 518, which communicate with each other via a bus 508.

[0122] In some embodiments, processing device 502 is provided by one or more processors, such as a general-purpose processor (e.g., a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor that executes other types of instruction sets, or a microprocessor that executes a combination of types of instruction sets, etc.) or a special-purpose processor (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor, etc.).

[0123] Additionally, in some embodiments, computer system 500 includes a network interface device 522 (e.g., coupled to a network 574). Also, in some embodiments, computer system 500 includes a video display unit 510 (e.g., a liquid crystal display (LCD)), an alphanumeric input device 512 (e.g., a keyboard), a cursor control device 514 (e.g., a mouse), and a signal generating device 520.

[0124] In some embodiments, the data storage device 518 includes a non-transitory computer-readable storage medium 524 having stored thereon instructions 526 encoding one or more of the methods or functions described herein, including instructions encoding the components of FIG. 1 (e.g., the server component 115, the edge component 130, etc.) and instructions for performing the methods described herein (e.g., one or more of methods 400A-B).

[0125] In some embodiments, the instructions 526 reside, completely or partially, within the volatile memory 504 and / or within the processing unit 502 during execution by the computer system 500. Thus, in some embodiments, the volatile memory 504 and the processing unit 502 also constitute machine-readable storage media.

[0126] Although computer-readable storage medium 524 is shown as a single medium in the illustrated example, the term "computer-readable storage medium" includes a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of executable instructions. The term "computer-readable storage medium" also includes any tangible medium that stores or encodes a set of instructions that can be executed by a computer to cause the computer to perform any one or more of the methodologies described herein. The term "computer-readable storage medium" includes, but is not limited to, solid-state memory, optical media, magnetic media, etc.

[0127] The methods, components, and features described herein can be implemented by discrete hardware components or integrated into the functionality of other hardware components, such as an application specific integrated circuit (ASIC), FPGA, DSP, or similar device. Additionally, the methods, components, and features can be implemented by firmware modules or functional circuitry within a hardware device. Additionally, the methods, components, and features can be performed by any combination of hardware devices and computer program components, or by a computer program.

[0128] Terms such as "determine," "provide," "receive," "perform," "identify," "sort," "execute," "use," "retrain," "obtain," "access," "add," "train," and the like refer to operations and processes performed or implemented by a computer system that manipulate and transform data represented as physical quantities (electronic quantities) in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display device. Additionally, terms such as "first," "second," "third," "fourth," and the like, as used herein, are intended as labels to distinguish different elements and are not intended to have an ordinal numerical designation.

[0129] Examples described herein also relate to apparatus for performing the methods described herein. The apparatus may be specially configured to perform the methods described herein, or may include a general-purpose computer system selectively programmed by a computer program stored on the computer system. Such a computer program may be stored on a computer-readable tangible storage medium.

[0130] The methods and examples described herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used in accordance with the teachings described herein. It may also be convenient to construct more specialized apparatus to perform the methods described herein and / or their individual functions, routines, subroutines, or operations. Examples of structures for these various systems are set forth in the description above.

[0131] The above description is intended to be illustrative, not limiting. While the present disclosure has been described with reference to specific examples and embodiments, it will of course be understood that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. determining reference distribution data associated with the features used to train the machine learning model and generating a trained machine learning model; providing reference distribution data to an edge device associated with the substrate processing equipment; receiving current distribution data associated with the feature from the edge device in response to using the trained machine learning model at the edge device; The method includes performing a corrective action associated with the trained machine learning model based on current distribution data.

2. The method of claim 1 , wherein the corrective action includes at least one of providing a warning or retraining a trained machine learning model.

3. The method of claim 1 , wherein the reference distribution data includes bin ranges of training data associated with the features, the training data being used to train the machine learning model.

4. identifying training data associated with the features; The method of claim 3 , wherein the step of classifying the training data into bins comprises determining bin ranges for the training data to be binned, and wherein determining the reference distribution data comprises determining bin ranges for the training data to be binned.

5. determining a difference between the current distribution data and the reference distribution data; The method of claim 1 , further comprising determining that a difference between the current distribution data and the reference distribution data meets a threshold, and performing corrective action in response to the difference meeting the threshold.

6. 5. The method of claim 4, wherein the current distribution data is based on classifying at least one of the input data or the expected data into bins via an edge device, and the current distribution data comprises count per bin histogram data associated with at least one of the input data or the expected data.

7. The method of claim 1 , comprising performing a distribution comparison based on reference distribution data and current distribution data.

8. The method of claim 1 , wherein the reference distribution data and the current distribution data are histogram data.

9. receiving, from a server device, reference distribution data associated with features used to train the machine learning model, and generating a trained machine learning model; using the trained machine learning model based on input data associated with the substrate processing equipment; determining current distribution data associated with the features in response to using the trained machine learning model; A method comprising providing current distribution data to a server device to cause the server device to perform corrective actions associated with the trained machine learning model.

10. The method of claim 9 , wherein taking corrective action includes at least one of providing a warning or retraining a trained machine learning model.

11. The method of claim 10 , wherein the reference distribution data includes bin ranges of training data associated with the features, the training data being used to train the machine learning model.

12. 12. The method of claim 11, wherein the current distribution data includes at least one bin range of the input data or the predicted data associated with the feature, and the input data is used to generate the predicted data via a machine learning model.

13. identifying input data and prediction data associated with the features; The method of claim 12 , wherein the step of classifying at least one of the input data or the predicted data into bins includes determining bin ranges for the input data and the predicted data to be binned.

14. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a processing device to perform operations, the operations including: determining reference distribution data associated with the features used to train the machine learning model and generating a trained machine learning model; providing reference distribution data to an edge device associated with the substrate processing equipment; receiving current distribution data associated with the feature from the edge device in response to using the trained machine learning model at the edge device; 12. A machine-readable storage medium comprising: performing a corrective action associated with a trained machine learning model based on current distribution data.

15. 15. The non-transitory machine-readable storage medium of claim 14, wherein the corrective action comprises at least one of providing a warning or retraining a trained machine learning model.

16. 15. The non-transitory machine-readable storage medium of claim 14, wherein the reference distribution data includes bin ranges of training data associated with the features, the training data being used to train the machine learning model.

17. The operation is identifying training data associated with the features; 17. The non-transitory machine-readable storage medium of claim 16, wherein binning training data, and determining the reference distribution data includes determining bin ranges for the binned training data.

18. The operation is determining a difference between the current distribution data and the reference distribution data; 15. The non-transitory machine-readable storage medium of claim 14, further comprising determining that a difference between the current distribution data and the reference distribution data meets a threshold, and performing corrective action in response to the difference meeting the threshold.

19. 20. The non-transitory machine-readable storage medium of claim 17, wherein the current distribution data is based on classifying at least one of the input data or the expected data into bins via an edge device, and the current distribution data comprises count per bin histogram data.

20. The non-transitory machine-readable storage medium of claim 14 , comprising performing a distribution comparison based on reference distribution data and current distribution data.

Images