Generating indication of learning of model for semiconductor processing

Abstract

A method includes receiving a first value associated with a first input parameter of a model. The method further includes receiving a first plurality of values associated with a second input parameter of the model. The method further includes providing the first value and the first plurality of values to a model. The method further includes receiving a first plurality of outputs from the model. The method further includes preparing a first plurality of outputs for presentation via a presentation component of the graphical user interface. The presentation assembly includes two shafts. The first axis is associated with a first property of a first feature associated with an output from the model. The second axis is associated with a second property of the first feature.

Description

Technical Field

[0001] The present specification relates to trained models. In particular, the present specification relates to generating instructions that present model learning for a model associated with semiconductor processing. Background Art

[0002] Chambers are used in many types of processing systems. Examples of chambers include etching chambers, deposition chambers, annealing chambers, and the like. Typically, a substrate (e.g., a semiconductor wafer) is placed on a substrate support within the chamber, and conditions in the chamber are set and maintained to process the substrate. Typically, models are used to improve the process. Models can include trained machine learning models as well as physics-based models. Summary of the invention

[0003] The following is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is neither intended to identify key or important elements of the disclosure, nor to describe any scope of a specific embodiment of the disclosure or any scope of the claims. The only purpose is to present some concepts of the disclosure in a simplified form as a preface to a more detailed description presented later.

[0004] In one aspect of the present disclosure, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform operations. The operations include receiving a first value associated with a first input parameter of a model. The first input parameter is associated with a first processing condition of a semiconductor wafer processing process. The operations further include receiving a first plurality of values. The first plurality of values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values. Each of the first plurality of values ​​is associated with a second input parameter of the model. The second input parameter is associated with a second processing condition of the semiconductor wafer processing process. The operations further include providing the first value and the first plurality of values ​​to the model. The operations further include receiving a first plurality of outputs from the model. Each of the first plurality of outputs is associated with a first value and one of the first plurality of values. Each of the first plurality of outputs is associated with a first feature of one of the first plurality of simulated substrates. The operations further include preparing the first plurality of outputs for presentation via a presentation component of a graphical user interface (GUI). The presentation component includes two axes. A first axis of the two axes corresponds to a first property of the first feature. A second axis of the two axes corresponds to a second property of the first feature. Preparing the first plurality of outputs for presentation includes facilitating generation of a graphic for display in a presentation component, the graphic indicating a value of the first property of the first feature and a value of the second property of the first feature associated with each of the first plurality of outputs.

[0005] In another aspect of the present disclosure, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform operations. The operations receive a first plurality of outputs from a first model. Each of the first plurality of outputs is associated with a first input value and one of the first plurality of input values. Each of the first plurality of outputs is associated with a first characteristic of one of the first plurality of simulated substrates. The operations further include receiving a second plurality of outputs from a second model. Each of the second plurality of outputs is associated with a first value and one of the first plurality of values. Each of the second plurality of outputs is associated with a second characteristic of one of the first plurality of simulated substrates. The operations further include preparing the first plurality of outputs and the second plurality of outputs for presentation via a presentation component of a GUI. The presentation component includes two axes. The first axis corresponds to a first property of the first characteristic. The second axis corresponds to a second property of the second characteristic. Preparing the first plurality of outputs and the second plurality of outputs for presentation includes facilitating the generation of a graphic for display in the presentation component, the graphic indicating a value of the first characteristic of each simulated substrate of the first plurality of simulated substrates and a value of the second characteristic of the second characteristic.

[0006] In another aspect of the present disclosure, a method includes receiving a first value associated with a first input parameter of a first model. The first input parameter is associated with a process recipe for processing a substrate. The method further includes receiving a first plurality of values ​​by one or more processors. The first plurality of values ​​ranges from a minimum value to a maximum value. Each of the first plurality of values ​​is associated with a second input parameter of the first model. The second input parameter is associated with a process recipe. The method further includes providing the first value and the first plurality of values ​​to the first model. The method further includes receiving a first plurality of outputs from the first model. Each of the first plurality of outputs is associated with the first value and one of the first plurality of values. Each of the first plurality of outputs is associated with a first feature of a simulated substrate. The method further includes preparing the first plurality of outputs for presentation via a presentation component of a GUI. The presentation component includes two independent axes. A first axis of the two independent axes corresponds to a first property of the first feature. Preparing the first plurality of outputs for presentation includes facilitating the generation of a graph for use in the presentation component, the graph visually displaying a relationship of an output in the first plurality of outputs to the first property of the first feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0008] Figure 1 is a block diagram illustrating an exemplary system (exemplary system architecture) in accordance with some embodiments.

[0009] Figure 2is a block diagram of an exemplary dataset generator for creating a dataset for a model according to some embodiments.

[0010] Figure 3 is a block diagram illustrating a system for generating output data for learning of an analytical model according to some embodiments.

[0011] Figure 4A is a flowchart of a method for generating a data set for a machine learning model according to some embodiments.

[0012] Figure 4B is a flow diagram of a method for generating a learning visual representation of a model according to some embodiments.

[0013] Figure 4C is a flow chart of a method for generating a visual representation of learning of multiple models according to some embodiments.

[0014] Figure 4D is a flow chart of a method for generating a graph demonstrating the learning of a model according to some embodiments.

[0015] Figure 5A is an exemplary presentation component that displays an indication of model learning for one or more models in accordance with some embodiments.

[0016] Figure 5B is an exemplary presentation component depicting learning of one or more models according to some embodiments.

[0017] Figure 5C is an exemplary GUI according to some embodiments.

[0018] Figure 5D is an exemplary presentation component depicting model learning associated with various varying input parameters, according to some embodiments.

[0019] Figure 6 is a block diagram illustrating a computer system according to some embodiments. DETAILED DESCRIPTION

[0020] Described herein are techniques, methods, systems, and apparatuses related to systematically tracking and / or reporting learning performed by machine learning models and / or other types of models (e.g., statistical models). For example, in some embodiments, a machine learning model associated with the manufacture and / or processing of a substrate is discussed. In some embodiments, a visual indication of model learning is presented. In some embodiments, visual indications are provided for high-dimensional input and output spaces (e.g., for machine learning models and for statistical models). An embodiment provides a graphical user interface (GUI) that enables a user to easily visualize high-dimensional input and output spaces and the relationships therebetween. The GUI enables a user to see curves showing how the input is tracked through the output space under any input condition. Thus, the GUI enables a user to see how changing the input affects the output of a machine learning model or a statistical model. For example, in an embodiment, the GUI can provide insights into what the machine learning model has learned and give information about the process being modeled.

[0021] Manufacturing equipment is used to produce substrates (e.g., semiconductor wafers). The properties of these substrates are determined by the conditions used to process the substrates. Accurate knowledge of the values ​​of properties in a manufacturing chamber during operation (especially in close proximity to a substrate) can be used to predict the properties of the final product, consistently produce substrates with the same properties, and tailor processing parameters to optimize substrate production.

[0022] Machine learning models and statistical models can be used to understand and / or predict substrate processing. For example, machine learning models and / or statistical models can be trained to receive input values ​​indicating processing conditions of a substrate. In some cases, processing parameters can be provided as input to the model (e.g., processing recipe set points (e.g., heater power, plasma generation power, duration of processing, etc.)). In some cases, processing conditions can be provided as input to the model (e.g., sensor data collected during substrate processing (e.g., temperature, pressure, component actuation, etc.)). In some cases, a combination of data types can be provided as input to the model. The model can be configured to generate an indication of the output of the processing process (e.g., one or more predicted measurements of the substrate generated by the processing conditions indicated by the input of the model).

[0023] In some cases, it may be less convenient to generate an understanding for model learning. For example, in some cases, a user may consider using a trained machine learning model to develop an understanding (e.g., an intuitive understanding) of how an output (e.g., a metrology of a produced substrate) is related to one or more inputs. In a conventional system, a user may provide a set of input conditions to a model (e.g., a machine learning model) and receive one or more predictions of an output substrate from the model. Then, the user may provide a set of different input conditions to the model and receive one or more predictions of an output substrate associated with the new input conditions from the model. Using a conventional system to generate an understanding for model learning, compactly displaying the correlation between input and output, or the like may be cumbersome, time-consuming, and the like. For example, it may be difficult to discern the effect of one input on multiple possible outputs (e.g., multiple predicted measurements of a simulated substrate, multiple predicted properties of a simulated substrate, simulated metrology measurements, etc.). It may be difficult to discern the effect of changing multiple inputs (e.g., changing multiple processing recipe parameters) on one or more outputs. It may be difficult to determine the strength of the correlation between input and output. It may be difficult to determine any nonlinearity of the correlation between input and output. It may be difficult to determine the range of the model's available input space or output space and the relationship between a specific input and / or output and the range of space.

[0024] In some conventional systems, the impact of inputs on outputs can be displayed, for example, via a bar graph. Multiple inputs can be placed on one axis (e.g., the x-axis) of a bar graph, while outputs can be placed on an orthogonal axis (e.g., the y-axis). The change of one input relative to some groups of "center" or "baseline" inputs at a time can be represented by a bar of a graph (e.g., the slope of the output space in the direction of the input space associated with one input can be represented by the size of the bar). In such a system, it may not be possible to represent the change in the output response to the change of the input over the span of the input space (e.g., the output may respond differently to the same change in the input from different groups of baseline input conditions). In addition, in such a representation, the impact on a second feature of interest (e.g., a second output) that may be related to the first output is not captured. In some cases, it may be difficult to determine the range of the input and / or output space of the model from such a representation. It may be difficult to determine the relationship between the output associated with a set of inputs and the range of the output space.

[0025] The technology, method, and system of the present disclosure can solve one or more of these shortcomings of conventional systems. In some embodiments, one or more models (e.g., machine learning models, physics-based models, statistical models, etc.) can receive a set of inputs indicating processing parameters of a substrate. The model can generate one or more indications of predicted features of a substrate (e.g., a semiconductor wafer) associated with the set of inputs as output. Features can include measurable quantities on any substrate (e.g., thickness, resistivity, refractive index, extinction coefficient, sheet resistance, geometric measurements (critical dimensions, line segment width, etch depth, sidewall height, etc.), reflectivity, surface characteristics, or the like). In some embodiments, the model can be configured to generate data indicating features of the substrate (e.g., the model can generate one or more indications of predicted thickness measurements of the substrate as output). In some embodiments, the nature of the feature may be of interest. For example, the model can predict the thickness of the substrate at several spatial locations of the substrate. The statistical properties of the predicted features (e.g., the mean, median, standard deviation, uniformity, etc. of the thickness) can be calculated.

[0026] In some embodiments, a graphical user interface (GUI) can be used to update the settings of a tool (e.g., a software tool) to be used when presenting model learning and / or the relationship between the input and output of the model. The GUI can further display the results of the model learning and / or the relationship between the input and output of the model. The GUI may include a presentation component for displaying one or more graphs presenting information related to model learning and / or the relationship between the input and output of the model. The GUI may receive one or more instructions (e.g., via user input). The GUI may receive, for example, user indications of the properties of features and / or features of interest (e.g., a user may select one or more outputs of one or more models drawn via a presentation component), one or more configuration settings (e.g., customizing the appearance of a graph depicting model learning, customizing the range of data calculated or shown, etc.), or the like. In some embodiments, various data may be displayed on the same graph via a presentation component (e.g., outputs associated with multiple sets of input conditions may be displayed on the same graph).

[0027] In some embodiments, the output of one or more machine learning models and / or other models (e.g., statistical models) can be presented on a graph (e.g., a scatter plot). In some embodiments, the output from a model can be represented on two axes (e.g., one axis can be associated with one property of an output feature (e.g., a statistical property) and a second axis can be associated with a second property of the output feature). In some embodiments, one axis can be associated with a property of a first output feature and a second axis can be associated with a property of a second output feature (e.g., output from a second model).

[0028] In some embodiments, the range of the output space of one or more models can be represented. For example, many combinations of inputs (e.g., spanning the input space used to train or configure one or more models) can be provided to one or more models. Each of the outputs associated with a combination of inputs can be presented on the same graph as one or more outputs associated with a particular set of inputs of interest. In some embodiments, data points representing the range of the model output space can be presented differently from specific model outputs (e.g., can be presented in the context of model outputs associated with user-specified inputs, inputs associated with process optimization, inputs associated with a substrate processing recipe to be optimized, or the like).

[0029] In some embodiments, a set of center or baseline inputs may be provided to one or more models. Outputs associated with the set of baseline inputs may be presented on the same graph as outputs associated with a range of output space. Outputs associated with a range of output space may provide a visual indication of the limits of a range of outputs for one or more models (e.g., a training output space for one or more models) compared to specific outputs of interest (e.g., associated with a particular set of inputs, associated with model learning, etc.).

[0030] In some embodiments, one or more groups of inputs of interest can be further provided to one or more models, and the associated outputs are displayed on the graph. For example, multiple values ​​of an input parameter can be provided to one or more models, while other input parameters are maintained at baseline values. For example, all inputs except one input can be frozen at initial values ​​(e.g., baseline values). A series of values ​​associated with an input that is not frozen can be generated. For example, the range of the series of values ​​can be from the minimum input value to the maximum input value (e.g., from the minimum value of the input in the trained input space of one or more models to the maximum value of the input in the trained input space of one or more models). In some embodiments, the range of output can be displayed via the presentation component of GUI. In some embodiments, a visual indicator can be displayed to distinguish the output value associated with the minimum value of the input that is changed and the output associated with the maximum value of the input that is changed. This series of outputs can generate a curve indicating model learning or configuration (the shape, length, and density of the data points along the curve can indicate the impact of changing the input from the minimum input value to the output space of the maximum input value).

[0031] In some embodiments, multiple inputs can be changed. For example, for the first input, a range of ranges can be generated. A series of outputs can be received from one or more models, each associated with different values ​​of the first input and the baseline values ​​of all other inputs. Then, the second input can be changed. For the second input, a range of values ​​can be generated. A series of outputs can be received from one or more models, each associated with different values ​​of the second input and the baseline values ​​of all other inputs (e.g., including the first input). Then, all outputs (e.g., outputs associated with the range of the output space of the one or more models described, outputs associated with changing the first input, outputs associated with changing the second output, etc.) can be displayed via a presentation component. In some embodiments, the presentation of the output associated with changing the first input can be visually distinguished from the presentation of the output associated with changing the second input, for example, via color, shape, pattern, or the like. In some embodiments, a presentation component can display the results of changing several input parameters. A graph including several curves (e.g., a set of points associated with various values ​​of the range from the minimum to the maximum value of the input parameter) can be generated, each of which intersects / intersects at the output associated with the baseline input condition.

[0032] In some embodiments, the system may receive (e.g., via a GUI) a user indication of a baseline condition. In some embodiments, the system may receive (e.g., via a GUI) a user indication of a range (e.g., minimum and maximum values) of one or more inputs to be changed. In some embodiments, the system may receive (e.g., via a GUI) an indication of the spacing of the inputs to be changed (e.g., how much the value of the input is to be changed to provide spacing between input values ​​to one or more models, etc.). In some embodiments, the system may provide one or more sets of inputs to one or more models in response to user selection of one or more settings (e.g., via a GUI).

[0033] Compared to conventional methods, aspects of the present disclosure provide technical advantages. In some embodiments, as input parameters are changed through a range of values, the impact on one or more output metrics can be displayed (e.g., presented by the placement of a set of data points relative to multiple axes). Nonlinearity can be captured visually (e.g., the difference in the change of output values ​​for a given change of input values ​​depending on the initial input value). A range of values ​​associated with multiple input parameters can be displayed simultaneously or together. As understood by one or more models, the display (e.g., a graph displayed via a presentation component of a GUI) can include a visual record of learning associated with training one or more machine learning models (or, in some embodiments, a physics-based model, etc.) (e.g., a visual record of the impact of changing one or more input values ​​on output values). In some embodiments, the user can easily transform the settings / parameters of the presentation component, graph, or the like. For example, user selections of one or more features (e.g., one or more predicted properties of simulated wafers, simulated metrology measurements, etc.), user selections of one or more properties of features (e.g., statistical metrics, geometric constraints, etc.), user selections of baseline groups of input values, or the like can be easily adjusted. The graph can be quickly updated to display a record of model learning associated with the updated user selection. In this way, a user can navigate the output space of a model, can develop an intuitive understanding of model learning, can develop an intuitive understanding of the mapping of process inputs to output properties, can adjust process parameters or process recipes for target properties, or the like.

[0034] In one aspect of the present disclosure, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform operations. The operations include receiving a first value associated with a first input parameter of a model. The first input parameter is associated with a first processing condition of a substrate processing process. The operations further include receiving a first plurality of values. The first plurality of values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values. Each of the first plurality of values ​​is associated with a second input parameter of the model. The second input parameter is associated with a second processing condition of the substrate processing process. The operations further include providing the first value and the first plurality of values ​​to the model. The operations further include receiving a first plurality of outputs from the model. Each of the first plurality of outputs is associated with a first value and one of the first plurality of values. Each of the first plurality of outputs is associated with a first feature of one of the first plurality of simulated substrates. The operations further include preparing the first plurality of outputs for presentation via a presentation component of a graphical user interface (GUI). The presentation component includes two axes. A first axis of the two axes corresponds to a first property of the first feature. A second axis of the two axes corresponds to a second property of the first feature. Preparing the first plurality of outputs for presentation includes facilitating generation of a graphic for display in a presentation component, the graphic indicating a value of a first property of a first feature and a value of a second property of the first feature associated with each of the first plurality of outputs.

[0035] In another aspect of the present disclosure, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform operations. The operations receive a first plurality of outputs from a first model. Each of the first plurality of outputs is associated with a first input value and one of the first plurality of input values. Each of the first plurality of outputs is associated with a first characteristic of one of the first plurality of simulated substrates. The operations further include receiving a second plurality of outputs from a second model. Each of the second plurality of outputs is associated with a first value and one of the first plurality of values. Each of the second plurality of outputs is associated with a second characteristic of one of the first plurality of simulated substrates. The operations further include preparing the first plurality of outputs and the second plurality of outputs for presentation via a presentation component of a GUI. The presentation component includes two axes. The first axis corresponds to a first property of the first characteristic. The second axis corresponds to a second property of the second characteristic. Preparing the first plurality of outputs and the second plurality of outputs for presentation includes facilitating the generation of a graphic for display in the presentation component, the graphic indicating a value of the first characteristic of each simulated substrate of the first plurality of simulated substrates and a value of the second characteristic of the second characteristic.

[0036] In another aspect of the present disclosure, a method includes receiving a first value associated with a first input parameter of a first model. The first input parameter is associated with a process recipe for processing a substrate. The method further includes receiving a first plurality of values ​​by one or more processors. The first plurality of values ​​ranges from a minimum value to a maximum value. Each of the first plurality of values ​​is associated with a second input parameter of the first model. The second input parameter is associated with a process recipe. The method further includes providing the first value and the first plurality of values ​​to the first model. The method further includes receiving a first plurality of outputs from the first model. Each of the first plurality of outputs is associated with the first value and one of the first plurality of values. Each of the first plurality of outputs is associated with a first feature of a simulated substrate. The method further includes preparing the first plurality of outputs for presentation via a presentation component of a GUI. The presentation component includes two independent axes. A first axis of the two independent axes corresponds to a first property of the first feature. Preparing the first plurality of outputs for presentation includes facilitating the generation of a graph for use in the presentation component, the graph visually displaying a relationship of an output in the first plurality of outputs to the first property of the first feature.

[0037] Figure 1 1 is a block diagram illustrating an exemplary system 100 (exemplary system architecture) according to some embodiments. System 100 includes client device 120, manufacturing equipment 124, sensor 126, metrology equipment 128, prediction server 112, and data storage 140. Prediction server 112 can be part of prediction system 110. Prediction system 110 can further include server machines 170 and 180. Client device 120 can include presentation component 115, and can perform combined 4A to 4D One or more of the methods described. In some embodiments, presentation component 115 may be included in whole or in part in a different component of system 100 (eg, prediction server 112, server machine 170, or server machine 180).

[0038] In some embodiments, the manufacturing equipment 124 (e.g., a cluster tool) is part of a substrate processing system (e.g., an integrated processing system). The manufacturing equipment 124 includes one or more of a controller, a housing system (e.g., a substrate carrier, a front opening wafer transfer box (FOUP), a FOUP, a process kit housing system, a substrate housing system, a box, etc.), a side storage box (SSP), an aligner device (e.g., an aligner chamber), a factory interface (e.g., an equipment front end module (EFEM)), a load lock, a transfer chamber, one or more processing chambers, a robot (e.g., disposed in the transfer chamber, disposed in the front interface, etc.), and / or the like. The housing system, SSP, and load lock mounted to the factory interface and the robot disposed in the factory interface are used to transfer contents (e.g., substrates, process kit rings, carriers, verification wafers, etc.) between the housing system, SSP, load lock, and factory interface. The alignment device is disposed in the factory interface to align the contents. The load lock and processing chamber mounted to the transfer chamber and the robot arm disposed in the transfer chamber are used to transfer contents (e.g., substrates, process kit rings, carriers, verification wafers, etc.) between the load lock, the processing chamber, and the transfer chamber. In some embodiments, the manufacturing equipment 124 includes components of a substrate processing system. In some embodiments, the manufacturing equipment 124 is used to produce one or more products (e.g., substrates, semiconductors, wafers, etc.). In some embodiments, the manufacturing equipment 124 is used to produce one or more components to be used in a substrate processing system. In some embodiments, the manufacturing equipment 124 is used to produce and / or include a joined metal plate structure (e.g., a showerhead used in a processing chamber of a substrate processing system).

[0039] The sensor 126 may provide sensor data 142 associated with the manufacturing equipment 124 (e.g., associated with the production of a corresponding product (e.g., a wafer) by the manufacturing equipment 124). For example, the sensor data 142 may be used for equipment health and / or product health (e.g., product quality). The manufacturing equipment 124 may produce a product according to a recipe or by performing an operation within a time period. In some embodiments, the sensor data 142 may include values ​​of one or more of temperature (e.g., heater temperature), spacing (SP), pressure, high frequency radio frequency (HFRF), voltage of an electrostatic chuck (ESC), current, flow (e.g., flow of one or more gases), power, voltage, etc. The sensor data 142 may include historical sensor data 144 and current sensor data 146. The historical sensor data 144 may be related to historical processing (e.g., manufacturing or processing operations associated with previously produced products (e.g., substrates, semiconductor wafers, or the like)). The historical sensor data 144 may serve as training data for training one or more models (e.g., model 190). The model 190 may be a machine learning model, a physics-based model, a statistical model, etc. The current sensor data 146 may be associated with non-historical operations (eg, a substrate currently being processed, a substrate recently processed, a target substrate of interest, or the like).

[0040] The manufacturing equipment 124 may be configured according to the manufacturing parameters 150. The manufacturing parameters 150 may be associated with or indicate parameters (e.g., hardware parameters (e.g., settings or components (e.g., size, type, etc.) of the manufacturing equipment 124) and / or processing parameters of the manufacturing equipment. The manufacturing parameters 150 may include historical manufacturing data (historical parameters 152) and / or current manufacturing data (current parameters 154). The manufacturing parameters 150 may indicate input settings of a manufacturing device (e.g., heater power, gas flow, etc.). When the manufacturing equipment 124 performs a manufacturing process, sensor data 142 and / or manufacturing parameters 150 may be provided (e.g., equipment readings may be taken while processing a product / substrate). The sensor data 142 may be different for each product (e.g., each wafer). The manufacturing parameters 150 may be the same or substantially the same (e.g., excluding metadata or the like) for a product line (e.g., product design, process recipe, etc.). The historical parameters 152 may be related to historical processing (e.g., manufacturing or processing runs associated with previously produced products (e.g., substrates, semiconductor wafers, or the like). The historical parameters 152 may serve as training data for training one or more models (e.g., model 190). In an embodiment, the model 190 may be a machine learning model, a physics-based model, or a statistical model. The current parameters 154 may be associated with non-historical operations (e.g., a substrate currently being processed, a substrate recently processed, a target substrate of interest, or the like).

[0041] The metrology data 160 may include measurements of properties of products produced by the manufacturing equipment 124. The historical sensor data 144, the historical parameters 152, and the metrology data 160 may be associated with the produced substrates. The metrology data 160 may include data indicating the association between historical and / or metrology data sets (e.g., data sets corresponding to the same produced substrates). The metrology data 160 may include measured and / or predicted metrology (e.g., virtual metrology) associated with any substrate property of interest. The metrology data 160 may include data corresponding to product thickness, resistivity, sheet resistance (e.g., resistivity of the film in a direction parallel to the plane of the film), critical dimension (CD, e.g., width of a feature), line width, feature depth, sidewall height, or the like. The metrology data 160 may include multi-point metrology data (e.g., a feature (e.g., thickness) may be measured at multiple points of the substrate, such as at various locations distributed throughout the spatial range of the substrate).

[0042] In some embodiments, the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150 may be processed (e.g., by the client device 120 and / or by the prediction server 112). The processing of the sensor data 142 may include generating attributes (e.g., data features, vectors, feature vectors, etc.). In some embodiments, the attributes are patterns in the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150 (e.g., slope, width, height, peak, etc.) or combinations of values ​​from the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150 (e.g., power derived from voltage and current, etc.). The sensor data 142 may include the attributes, and the attributes may be used by the prediction component 114 to perform signal processing and / or to obtain prediction data 168, which may be used to perform corrective actions. The prediction data 168 may be any data associated with the prediction system 110 (e.g., predicted metrology data of a substrate, predicted properties of a substrate, predicted performance of a substrate or manufacturing equipment 124, or the like).

[0043] Each instance (e.g., set) of sensor data 142 may correspond to a product (e.g., a wafer), a set of manufacturing equipment, a type of substrate produced by the manufacturing equipment, a combination thereof, or the like. Each instance of metrology data 160 and manufacturing parameters 150 may likewise correspond to a product, a set of manufacturing equipment, a type of substrate produced by the manufacturing equipment, a combination thereof, or the like. The data store may further store information associating different sets of data types (e.g., information indicating that the sensor data set, the metrology data set, and / or the manufacturing data set are all associated with the same product, manufacturing equipment, type of substrate, etc.).

[0044] In some embodiments, the prediction system 110 may use machine learning to generate the prediction data 168 (e.g., the target output includes data indicating manufacturing defects provided in the prediction system 110, etc.). In some embodiments, the prediction system 110 may use physics-based modeling to generate the prediction data 168. In some embodiments, the prediction system 110 may use statistical modeling to generate the prediction data 168. Two or more of these techniques may also be combined. Figures 2 to 3 and 4A to 4D The operation of the prediction system 110 is discussed in more detail.

[0045] The client devices 120, the manufacturing equipment 124, the sensors 126, the metrology equipment 128, the prediction server 112, the data store 140, the server machine 170, and the server machine 180 may be coupled to each other via the network 130 to generate prediction data 168. The prediction data 168 may be used to perform corrective actions. The prediction data 168 may be used to display model learning and / or a relationship between inputs (e.g., manufacturing parameters) and outputs (e.g., substrate properties, processing chamber performance, etc.) of one or more models. The prediction data 168 may be used to present one or more indications of model learning and / or a relationship between inputs and outputs of one or more models.

[0046] In some embodiments, the network 130 is a public network to provide access to the prediction server 112, the data store 140, and / or other publicly available computing devices to the client device 120. In some embodiments, the network 130 is a private network to provide access to the manufacturing equipment 124, the sensor 126, the metrology equipment 128, the data store 140, and / or other privately available computing devices to the client device 120. The network 130 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., long term evolution (LTE) networks), personal area networks, routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof.

[0047] The client device 120 may include one or more computing devices (e.g., a personal computer (PC), a laptop computer, a mobile phone, a smart phone, a tablet computer, a netbook computer, a network-connected television ("smart TV"), a network-connected media player (e.g., a Blu-ray player), a set-top box, an over-the-top (OTT) streaming device, an operation box, etc.). The client device 120 may include a corrective action component 122. The corrective action component 122 may receive user input of an indication associated with the manufacturing equipment 124 (e.g., via a graphical user interface (GUI) that is displayed via the client device 120). In some embodiments, the corrective action component 122 transmits the indication to the prediction system 110, receives output (e.g., prediction data 168) from the prediction system 110, determines a corrective action based on the output, and causes the corrective action to be implemented. In some embodiments, the presentation component 115 of the client device 120 may perform one or more actions associated with providing information indicating model learning and / or a visualization of input and output dependencies to a user. Presentation component 115 may generate one or more graphs, may perform calculations, or cause calculations to be performed by another device (eg, prediction server 112), may receive user instructions via a GUI associated with presenting information to a user, or the like.

[0048] In some embodiments, the prediction system 110 may further include a prediction component 114. The prediction component 114 may employ data retrieved from the model 190 to generate prediction data 168. In some embodiments, the prediction component 114 provides the prediction data 168 to the client device 120, and the client device causes a corrective action (e.g., including displaying the prediction data 168 to a user) via a corrective action component 122 in view of the prediction data 168. In some embodiments, the corrective action component 122 obtains an indication of data to be included in the corrective action or presentation component, retrieves the data (e.g., from the data store 140, by supplying one or more inputs from the model 190 and receiving one or more outputs from the model 190, by providing instructions to the prediction component 114 or the prediction system 110 and receiving the outputs, etc.), and displays the data to the user. In some embodiments, the corrective action component 122 may store the data (e.g., storing one or more graphs, one or more set parameters, etc.), for example, via the data store 140, as analysis data 169. The analysis data 169 may include the analysis data obtained by any of the methods described herein (e.g., in combination with Figure 3 or 4A to 4D any data produced as output by the method described.

[0049] In some embodiments, the prediction server 112 may store the output of the trained model 190 (e.g., the prediction data 168) in the data store 140, and the client device 120 may retrieve the output from the data store 140. In some embodiments, the corrective action component 122 receives an indication of a corrective action from the prediction system 110, and causes the corrective action to be implemented (e.g., causing data to be displayed to the user). Each client device 120 may include an operating system that allows a user to generate, view, or edit one or more of the data (e.g., indications associated with the manufacturing equipment 124, corrective actions associated with the manufacturing equipment 124, etc.).

[0050] In some embodiments, the metrology data 160 corresponds to historical property data of the product (e.g., produced using manufacturing parameters associated with historical sensor data and historical manufacturing parameters), and the prediction data 168 is associated with the predicted property data (e.g., the product to be produced or has been produced under the conditions recorded by the current sensor data 144 and / or the current parameters 154). In some embodiments, the prediction data 168 is the predicted metrology data (e.g., virtual metrology data) of the product to be produced or has been produced according to the conditions recorded as the current sensor data and / or the current manufacturing parameters. In some embodiments, the prediction data 168 is an indication of an abnormality (e.g., an abnormal product, an abnormal component, an abnormal manufacturing equipment, abnormal energy use, etc.) and / or one or more causes of the abnormality, or includes an indication of an abnormality and / or one or more causes of the abnormality. In some embodiments, the prediction data 168 includes an indication of changes or drifts over time in some components of the manufacturing equipment 124, the sensor 126, the metrology equipment 128, and the like. In some embodiments, the prediction data 168 includes an indication of the end of the life of a component of the manufacturing equipment 124, the sensor 126, the metrology equipment 128, or the like.

[0051] In some embodiments, one or more outputs of model 190 may be provided to presentation component 115. Presentation component 115 may generate an indication of model learning, model input / output mappings and / or associations, or the like. The outputs of presentation component 115 (e.g., graphs, GUI components, etc.) may be similar to those associated with FIG. 5A to FIG. 5CThose discussed. In some embodiments, the presentation component 115 can generate an indication of the range of the output space of the model 190. For example, the simulation data 161 can be provided to the model 190 as multiple data sets. The simulation data 161 can span the input space of the model 190 or a portion of the input space. For example, the simulation data 161 can include data associated with multiple inputs of the model 190. The simulation data 161 can include a data set of values ​​of the inputs of the model 190 ranging from a minimum input value (e.g., a minimum value of the input of the training model, a minimum value of the input of the model to produce an output that meets a threshold confidence level, or the like) to a maximum input value (e.g., according to a constraint similar to the minimum input value). In some embodiments, the simulation data 161 can be generated to span the input space of the model 190. For example, the simulation data 161 can be generated in a grid on the dimension of the input space (e.g., a first input can take one of several values, a second input can take one of a second list of values, and a third input can take one of a third list of values, with all combinations to generate a multi-dimensional grid spanning the input space of the model 190). Multiple sets of inputs (e.g., sets that roughly span the input space of the model) can be provided to the model. The outputs of the model can be stored.

[0052] In some embodiments, the presentation component 115 may generate an indication of the range of the output space of the model 190. A set of inputs may be provided to the model 190, the set of inputs roughly spanning the input space of the model 190 (e.g., the space of input values ​​of the training model, the span of input values ​​of the model generating outputs that satisfy a threshold confidence condition, or the like). Multiple sets of randomly sampled inputs may be provided to the model 190. For example, a distribution of training inputs may be generated (e.g., a distribution of values ​​provided as a first input during training, a second distribution of values ​​provided as a second input during training, etc.). A random distribution of simulated training inputs that conforms to the statistical distribution of the training data of the model may be generated. In some embodiments, random sampling of input data within the input space, within a portion of the input space, or the like may be generated. In some embodiments, random sampling of input data may be distributed according to a metric other than the distribution of the inputs of the training data (e.g., a Gaussian, Lorentzian, or other distribution shape may be generated based on the minimum and maximum input values ​​for one or more inputs). Each of the multiple sets of inputs (e.g., roughly spanning the input space of the model together) may be provided to the model. The output of the model may be stored.

[0053] In some embodiments, a model (e.g., model 190) may generate a single output based on a set of inputs. In some embodiments, a model (e.g., model 190) may generate multiple outputs based on a set of inputs. In some embodiments, a model may generate several outputs for a single feature based on a set of inputs (e.g., thickness measurements from multiple locations of a substrate, etc.). In some embodiments, a model may generate several outputs for multiple features based on a set of inputs (e.g., thickness and resistivity measurements from multiple locations of a substrate). Here, a model is generally described as a model that receives a set of inputs and generates multiple values ​​for a single feature as an output (e.g., predicted measurements of a feature at multiple locations of a substrate). Models that operate in different ways (e.g., models that generate a single output value as an output, models that generate output values ​​for multiple features and / or properties of those features, etc.) are within the scope of the present disclosure. In some embodiments, an integrated model including multiple models may be utilized (e.g., a set of inputs may be provided to the model and two sets of outputs may be generated (e.g., each set of outputs is for a different feature)). Here, no distinction is made between an integrated model that produces an indication of multiple features and two separate models that produce indications of multiple features (e.g., providing a set of inputs to a model or receiving outputs from a model may be interpreted in some applications as operations associated with sub-models of the integrated model).

[0054] In some embodiments, the output of one or more models can be presented to the user. Each set of input conditions (e.g., input conditions that generally span the input space, input conditions that span a portion of the input space, etc.) can correspond to one or more outputs. An indication of the output of each set of input conditions can be presented on a graph. The graph can generally represent the span of the output space of model 190.

[0055] In some embodiments, each set of inputs (e.g., inputs that roughly span the input space of one or more models) can correspond to a plotted point on the scatter plot. In some embodiments, the axes of the graph (e.g., the horizontal ("x") axis and the vertical ("y") axis of a two-dimensional scatter plot, two horizontal axes and one vertical axis of a three-dimensional scatter plot, etc.) can correspond to the output of one or more models. In some embodiments, the axes of the graph can correspond to the properties of features associated with the output of one or more models. Features can include any interesting features of the substrate. Features can include measurable metrological features of the substrate. Features can include thickness, resistivity, sheet resistance, refractive index, extinction coefficient, critical dimension, segment width, depth, sidewall height, etc. The property can include a statistical measure of the feature (e.g., multiple measurements of the feature (e.g., simulated measurements at various locations on the substrate, simulated metrological measurements, etc.) can be provided as output from the model 190), and the property of the feature can include the mean, median, standard deviation, uniformity, interquartile range, kurtosis, skewness, or any other statistical measure that describes the distribution of feature values. In some embodiments, a feature or property of a feature can include a subset of the values ​​output by the model. For example, the value of a feature near the outer edge of the wafer, the value of a feature within an angular range or radial range, the value of a feature at the center of the wafer, or the like may be of interest. Features or properties of features may include spatial combinations of feature measurements, statistical combinations of feature measurements (e.g., an average of a portion of the measured values ​​(e.g., bottom quartile, median half, etc.)), or the like.

[0056] In some embodiments, the first axis of the output graph can correspond to the first property of the first feature (as an illustrative example: average thickness). The second axis of the graph can correspond to the second property of the first feature (continuing the illustrative example: standard deviation of simulated thickness measurements). Each set of input conditions can produce a point on the scatter plot having a specific value corresponding to the first axis and a specific value corresponding to the second axis (continuing the illustrative example: each set of input conditions can produce a simulated substrate with an average thickness and a standard deviation of the thickness measurement, and the points can be placed at appropriate locations on the graph for each set of inputs, intuitively mapping the range of the model's input space to the two-dimensional output space represented by the graph). In some embodiments, a three-dimensional scatter plot can be generated that includes a third axis corresponding to a third property of the first feature.

[0057] In some embodiments, the first axis of the output graph may correspond to the first property of the first feature (as an illustrative example: average thickness). The second axis of the graph may correspond to the second property of the second feature (continuing as an illustrative example: the standard deviation of the simulated resistivity measurement). The second property may be the same property as the first property or a different property. In some embodiments, a feature associated with the second axis of the graph may be generated by a second model. Each set of input conditions may generate a point on a scatter plot having a specific value corresponding to the first axis and a specific value corresponding to the second axis (continuing the illustrative example: each set of input conditions may generate an average thickness value and a resistivity standard deviation value, and data points may be plotted at positions on the graph corresponding to the average thickness and the resistivity standard deviation). In some embodiments, a three-dimensional scatter plot including a third axis may be generated. The third axis may correspond to a third property of the first feature, a third property of the second feature, a third property of the third feature, and the like.

[0058] In some embodiments, outputs associated with the set of inputs that span the input space of one or more models (e.g., substantially span the input space, span a portion of the input space, etc.) can be displayed on a graph as scattered points (e.g., with a background "cloud" indicating the extent of the output space of one or more models in the plotted dimensions (e.g., associated with the plotted features and properties of the features)).

[0059] In some embodiments, a set of inputs of interest (e.g., a set of baseline inputs, a set of center inputs, etc.) may be provided to one or more models. The one or more models may generate outputs associated with the set of baseline inputs. The outputs may be displayed on a graph, for example, along with a cloud of scatter plots representing the span of the output space of the one or more models.

[0060] In some embodiments, a series of groups of inputs of interest may be provided to one or more models. For example, a series of groups of inputs may include a baseline input group. The series of groups of inputs may further include multiple groups of inputs, one of which is changed compared to the baseline input group. For example, multiple groups of inputs may be provided to one or more models, wherein all inputs except one input are maintained at their baseline values, and the value associated with one input is changed between multiple groups of inputs. The value associated with one input may be changed from a minimum value to a maximum value. The minimum and maximum values ​​may be associated with the range of the input space of the model, the range of the input space orthogonal to the baseline value measurement, a confidence interval (e.g., an input space where the output satisfies a threshold confidence metric condition), a subset of one of these intervals, or the like. The value associated with an input may be changed systematically or randomly. The value associated with an input may be changed using uniform increments, increments according to some functions (e.g., logarithmic intervals), according to some distributions (e.g., Gaussian distribution, distribution defined by a training data set associated with one or more models), or the like. The group of outputs may depict learning of the effect of changing an input value on an output metric associated with a graph (e.g., associated with two or three axes of the graph). The visual depiction of the output associated with the series of inputs may include a visual distinction between the outputs associated with a minimum value of an input and a maximum value of an input. For example, the output point associated with the minimum input value may be colored green, while the output point associated with the maximum input value may be colored red, and the output point associated with the minimum input value may be a different shape (e.g., a unique shape in the graph) than the output point associated with the maximum input value, each of the output points may be presented in the shape of an arrow pointing to the next (e.g., the next higher value of the input) output point, or the like. The output associated with the series of input values ​​may generate a curve on the graph, showing the learning of the model, showing the effect of changing a variable on the output of the model, etc. In some embodiments, the points of the curve may be distinguished from each other (e.g., the presented data point associated with the lowest value of the changed input value may be visually distinguished from the presented data point associated with the highest value of the changed input value).

[0061] In some embodiments, multiple series of inputs can be provided to one or more models. For example, the input of each series of groups can limit all input values ​​except one input value to baseline values, and one input value (e.g., one input value of each series) can be changed as described above. For example, each series can be drawn in a visually different manner (e.g., using different colors, different shapes, different patterns, or the like of the visual representation of the output data). In some embodiments, the input of each series of groups can generate a curve of output data points. The input of each series of groups can be drawn and generate a curve indicating the model learning associated with the change associated with one input of the series. Each curve associated with the input of a series of groups can intersect and / or intersect at the exit point associated with the baseline input group.

[0062] In some embodiments, presentation component 115 can update the graph based on user selections (e.g., settings, parameters, baseline input conditions, or the like). A user can navigate the input and / or output space via a GUI (e.g., associated with client device 120). Operation of the GUI, operation of presentation component 115, example graphs, and the like are described in detail below. FIG. 5A to FIG. 5C Let’s discuss it in more detail together.

[0063] Performing a manufacturing process that results in a defective product can be costly due to time, energy, product, parts, manufacturing equipment 124, costs of identifying defects and discarding defective products, etc. By inputting sensor data 142 (e.g., measurements of conditions in a process chamber) and / or manufacturing parameters 150 (e.g., process recipe parameters) into a model 190 (e.g., a machine learning model), receiving output of prediction data 168, and performing corrective actions based on the prediction data 168, a technical advantage of the system 100 is that the costs of generating, identifying, and discarding defective products can be avoided.

[0064] In some embodiments, the learnings of the model 190 (e.g., a machine learning model) (e.g., one or more aspects of the model's input / output mapping) can be subjected to additional testing, validation, or the like. In some embodiments, one or more aspects of the association between the model's inputs and outputs can be displayed for review, inspection, validation, utilization, for updating processing parameters, for design of new processing recipes, or the like.

[0065] In some embodiments, the user may choose to implement the model (e.g., the user may determine what action to take based on the output of the model). In some embodiments, the user may not be involved in the development of the model (e.g., a customer of the seller of the model), may not be an expert in modeling techniques (e.g., may be an engineer or technician, but not an expert in model building, model training, or the like), may not have experience in utilizing model outputs (e.g., may have developed an intuitive understanding, may rely on internal knowledge or past experience, etc.), or the like. The user may present the output of the model in a conventional format and may not be able to conceptualize the results as an indication of the connections, mappings, or learning (e.g., input / output mappings) learned by the model (e.g., during model training). The user may be reluctant to utilize a model that the user does not understand, may not trust a model that does not produce results that are consistent with the user's understanding, or the like. A model with learning associated with a model that is presented to the user in an understandable manner may have the advantage of being trusted by the user, verified by the user, etc. A model trusted by the user may be more fully utilized and may allow for more comprehensive improvements in the operation of a process or a processing facility, etc.

[0066] Performing a manufacturing process that results in a component failure of the manufacturing equipment 124 may be costly due to downtime, product damage, equipment damage, rapid ordering of replacement parts, etc. By inputting sensor data 142 (e.g., measurements of conditions in a process chamber) and / or manufacturing parameters 150 (e.g., recipe parameters of a process recipe) into a model 190 (e.g., a machine learning model, a physics-based model, a statistical model, etc.) and receiving output of prediction data 168, and performing corrective actions (e.g., predicted operational maintenance (e.g., replacement, treatment, cleaning, etc. of a component) based on the prediction data 168, a technical advantage of the system 100 is that the cost of one or more of unexpected component failure, unplanned downtime, productivity loss, unexpected equipment failure, product scrap, or the like can be avoided. Monitoring the performance of a component (e.g., manufacturing equipment 124, sensor 126, metrology equipment 128, and the like) over time can provide an indication of component degradation. Monitoring the performance of a component (e.g., a substrate support) over time may extend the operational life of the component (e.g., if after a standard replacement interval has passed, measurements indicate that the component is still performing well (e.g., performance is above a threshold) for a period of time (e.g., until the next scheduled maintenance event)).

[0067] In some embodiments, there may be costs associated with executing the actions recommended by the model. For example, the model may suggest that a component fails and / or needs maintenance before the component is scheduled to be replaced / maintained (e.g., the model may generate an output indicating that the component needs to be replaced or maintained). Performing the recommended maintenance may be expensive in terms of downtime, the cost of replacing the component, a reduction in effective operating time (e.g., green time), etc. If the user is not familiar with the learning, association, mapping, or the like of the model, the user may choose to avoid the cost of performing the action. In some embodiments, avoiding the cost of performing corrective actions may incur greater costs (e.g., the component may fail, which may result in loss of product, expensive unplanned downtime, damage to other components of the system, etc.). In some embodiments, the machine learning model may recommend delayed maintenance (e.g., an output indicating delayed maintenance may be generated) (e.g., the component may perform beyond the planned expectations). If the user chooses to follow the instructions (e.g., perform corrective actions (e.g., update the maintenance plan)) and delay maintenance (e.g., by extending the activity processing time), the operating cost of the system may be lower. Since the user takes action in response to the predicted component life of the component of the manufacturing system, a model that generates user trust can have benefits in terms of the operating cost of the manufacturing system.

[0068] Manufacturing parameters may be suboptimal for producing products, which may result in expensive consequences such as increased consumption of resources (e.g., energy, coolant, gas, etc.), increased amount of time to produce products, increased component failures, increased proportion of defective products produced, etc. By inputting sensor data 142 into a trained model 190 (e.g., a machine learning model, a physics-based model, etc.), receiving the output of prediction data 168, and performing (e.g., based on prediction data 168) a corrective action to update manufacturing parameters (e.g., set optimal manufacturing parameters), the system 100 may have the technical advantage of using optimal manufacturing parameters (e.g., hardware parameters, processing parameters, optimal design) to avoid the expensive consequences of suboptimal manufacturing parameters. In some embodiments, a user may choose to perform the recommended corrective action. The user's choice of whether to perform the corrective action may be based on the user's trust in the model, the user's understanding of the operation of the model, the user's understanding of the learning of the model, or the like. The methods and systems disclosed herein may enable users to deepen their understanding of the operation and the learning of the model. The methods and systems disclosed herein may facilitate users to perform recommended corrective actions associated with the manufacturing system (e.g., updating manufacturing parameters, processing recipes, or the like). This move can improve the efficiency of the manufacturing system, reduce operating costs, etc.

[0069] Corrective actions may be associated with one or more of computational process control (CPC), statistical process control (SPC) (e.g., SPC on electronic components for determining a controlled process, SPC for predicting useful service life of a component, SPC compared to a three standard deviation plot, etc.), advanced process control (APC), model-based process control, predictive operational maintenance, design optimization, updating of manufacturing parameters, updating process recipes, feedback control, machine learning modifications, or the like.

[0070] In some embodiments, the corrective action includes providing an alert (e.g., providing an alert to stop or not perform a manufacturing process if the prediction data 168 indicates a predicted anomaly (e.g., an anomaly of a product, component, or manufacturing equipment 124). In some embodiments, the corrective action includes providing feedback control (e.g., modifying a manufacturing parameter in response to the prediction data 168 indicating an anomaly). In some embodiments, the corrective action includes updating a process recipe (e.g., modifying one or more manufacturing parameters based on the prediction data 168). In some embodiments, execution of the corrective action includes causing an update to one or more manufacturing parameters. In some embodiments, execution of the corrective action includes causing an update to one or more calibration tables and / or equipment constants (e.g., a set point provided to a component may be adjusted by a value across multiple process recipes (e.g., the voltage applied to the heater may be increased by 3% for all processes using a heater)).

[0071] Manufacturing parameters may include hardware parameters (e.g., a history of replacement parts, an indication that the manufacturing system is using certain parts, an indication of an update to the process (e.g., replacing a processing chip or updating firmware, etc.)) and / or process parameters (e.g., temperature, pressure, flow, rate, current, voltage, gas flow, lifting speed, etc.). In some embodiments, the corrective action includes causing preventive operational maintenance (e.g., replacing, processing, cleaning, etc. of a part of the manufacturing equipment 124). In some embodiments, the corrective action includes causing design optimization (e.g., updating manufacturing parameters, manufacturing processes, manufacturing equipment 124, etc. for an optimized product). In some embodiments, the corrective action includes updating a recipe (e.g., the manufacturing equipment 124 is in an idle mode, a sleep mode, a preheating mode, etc.). In some embodiments, the corrective action (e.g., recommended by the model 190, performed by the client device 120, performed by the presentation component 115, or the like) may include providing an alert to a user (e.g., preparing data for presentation to the user).

[0072] Each of the prediction server 112, the server machine 170, and the server machine 180 includes one or more computing devices (e.g., a rack server, a router computer, a server computer, a personal computer, a mainframe computer, a notebook computer, a tablet computer, a desktop computer, a graphics processing unit (GPU), an accelerator application specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc.).

[0073] Prediction server 112 may include prediction component 114. Prediction component 114 may be used to generate prediction data 168. In some embodiments, prediction component 114 may receive current sensor data 146 and / or current parameters 154 (e.g., received from client device 120, retrieved from data store 140) and generate output for performing corrective actions associated with manufacturing equipment 124 based on the current data.

[0074] The manufacturing equipment 124 can be associated with one or more machine learning models, physics-based models, statistical models, and the like (e.g., model 190). The machine learning models and other models associated with the manufacturing equipment 124 can perform a number of tasks (including process control, classification, performance prediction, and the like). The model 190 can be trained using data associated with the manufacturing equipment 124 or products processed by the manufacturing equipment 124 (e.g., sensor data 142 (e.g., collected by sensors 126), manufacturing parameters 150 (e.g., associated with process control of the manufacturing equipment 124), metrology data 160 (e.g., generated by metrology equipment 128), and the like).

[0075] One type of machine learning model that can be used to perform some or all of the above tasks is an artificial neural network (e.g., a deep neural network). An artificial neural network typically includes a feature representation component with a classifier or regression layer that maps features to a desired output space. For example, a convolutional neural network (CNN) has multiple layers of convolutional filters. Pooling is performed at the lower layers and nonlinear problems can be solved. On top of it, a multi-layer perceptron is usually attached to map the top features extracted by the convolutional layers to decisions (e.g., classification outputs).

[0076] Recurrent Neural Networks (RNNs) are another type of machine learning model. Recurrent Neural Network models are designed to interpret a series of inputs that are inherently related to each other (e.g., time trajectory data, sequential data, etc.). The output of the RNN's perceptron is fed back into the perceptron as input to produce the next output.

[0077] Deep learning describes a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in an image recognition application, the raw input may be a matrix of pixels; the first representation layer may abstract the pixels and encode the edges; the second layer may combine and encode the arrangement of the edges; the third layer may encode a higher-level shape (e.g., identifying the structure of the substrate (e.g., gates, shields, etc.)); and the fourth layer may produce a classified output. It should be noted that the deep learning process can learn on its own which features are best placed at which level. The "depth" in "deep learning" refers to the number of layers of data transformation. More precisely, deep learning systems have a considerable credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAP describes the potential causal relationship between inputs and outputs. For feedforward neural networks, the depth of CAP can be the depth of the network, and can be the number of hidden layers plus one. For recurrent neural networks where signals may propagate through a layer multiple times, the CAP depth may be infinite.

[0078] Training of neural networks can be accomplished using supervised learning, which involves feeding a training dataset consisting of labeled inputs through the network, observing its output, defining an error (by measuring the difference between the output and the labeled value), and using techniques such as deep gradient descent and backpropagation to adjust the weights of all layers and nodes of the network to minimize the error. In many applications, repeating this process over many labeled inputs in the training dataset results in a network that can produce the correct output when the input is different from the input present in the training dataset.

[0079] In some embodiments, the prediction component 114 can use one or more models 190 to determine outputs for performing corrective actions based on current data. Model 190 can be a single model, an overall model, or a collection of models for processing data. Model 190 can include one or more physics-based digital twin models, supervised machine learning models, unsupervised machine learning models, semi-supervised machine learning models, statistical models, etc.

[0080] In some embodiments, client device 120 may provide current sensor data 146 (e.g., sensor data of interest) to prediction system 110. In some embodiments, presentation component 115 may provide simulation data 161 to prediction system 110. Simulation data 161 may include data related to sensor data 142, manufacturing parameters 150, etc. Simulation data 161 may be or include user input (e.g., input via a GUI of client device 120). Simulation data may be provided as input (e.g., as input of interest) to model 190 to generate (e.g., via presentation component 115) a graphical representation of model learning, model knowledge, model mapping / association between inputs and outputs, etc.

[0081] In some embodiments, data indicating properties of substrates to be produced using the manufacturing system (e.g., predicted data) is provided to a model (e.g., a trained machine learning model (e.g., model 190)). The model can be trained to output data indicating corrective actions to produce substrates with different properties. In some embodiments, data indicating predicted properties of substrates produced using the manufacturing equipment 124 and metrology data of substrates produced using the substrate support are provided as inputs to the model (e.g., model 190). The model can predict root causes for differences between the predicted and measured data (e.g., manufacturing defects, component aging or drift, etc.).

[0082] The historical sensor data 142, the historical parameters 152, and / or the metrology data 160 can be used in conjunction with the current sensor data 146 and the current parameters 154 to detect drift, change, aging, etc. of components of the manufacturing equipment 124. The sensor data 142 monitored over time may include information indicating changes in one or more components of the manufacturing equipment 124 (e.g., due to aging, drift, component failure, deposition or removal of material, etc.). The prediction component 114 may use a combination and comparison of these data types to generate prediction data 168. In some embodiments, the prediction data 168 includes data predicting the life of components of the manufacturing equipment 124, the sensor 126, etc. The presentation component 115 used in conjunction with the prediction system 110, the data store 140, etc. may provide a representation of the learning of the model 190.

[0083] In some embodiments, the prediction component 114 receives data (e.g., sensor data 142, manufacturing parameters 150, metrology data 160, etc.) and may perform preprocessing (e.g., extracting patterns in the data or combining the data into new composite data). The prediction component 114 may then provide the data to the model 190 as input. The model 190 may include a physics-based digital twin model that accepts sensor data 142, manufacturing parameters 150, simulation data 161, or the like as input. A trained machine learning model, a statistical model, or the like configured to further process data associated with properties of a substrate, performance of manufacturing equipment 124, or the like may be included. The prediction component 114 may receive prediction data indicating substrate support performance, predicted substrate properties, manufacturing defects, component drift, or the like from the model 190. The prediction component 114 may then cause a corrective action to occur (e.g., recommending a corrective action to a user). The corrective action may include sending an alert to the client device 120. The corrective action may also include updating the manufacturing parameters of the manufacturing equipment 124. Corrective action may also include generating predictive data 168 indicative of chamber or instrument drift, aging, or failure.

[0084] In some embodiments, the model can be trained and utilized to generate recommended corrective actions, and / or cause corrective actions (e.g., the model can be operably coupled to one or more components of the manufacturing equipment 124, and the output of the model can automatically update future processing parameters or the like). In the case of recommended actions and the case of automatically performing corrective actions, it may be valuable to provide a record of model learning to the user (e.g., for verification of model stability, verification of model mapping, etc.). For example, an alert describing model learning can be presented to the user (e.g., via presentation component 115). In some embodiments, the alert can include a graphical representation of the model output over a range of smoothly varying inputs. The user can predict that smoothly varying inputs will produce smoothly varying outputs. The user can verify that the model has learned input / output associations that the user trusts, and the user can continue to utilize the model, can perform recommended corrective actions, etc.

[0085] The data store 140 may be a memory (e.g., random access memory), a drive (e.g., a hard drive, a flash drive), a database system, or another type of component or device capable of storing data. The data store 140 may include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers). The data store 140 may store sensor data 142, manufacturing parameters 150, analysis data 169, simulation data 161, metrology data 160, and prediction data 168. The sensor data may include a temporal trajectory of sensor data over the duration of a manufacturing process, an association of data with a physical sensor, pre-processed data (e.g., average and composite data), and data indicating sensor performance over time (i.e., many manufacturing processes). The manufacturing parameters 150 and metrology data 160 may include similar features. The prediction data 168 may include data output by the prediction system 110. The analysis data 169 may include data (e.g., graphs, visualizations, etc.) output by the presentation component 115 (e.g., for further analysis). The simulation data 161 may include features similar to one or more of the sensor data 142 and / or the manufacturing parameters 150. The simulation data 161 may be provided to the model 190 (e.g., to generate an output for presenting the component 115). The simulation data 161 may include a data range (e.g., a range of values ​​associated with an input parameter of the model 190). The historical sensor data 144 and / or the historical parameters 152 may be used to train the model 190. The metrology data 160 may be used to train the model 190 and may include, among other things, predicted metrology data output by the model 190. The metrology data 160 may be metrology data of the produced substrates, as well as sensor data, manufacturing data, and model data corresponding to these products. The metrology data 160 may be used to design a process for producing additional substrates. The prediction data 168 may include predictions of metrology data generated by the operation of the substrate support, predictions of component drift, aging, or failure, predictions of component life, and the like. The prediction data 168 may also include data indicating component aging and failure of the system 100 over time.

[0086] In some embodiments, prediction system 110 further includes server machine 170 and server machine 180. Server machine 170 includes a dataset generator 172 capable of generating a dataset (e.g., a set of data inputs and a set of target outputs) for training, validating, and / or testing model 190. Some operations of dataset generator 172 are described below with respect to Figure 2 and Figure 4ADetailed description. In some embodiments, the data set generator 172 may divide the historical data (e.g., historical sensor data 144, historical metering data of metering data 160, historical parameters 152, etc.) into a training set (e.g., 60 percent of the data), a validation set (e.g., 20 percent of the data), and a test set (e.g., 20 percent of the data). In some embodiments, the prediction system 110 (e.g., via the prediction component 114) generates multiple sets of attributes (e.g., feature vectors, vectors, etc.). For example, a first set of attributes may correspond to a first set of types of sensor data (e.g., from a first set of sensors, a first combination of values ​​from the first set of sensors, a first pattern of values ​​from the first set of sensors), the first set of types of sensor data corresponding to each of the data sets (e.g., training set, validation set, and test set), and a second set of attributes may correspond to a second set of types of sensor data (e.g., from a second set of sensors different from the first set of sensors, a second combination of values ​​different from the first combination, a second pattern different from the first pattern), the second set of types of sensor data corresponding to each of the data sets.

[0087] In some embodiments, the server machine 180 includes a training engine 182, a validation engine 184, a selection engine 185, and / or a test engine 186. An engine (e.g., a training engine 182, a validation engine 184, a selection engine 185, and a test engine 186) may refer to hardware (e.g., circuits, dedicated logic, programmable logic, microcode, a processing device, etc.), software (e.g., instructions running on a processing device, a general-purpose computer system, or a dedicated machine), firmware, microcode, or a combination thereof. The training engine 182 is capable of training a model 190 using one or more sets of attributes associated with a training set from the data set generator 172. The training engine 182 may generate a plurality of trained models 190, wherein each trained model 190 corresponds to a different set of attributes of the training set (e.g., sensor data from a different sensor). For example, a first trained machine learning model may be trained using all attributes (e.g., X1-X5), a second trained machine learning model may be trained using a first subset of attributes (e.g., X1, X2, X4), and a third trained machine learning model may be trained using a second subset of attributes (e.g., X1, X3, X4, and X5) that partially overlaps with the first subset of attributes. Dataset generator 172 may receive the output of a trained model (e.g., 190), collect the data into training, validation, and test data sets, and use the data sets to train a second model. Some or all of the operations of server machine 180 may be used to train various types of models (including physics-based models, supervised machine learning models, unsupervised machine learning models, etc.).

[0088] The validation engine 184 can validate the trained models 190 using the corresponding feature set of the validation set from the dataset generator 172. For example, a first trained model 190 trained using the first set of attributes of the training set can be validated using the first set of attributes of the validation set. The validation engine 184 can determine the accuracy of each of the trained models 190 based on the corresponding feature set of the validation set. The validation engine 184 can discard trained models 190 with an accuracy that fails to meet a threshold accuracy. In some embodiments, the selection engine 185 can select one or more trained models 190 with an accuracy that meets a threshold accuracy. In some embodiments, the selection engine 185 can select a trained model 190 with the highest accuracy of the trained models 190.

[0089] The testing engine 186 can test the trained model 190 using a corresponding set of attributes of the test set from the data set generator 172. For example, a first trained model 190 trained using a first set of attributes of the training set can be tested using a first set of attributes of the test set. The testing engine 186 can determine the trained model 190 with the highest accuracy among the trained models based on the test set.

[0090] Model 190 may refer to a machine learning model, which may refer to a model built by training engine 182 using a training set that includes data inputs and corresponding target outputs (correct answers for respective training inputs). Model 190 may additionally or alternatively refer to a statistical model or a physics-based model. Patterns in the data set may be found to map the data inputs to target outputs (correct answers), and a mapping that captures these patterns may be provided to model 190. In some embodiments, model 190 may predict properties of a substrate. In some embodiments, model 190 may predict failure modes of manufacturing chamber components.

[0091] Model 190 may refer to a trained physics-based model. A trained physics-based model may be configured to find solutions to one or more equations describing physical quantities of a processing chamber (e.g., mass flow (e.g., gas flow), heat transfer equations, fluid dynamics equations, or the like). In some embodiments, the assumptions used to generate the physics-based model may not be completely accurate (e.g., due to inaccurate measurements, manufacturing or material defects, mismatches in manufacturing tolerances of components, component aging, drift, or behavior that differs from predictions, or the like). Training the physics-based model may correct one or more of these assumptions that introduce error into the physics-based model (e.g., by allowing one or more parameters of the model to be transformed to better fit the training data).

[0092] The prediction component 114 can provide input data to the trained model 190, and can run the trained model 190 on the input to obtain one or more outputs. The prediction component 114 can determine (e.g., extract) prediction data 168 from the output of the model 190, and can determine (e.g., extract) confidence data from the output to indicate the confidence level that the prediction data 168 is an accurate predictor of the process associated with the input data of the product produced or to be produced or an accurate predictor of the component of the manufacturing equipment 124. The prediction component 114 can determine the prediction data 168 (including predictions for the final substrate properties and the prediction of the effective life of the component of the manufacturing equipment 124, the sensor 126, or the metrology equipment 128) based on the output of the model 190. The prediction component 114 or the corrective action component 122 can use the confidence data to determine whether to cause a corrective action associated with the manufacturing equipment 124 based on the prediction data 168. Presentation component 115 can utilize confidence data (e.g., to visually present some areas of the model's output space as uncertain (e.g., by displaying data in different colors, shapes, shading, sizes, transparency levels, or the like to indicate pattern confidence)).

[0093] The confidence data may include or indicate a confidence level. As an example, the prediction data 168 may indicate the properties of the final wafer given a set of manufacturing inputs (e.g., current parameters 154) (including the use of manufacturing equipment 124). The confidence data may indicate that the prediction data 168 is an accurate prediction of a product associated with at least a portion of the input data. In one example, the confidence level is a real number between 0 and 1 (including 0 and 1), where 0 indicates that there is no confidence that the prediction data 168 is an accurate prediction of a product processed according to the input data, and 1 indicates that the absolute confidence prediction data 168 accurately predicts the properties of the product processed according to the input data. In response to the confidence data indicating that the level of confidence is below a threshold level for a predetermined number of examples (e.g., a percentage of examples, a frequency of examples, a total number of examples, etc.), the prediction component 114 may retrain the model 190 (e.g., based on the current sensor data 146, the current manufacturing parameters 150, etc.).

[0094] For purposes of illustration and not limitation, aspects of the present disclosure describe using historical data to train one or more models 190 and inputting current data into one or more trained models 190 to determine predicted data 168. In other embodiments, heuristic models or rule-based models are used to determine predicted data (e.g., without using a trained machine learning model). The prediction component 114 can monitor historical data and metering data 160. The heuristic or rule-based models can monitor or use information relative to the predicted data. Figure 2 The data input 210 describes any information.

[0095] In some embodiments, the functionality of client device 120, prediction server 112, server machine 170, and server machine 180 may be provided by a fewer number of machines. For example, in some embodiments, server machines 170 and 180 may be integrated into a single machine, while in some other embodiments, server machine 170, server machine 180, and prediction server 112 may be integrated into a single machine. In some embodiments, client device 120 and prediction server 112 may be integrated into a single machine.

[0096] In general, functions described in one embodiment as being performed by client device 120, prediction server 112, server machine 170, and server machine 180 may also be performed on prediction server 112 in other embodiments, if appropriate. In addition, functions attributed to a particular component may be performed by different or multiple components operating together. For example, in some embodiments, prediction server 112 may determine corrective actions based on prediction data 168. In another example, client device 120 may determine prediction data 168 based on output from model 190 (e.g., a trained machine learning model or a physics-based digital twin model).

[0097] Furthermore, the functionality of a particular component may be performed by different or multiple components operating together. One or more of the prediction server 112, server machine 170, or server machine 180 may be accessed through a suitable application programming interface (API) as a service provided to other systems or devices.

[0098] In an embodiment, a "user" may be represented as a single person. However, other embodiments of the present disclosure encompass a "user" of an entity controlled by multiple users and / or automated sources. For example, a group of individual users united in an administrator group may be considered a "user."

[0099] Embodiments of the present disclosure may be applied to data quality assessment, feature enhancement, model assessment, virtual metrology (VM), predictive maintenance (PdM), extreme optimization, or the like. Embodiments of the present disclosure may be applied to any trained modeling system (e.g., model assessment, model validation, representation of model learning, or the like may be provided for any machine learning model, any machine learning model associated with manufacturing and / or processing a product, any machine learning model that predicts metrology of wafers to be processed, or the like).

[0100] Figure 2 is a method for establishing a model (e.g., Figure 1 An exemplary dataset generator 272 (eg, Figure 1 The data set generator 272 may be Figure 1 A portion of a server machine 170. In some embodiments, Figure 1 The system 100 includes multiple models. In this case, each model can have a separate data set generator, or the models can share a data set generator. For example, a separate model can be used for each feature of interest of the wafer associated with a set of input data. Figure 2 Depicting a dataset generator associated with a machine learning model, the dataset generator being configured to receive sensor data (e.g., Figure 1 The current sensor data 146) is taken as input and provides as output a predicted metrology of the substrate (e.g., Figure 1 168 of prediction data). The model (e.g., a machine learning model, a physics-based model, a statistical model, etc.) can be configured to perform one or more of many different tasks. For example, a model can receive sensor data and generate as output a feedback control signal for adjusting a processing condition. A model can receive processing parameters and predict the performance of a substrate (e.g., predicting the metrology of a substrate produced by a processing recipe). A model can receive an image (e.g., a block diagram of a target substrate design) and generate a related image (e.g., a real image of a simulated substrate). A model can receive an indication of a target product design and can generate as output a processing recipe that predicts the production of a product of the design. A model can receive measurements of one or more components of a manufacturing system and generate as output a predicted performance of the system. Any of these or many other specific use cases of a model (e.g., any machine learning model that maps a set of inputs to one or more outputs, a model associated with manufacturing, a model associated with substrate processing, a model associated with semiconductor wafers, etc.) can benefit from the methods and systems described herein (e.g., one or more indications for displaying model learning).

[0101] Reference Figure 2 , including a data set generator 272 (e.g., Figure 1 The system 200 of the dataset generator 172) builds a machine learning model (e.g., Figure 1 The dataset generator 272 may use sensor data (e.g., Figure 1The dataset generator 272 may use manufacturing parameters (e.g., historical parameters 152) to build a dataset. The dataset generator 272 may use metrology data (e.g., metrology data 160) to build a dataset. In some embodiments, the dataset generator 272 may use data output by another model, another function, another processing tool, or the like (e.g., prediction data 168, analysis data 169, simulation data 161, etc.) to build a dataset for a model. Models that receive data of different types as input or generate data of different types as output may receive datasets built according to corresponding data types from the dataset generator 272. In some embodiments, the dataset generator 272 builds training inputs (e.g., data inputs 210) based on sensor data associated with one or more processing processes (e.g., associated with one or more substrates produced) and / or processing parameter data associated with one or more processing recipes. The dataset generator 272 also generates target outputs 220 for training a machine learning model. The target output may include metrology data (e.g., metrology data for one or more substrates processed under the conditions indicated by the sensor data, processed according to one or more process recipes, or the like). In some embodiments, the metrology data 230 may include multiple measurements of a simulated wafer (e.g., each measurement corresponding to a predicted value of a feature of interest at a different spatial location of the simulated wafer). The training input data 210 and the target output data 220 are supplied to the machine learning model. For purposes of illustrating the operation of a dataset generator for training a model, the dataset generator 272 is described as training a machine learning model that accepts input data indicating processing conditions of a substrate and generates predicted metrology data for the substrate as output, but any other configuration of a model (e.g., a machine learning model) may also benefit from aspects of the present disclosure.

[0102] It is also within the scope of the present disclosure to represent the training input 210 and the target output 220 in a variety of different ways. A two-dimensional map of substrate properties, a function that recreates the map, or other data indicative of performance data of the substrate can be used as the target output 220. The data set can include processed data, smoothed data, cleaned data (e.g., outliers removed, etc.), combined data, data collected into data attributes (e.g., vectors, feature vectors, etc.), or the like.

[0103] In some embodiments, the dataset generator 272 generates a dataset (e.g., a training set, a validation set, a test set) that includes one or more data inputs 210 (e.g., a training input, a validation input, a test input) and may include one or more target outputs 220 corresponding to the data inputs 210. The dataset may also include mapping data that maps the data inputs 210 to the target outputs 220. The data inputs 210 may also be referred to as “features,” “attributes,” or “information.” In some embodiments, the dataset generator 272 may provide the dataset to Figure 1 A training engine 182, a validation engine 184, or a test engine 186, wherein the data set is used for training, validation, or testing Figure 1 Model 190. Can be relative to Figure 4A Some embodiments of generating a training set are further described.

[0104] In some embodiments, the dataset generator 272 can generate a first data input corresponding to the first sensor dataset 244A and / or the first manufacturing parameter dataset 252A to train, verify, or test the first model, and the dataset generator 272 can generate a second data input corresponding to the second sensor dataset 244B and the second manufacturing parameter dataset 252B to train, verify, or test the second model.

[0105] In some embodiments, the data set generator 272 may perform operations on one or more of the data input 210 and the target output 220. The data set generator 272 may extract patterns (slope, curvature, etc.) from the data, may combine the data (average, feature production, etc.), or may separate the data into groups (e.g., train a model on a subset of the predictive performance data) and use the groups to train separate models.

[0106] The data input 210 and target output 220 used to train, validate, or test the model may include information for a specific substrate processing recipe (e.g., a specific substrate design). The data input 210 and target output 220 may include information for a specific substrate processing system (e.g., for a specific set of manufacturing equipment). The data input 210 and target output 220 may include information for a specific type of process, target substrate design, target substrate properties, or may be grouped together in another manner.

[0107] In some embodiments, the data set generator 272 can generate a set of target outputs 220 including the metering data 230. The target outputs 220 can be divided into groups corresponding to the input data groups. Different groups of target outputs 220 can be used in conjunction with similarly defined groups of data inputs 210 (including training different models, using different groups for training, validation, and testing, etc.).

[0108] In some embodiments, a model (e.g., an unsupervised or semi-supervised model) may be trained without a target output 220. A trained model that does not provide a target output may, for example, be trained to identify significant differences (e.g., outside an error threshold) between predicted and measured performance data.

[0109] In some embodiments, the information used to train the model may come from a particular type of manufacturing equipment at a manufacturing facility with particular characteristics (e.g., Figure 1 The invention can be used to train a machine learning model for a particular group of manufacturing equipment 124 and allow the trained machine learning model to determine results for a particular group of manufacturing equipment 124 based on input of predicted performance data and measured performance data associated with one or more components that share the characteristics of the particular group. In some embodiments, the information used to train the model can be used for components from two or more manufacturing facilities and can allow the model to determine results for the components based on input from one manufacturing facility.

[0110] In some embodiments, after a data set is generated and used to train, validate, or test a model, the model may be further trained, validated, or tested or adjusted. For example, additional data may be provided to the model from processed substrates after the model was trained, validated, and tested as retraining data, revalidation data, retest data, or the like.

[0111] The data set generator 272 can generate a data set to train, validate, and / or test a model. Training a model can include generating a model mapping 273 so that the model can be used to connect input data to output data (e.g., to generate output from a provided set of inputs). In some embodiments, the model mapping 273 can include weights and biases of the model (e.g., weights and biases of nodes connecting layers of a machine learning model).

[0112] In some embodiments, a data set generator that performs functions similar to data set generator 272 can be used to train a physics-based model. A physics-based model can be configured to generate outputs based on a physical understanding of the system, based on physical assumptions about the operation of the system, based on one or more numerical solutions of one or more physical equations (e.g., heat transfer equations, mass balance equations, fluid dynamics equations, etc.), or the like. A physics-based model can be trained in a manner similar to a machine learning model. A physics-based model can set training inputs and target outputs, and can adjust one or more parameters, weights, biases, or the like to better align the model outputs with the target outputs.

[0113] In some embodiments, the physics-based model may receive a set of inputs (e.g., indicative of processing conditions of a substrate). The physics-based model may generate an output based on the set of inputs (e.g., predicted metrology data of the substrate). The physics-based model may set a target output (e.g., measured metrology data of the substrate). The physics-based model may adjust one or more parameters of the model to generate an output that is more similar to the target output (e.g., predicted metrology data) than before the adjustment was made.

[0114] In some embodiments, data set generator 272 is used to facilitate the generation of models for improving manufacturing systems. In some embodiments, a record of the learning of the trained model will be generated. For example, a visual representation of the lessons, associations, mappings, or the like learned by the model can be generated (e.g., learned by providing the model with a data set established by data set generator 272). The operation of data set generator 272 can facilitate the generation of model mapping 273 (e.g., weights and biases of a machine learning model, values ​​of adjustable parameters of a physics-based model, or the like). Visualization of the learning of the model can include visualizing model mapping 273 (e.g., changes in the output of the model as the input is transformed).

[0115] Figure 3 is an illustration of output data of the analysis of learning, association, and mapping used to generate a model according to some embodiments (e.g., generating Figure 1 169). System 300 can be used to train a model (e.g., a machine learning model) and generate data that can be used to describe the associations learned by the model. Some or all of the operations of system 300 can be used to generate output data of the machine learning model (e.g., Figure 1 Some or all of the operations of system 300 may be used to generate output data of a physics-based model (e.g., Figure 1 Forecast data 168).

[0116] Reference Figure 3 At block 310, the system 300 (eg, Figure 1 Components of the prediction system 110) perform data partitioning (e.g., via Figure 1 The data set generator 172 of the server machine 170 of the embodiment of the present invention is used to generate a training set 302, a validation set 304, and a test set 306. For example, the training set may be 60% of the historical data, the validation set may be 20% of the historical data, and the test set may be 20% of the historical data.

[0117] In block 312, system 300 performs model training using training set 302 (e.g., via Figure 1 The system 300 may train one model, or may use multiple sets of attributes (e.g., feature vectors) of the training set 302 to train multiple models (e.g., a first set of attributes including a subset of the historical data of the training set 302, a second set of attributes including a different subset of the historical data of the training set 302, etc.). For example, the system 300 may train a machine learning model using a first set of attributes in the training set to produce a first trained machine learning model, and use a second set of attributes in the training set (e.g., data different from the data used to train the first machine learning model) to produce a second trained machine learning model. In some embodiments, the first trained machine learning model may be combined with the second trained machine learning model to produce a third trained machine learning model (e.g., which may be a better predictor than the first or second trained machine learning model). In some embodiments, the multiple sets of attributes used to compare models may overlap (e.g., one model may be trained using performance data indicative of film thickness, while another model is trained using performance data indicative of both film thickness and film stress, different models may be trained using data from different locations of a substrate, models may be trained to include inputs from different sets of sensors or manufacturing parameters, etc.). In some embodiments, hundreds of models may be generated, including models with various permutations of attributes as well as combinations of models.

[0118] At block 314, system 300 performs model validation using validation set 304 (e.g., via Figure 1 314 ). The system 300 may validate each of the trained models using a corresponding set of attributes of the validation set 304. For example, the validation set 304 may use the same subset of historical data types used in the training set 302 (e.g., associated with the same on-wafer performance characteristics, the same sensors, the same input parameters, etc.), but for different input conditions. In some embodiments, the system 300 may validate hundreds of models generated at block 312 (e.g., models with various permutations of attributes, combinations of models, etc.). At block 314, the system 300 may determine the accuracy of each of the one or more trained models (e.g., via model validation), and may determine whether the accuracy of one or more of the trained models meets a threshold accuracy. In response to determining that the accuracy of no trained model meets the threshold accuracy, the process returns to block 312, where the system 300 performs model training using a different set of attributes of the training set. In response to determining that the accuracy of one or more of the trained models meets the threshold accuracy, the process continues to block 316. The system 300 can discard trained machine learning models that have an accuracy below a threshold accuracy (e.g., based on a validation set).

[0119] At block 316, the system 300 may perform model selection (eg, via Figure 1 The selection engine 185 of block 314 may be used to determine which of the one or more trained models that meet the threshold accuracy has the highest accuracy (e.g., the selected model 308 as verified in block 314). If only a single model is being trained, the operation of block 316 may be skipped. In response to determining that two or more of the trained models that meet the threshold accuracy have the same accuracy, the process may return to block 312, where the system 300 performs model training using a further refined training set (e.g., corresponding to a further refined feature set) to determine the trained model with the highest accuracy.

[0120] At block 318, system 300 performs model testing using test set 306 (eg, via Figure 1 The system 300 may test the first trained machine learning model using the first set of attributes in the test set to determine whether the first trained machine learning model meets the threshold accuracy (e.g., based on the first set of attributes in the test set 306). In response to the accuracy of the selected model 308 not meeting the threshold accuracy (e.g., the selected model 308 overfits the training set 302 and / or the validation set 304 and cannot be applied to other data sets (e.g., the test set 306)), the process continues to box 312, where the system 300 performs model training (e.g., retraining) using a different training set that may correspond to a different set of attributes or a reorganization of the substrate (e.g., the data set) divided into a training set, a validation set, and a test set. In response to determining that the accuracy of the selected model 308 meets the threshold accuracy based on the test set 306, the process continues to box 320. In at least block 312 , the model may learn patterns in the simulated sensor data for making predictions, and in block 318 , the system 300 applies the model to the remaining data (eg, the test set 306 ) to test the predictions.

[0121] At block 320, the system 300 receives simulated data 354 (eg, Figure 1 The trained model may be used to determine (e.g., extract) analytical data 369 (e.g., Figure 1 of analytical data 169) to perform actions (e.g., Figure 1 Corrective actions associated with manufacturing equipment 124, Figure 1 The client device 120 provides an alert, Figure 1The presentation component 115 provides a warning to the user, etc.).

[0122] In some embodiments, retraining of the machine learning model is performed by supplying additional data to further train the model. Current data 346 may be provided at block 312. Current data 346 may differ from the data originally used to train the model by incorporating input parameters that were not part of the original training, combinations of input parameters outside of the parameter space spanned by the original training, or may be updated to reflect chamber-specific knowledge (e.g., differences from an ideal chamber due to manufacturing tolerance ranges, aging parts, drifting parts, maintenance performed, etc.). The selected model 308 may be retrained based on this data.

[0123] In some embodiments, one or more of actions 310-320 may occur in various orders and / or with other actions not presented and described herein. In some embodiments, one or more of actions 310-320 may not be performed. For example, in some embodiments, one or more of the data partitioning of block 310, the model validation of block 314, the model selection of block 316, or the model testing of block 318 may not be performed. When training a physics-based digital twin model (e.g., taking measurements of processing conditions as input and generating predicted performance data of a substrate as output), a subset of these operations may be performed.

[0124] 4A to 4D is a flow chart of methods 400A-D associated with describing and / or visualizing model learning according to certain embodiments. Methods 400A-D may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, processing devices, etc.), software (e.g., instructions running on a processing device, a general-purpose computer system, or a dedicated machine), firmware, microcode, or a combination thereof. In some embodiments, methods 400A-D may be performed in part by prediction system 110. Method 400A may be performed in part by prediction system 110 (e.g., Figure 1 The server machine 170 and the data set generator 172, Figure 2According to an embodiment of the present disclosure, the prediction system 110 may use method 400A to generate a data set to train, verify, or test at least one of the models. The model may be a physics-based (e.g., digital twin) model (e.g., to generate predicted performance data of a substrate), a machine learning model (e.g., to generate predicted performance data of a wafer, to generate data indicating corrective actions associated with a component of manufacturing equipment, etc.), a statistical model, or another model trained to receive inputs and generate outputs related to substrate manufacturing or processing. Methods 400B-D may be performed by a prediction server 112 (e.g., prediction component 114, etc.). Methods 400B-D may be performed by other components of the prediction system 110. The operations described as being associated with methods 400B-D may be performed by a server machine 180 (e.g., training engine 182). In some embodiments, the non-transitory storage medium stores instructions that, when executed by a processing device (eg, prediction system 110, server machine 180, prediction server 112, etc.), cause the processing device to perform one or more of methods 400A-D.

[0125] For simplicity of description, methods 400A-D are illustrated and described as a series of operations. However, operations according to the present disclosure may occur in various orders and / or simultaneously, and with other operations not presented and described. In addition, not all of the operations shown may be performed to implement methods 400A-D according to the disclosed objectives. In addition, those with ordinary knowledge in the art will understand that methods 400A-D may alternatively be represented as a series of interrelated states via state diagrams or events.

[0126] Figure 4A is a method for generating output data (e.g., Figure 1 Flowchart of method 400A for predicting data 168 using a model (e.g., a machine learning model) on a data set.

[0127] Reference Figure 4A In some embodiments, at block 401 , processing logic for implementing method 400A initializes a training set T to an empty set.

[0128] At block 402, processing logic generates a first data input (e.g., a first training input, a first validation input), which may include sensor data, manufacturing parameter data, measured substrate property data, substrate metrology data (e.g., film properties (e.g., thickness, material composition, optical properties, roughness, etc.)). In some embodiments, the first data input may include a first set of attributes for a type of data, and the second data input may include a second set of attributes for a type of data (e.g., Figure 3 described).

[0129] At block 403, processing logic generates a first target output for one or more of the data inputs (e.g., a first data input). In some embodiments, the first target output is performance data of the substrate. In some embodiments, the first target output is data indicating a corrective action. In some embodiments, no target output is generated (e.g., for training an unsupervised machine learning model).

[0130] At block 404, processing logic optionally generates mapping data indicating an input / output mapping. The input / output mapping (or mapping data) may refer to a data input (e.g., one or more of the data inputs described herein), a target output for the data input, and an association between the data input and the target output. In some embodiments (e.g., embodiments without target output data), these operations may not be performed.

[0131] At block 405 , in some embodiments, processing logic adds the mapping data generated at block 404 to data set T.

[0132] At block 406, the data set T is sufficient to satisfy Figure 1 The processing logic is branched by at least one of training, validation, and / or testing of the model 190. If so, execution proceeds to block 407, otherwise execution continues back to block 402. It should be noted that in some embodiments, the adequacy of the data set T can be determined simply based on the number of inputs in the data set that are mapped to outputs in some embodiments, while in some other embodiments, in addition to or without the number of inputs, the adequacy of the data set T can be determined based on one or more other criteria (e.g., a measure of the diversity of data examples, precision, the span of the input and / or output space, etc.).

[0133] At block 407, processing logic provides a data set T (eg, provided to Figure 1 In some embodiments, the data set T is a training set and is provided to the training engine 182 of the server machine 180 for performing training. In some embodiments, the data set T is a validation set and is provided to the validation engine 184 of the server machine 180 for performing validation. In some embodiments, the data set T is a test set and is provided to the test engine 186 of the server machine 180 for performing testing.

[0134] The operations of box 407 can produce a trained model (e.g., a model mapping between inputs and outputs (e.g., values ​​of weights and biases between nodes of a layer of a machine learning model, values ​​of adjustable parameters of a physics-based model, etc.) can be produced). The model mapping can be described (e.g., an indication of the learning of the model can be visualized) via a method of the present disclosure (e.g., any of methods 400A-D).

[0135] Figure 4B is a flow chart of a method 400B for generating a visual representation of the learning (e.g., input to output mapping) of a model according to some embodiments. In some embodiments, the model comprises a machine learning model. In some embodiments, the model comprises a physics-based model. In some embodiments, the model comprises a statistical model. In some embodiments, the model is configured to receive as input one or more indications of processing conditions for processing a substrate (e.g., sensor data, manufacturing parameters, etc.). In some embodiments, the model is configured to generate as output a predicted feature of a substrate processed according to the conditions indicated by the input. In some embodiments, the model is configured to generate an indication of the value of one or more predicted features of the substrate (e.g., substrate thickness at one or more locations, substrate resistivity at one or more locations, substrate film resistance, optical properties (e.g., extinction coefficient, refractive index), indication of substrate geometry (e.g., critical dimension, sidewall height, line segment width, depth, etc.), etc.). In some embodiments, one or more properties (e.g., statistical measures) of the predicted features can be calculated. The properties can include the mean, median, standard deviation, uniformity, skewness, kurtosis, interquartile range, or any other statistical measure suitable for predicting feature values. In some embodiments, methods similar to method 400B can be used to generate visual displays of learning of a differently configured model (e.g., configured to receive data as input that is different from the model described in method 400B, and / or to generate data as output that is different from the model described in method 400B).

[0136] In some embodiments, the model may be trained (e.g., before method 400B begins). Training the model may include providing training inputs to the model and providing target outputs to the model. Training the model may include processing logic receiving multiple sets of input values ​​(e.g., values ​​indicating processing conditions for processing multiple substrates). The processing logic may also receive target output data for training the model (e.g., metrology measurements of substrates processed under the processing conditions associated with the set of input values). Training the model may include processing logic providing multiple sets of input values ​​to the model as training inputs and providing target output data as target outputs.

[0137] At block 410, processing logic receives a first value. The first value is associated with a first input parameter of a model (e.g., the model may accept the first value as the first input parameter). The first input parameter is associated with a first processing condition of a substrate processing process (e.g., represents a value corresponding thereto). The first processing condition (and subsequent processing conditions of methods 400B-D) may be any processing condition of a substrate processing system that has an impact on a product (e.g., temperature, gas flow, gas characteristics and / or mix (e.g., 20% reactive gas in a non-reactive carrier, 15% reactive gas in a non-reactive carrier, etc.), pressure, clamping power, RF power, processing time, etc.).

[0138] At block 412, processing logic receives a first plurality of values. The first plurality of values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values ​​(e.g., a plurality of values ​​spanning a range of values). In some embodiments, the values ​​span a range determined by a user. In some embodiments, the values ​​span a range determined by a trained model (e.g., defined relative to a model input space). In some embodiments, the values ​​span a range that is a portion of a range associated with an input space of the model (e.g., a user may select a range that is 10% of the entire range, 20% of the entire range, 30% of the entire range, or any subset of the entire range). Each of the first plurality of values ​​is associated with a second input parameter of the model. The second input parameter of the model is associated with a second processing condition of a substrate processing process.

[0139] At block 414, the processing logic provides the first value and the first plurality of values ​​to the model. The processing logic may provide the first value and the first plurality of values ​​as inputs to the model. The model may be configured to generate outputs based on the inputs. The inputs may include additional values ​​(e.g., values ​​associated with a third, fourth, etc. input parameter). The model may generate outputs associated with each set of inputs (e.g., an input set including the first value and a first one of the first plurality of values, an input set including the first value and a second one of the first plurality of values, etc.) separately (e.g., sequentially) to generate outputs corresponding to each set of input values.

[0140] In some embodiments, the processing logic may provide an additional set of inputs to the model. For example, the processing logic may receive a second value associated with a second input parameter of the model. The processing logic may further receive a second plurality of values, each of which is associated with a first input parameter of the model. The processing logic may provide multiple sets of inputs to the model (e.g., a first set including a second value and a first value in a second plurality of values, a second set including a second value and a second value in a second plurality of values, etc.). Similar operations may be performed for additional multiple values ​​associated with additional input parameters (e.g., associated with additional processing conditions). In some embodiments, a series of sets of values ​​may be provided to the model, wherein each of the multiple sets of values ​​includes all values ​​except one of a central value, a baseline value, a value of interest, etc., and other values ​​are allowed to vary within the entire range. The values ​​of the series of sets may include a series of sets, wherein each of the series includes input parameters with different variations. In this way, a set of outputs may be obtained from the model, each corresponding to a set of baseline input values, but one parameter varies. In some embodiments, more than one value may vary in a set. For example, multiple sets of baseline values ​​may be used instead of a single set of baseline values, multiple values ​​associated with input parameters may be varied within the same set (e.g., to demonstrate how varying two or more input parameters affects the output, to demonstrate learning of a model about varying two or more input parameters, etc.), or the like.

[0141] In some embodiments, processing logic may receive a second value associated with a first input parameter of the model. For example, the GUI may include an option for a user to change the baseline value of the first input parameter. The system may perform similar operations (e.g., providing the second value and the first plurality of values ​​to the model, providing the second value and the second plurality of values ​​to the model, etc.) in response to receiving user input, in response to receiving an instruction to perform an operation based on the second value, etc. In some embodiments, a user may transform one or more baseline inputs, one or more ranges associated with multiple values, one or more features to be studied, one or more properties to be displayed, etc., and the system may perform the operations described herein in response to user selections to demonstrate model learning, associations, mappings, or the like.

[0142] At block 416, processing logic receives a first plurality of outputs from the model. Each of the first plurality of outputs is associated with a first value (associated with a first input parameter of the model) and one of the first plurality of values ​​(associated with a second input parameter of the model). Each of the first plurality of outputs is associated with a first characteristic of one of the first plurality of simulated substrates (e.g., a characteristic of the substrate that the model is configured to predict).

[0143] In some embodiments, further results can be received from the model. For example, the model can provide a second plurality of outputs, a third plurality of outputs, etc. The second plurality of outputs can be associated with a second value (e.g., a center value or baseline value associated with a second input parameter) and a second plurality of values ​​(e.g., a plurality of values ​​associated with a first input parameter). In this way, the model can provide two or more series of output data, one of which is associated with a change in the first input parameter while keeping the rest of the parameter at a set of baseline values, and the second is associated with a change in the second input parameter while keeping the rest of the parameter at the set of baseline values, and so on. In the example of two series of output data, visually, this can be represented as two arcs, curves, etc., each arc including a plurality of data points (e.g., each of a plurality of data points associated with one of the plurality of inputs). The curves can intersect at points (represented in the output space) associated with a baseline group of input conditions (e.g., a model output received when a baseline input group is provided as an input to the model).

[0144] In some embodiments, more input parameters can be changed, which may result in a greater number of curves. In some embodiments, multiple inputs can be changed to generate a curve (e.g., a curve can be generated, wherein the data points on the curve are associated with the output of the model generated by adjusting two or more input parameters simultaneously). Changing multiple inputs can allow a change in an output feature, a property of a feature, or the like, without affecting another. For example, transforming two input parameters may have a similar effect on a property, a feature, etc. (e.g., adjusting the two input parameters to a higher value may shift the property of the output to a higher value), but may produce an opposite effect on a second property, a second feature, or the like (e.g., adjusting the first input parameter to a higher value may shift the value of the property of the output in the direction opposite to adjusting the second parameter to a higher value). A set of input parameters can be provided that maintains all inputs except these two at a baseline value, wherein these two are changed simultaneously, for example, to maintain an output property or feature while adjusting another property or feature.

[0145] In some embodiments, further results received from the model may include a third plurality of outputs, wherein each of the third plurality of outputs is associated with a second value and a plurality of values, the second value being associated with the first input parameter (e.g., a different set of baseline conditions), and the plurality of values ​​being associated with the second input parameter. In some embodiments, the plurality of values ​​associated with the second input parameter may be the same as the values ​​used in conjunction with the first value associated with the first input parameter. In some embodiments, the plurality of values ​​may be different from the plurality of values ​​used in conjunction with the first value of the first input parameter (e.g., one or more values ​​may be different, there may be a different total number of values, etc.). In this way, a user may specify a different set of baseline input conditions. In some embodiments, more than one baseline condition may be transformed (e.g., a baseline value associated with the second input parameter, a baseline value associated with the third input parameter, etc.).

[0146] At block 418, a first plurality of outputs is prepared for presentation by the processing logic. The first plurality of outputs (e.g., indicators of the outputs of the model) are presented via a presentation component of a graphical user interface (GUI). In one embodiment, the presentation component includes two axes (e.g., two orthogonal axes). In some embodiments, the presentation component may include three axes (e.g., three orthogonal axes). In some embodiments, the presentation component may include a scatter plot (e.g., each simulated substrate may be associated with a value corresponding to a first axis (e.g., a first property of a first feature) and a value corresponding to a second axis (e.g., a second property of a second feature), and data points may be displayed at locations indicating the two values). The first of the two axes may correspond to the first property of the first feature (e.g., a first statistical measure of a feature that the model is configured to predict). The second of the two axes may correspond to the second property of the first feature. Preparing the first plurality of outputs for presentation includes facilitating the generation of a graph for display in the presentation component. The graph indicates the value of the first property of the first feature associated with each of the first plurality of outputs (e.g., by the position of the data point relative to the first axis) and the value of the second property of the first feature (e.g., by the position of the data point relative to the second axis).

[0147] In some embodiments, data points associated with each of the first plurality of outputs may generate a curve in a presentation component. In some embodiments, further outputs may be plotted (e.g., generating additional curves). Further outputs may be represented as additional curves (e.g., meeting at data points associated with a baseline input group). In some embodiments, data associated with a second value of one or more input parameters (e.g., a second set of baseline conditions) may be presented. In some embodiments, in response to a change in the baseline input parameters, the processing logic may generate a new graph (e.g., one or more curves of data points) including new data points associated with the new baseline input conditions. In some embodiments, a graph may include data points associated with two or more sets of baseline conditions (e.g., one or more curves of data points). In some embodiments, the graph of the presentation component may include other features (e.g., an indication of the range of the input space and / or output space of the model). In some embodiments, the graph of the presentation component may include one or more visual indicators (e.g., for distinguishing outputs associated with each of the plurality of inputs). For example, the plurality of input values ​​may range from a lowest value to a highest value. The output associated with the lowest input value may be distinguished from the output associated with the highest input value, for example, via color, shape, pattern, label, or the like. In conjunction with FIG. 5A to FIG. 5C Presentation components and features of the GUI are discussed in more detail.

[0148] Figure 4C is a flow chart of a method 400C for generating a visual representation of learning of multiple models according to some embodiments. The model associated with method 400C can be combined with Figure 4B The described models share one or more characteristics.The operations associated with method 400C may share one or more characteristics with the operations of method 400B.

[0149] At box 420, the processing logic receives a first value associated with a first input parameter of the first model. The first value is also associated with a first input parameter of the second model. The first input parameter is associated with a first processing condition of a substrate processing process (for example, the first input parameter can be a measured value and / or predicted value of a condition in a processing chamber (for example, temperature, pressure, etc.)). The first model can be configured to receive one or more inputs and generate one or more predictions of substrate properties (for example, metrology of the substrate, material properties of the substrate, etc.) as output. The first model can be configured to provide one or more indications associated with a feature of the simulated / predicted substrate (the feature can be thickness, another physical dimension, resistivity, or any other metric of interest). In some embodiments, the processing logic can further receive a second value also associated with the first input parameter, a third value associated with the second input parameter, and the like.

[0150] At block 422, processing logic receives a first plurality of values. The first plurality of values ​​ranges from the lowest value in the first plurality of values ​​to the highest value in the first plurality of values. Each of the first plurality of values ​​is associated with a second input parameter of the first model. Each of the first plurality of values ​​is also associated with a second input parameter of the second model. The second input parameter is associated with a second processing condition of the substrate processing process. In some embodiments, processing logic may receive another plurality of values. Processing logic may receive a second plurality of values ​​associated with the first input parameter of the first model and the second model. Processing logic may receive a third plurality of values ​​associated with the third input parameter of the first model and the second model. Processing logic may receive a fourth plurality of values, which may be associated with the first input parameter, but may include different values ​​(e.g., different ranges, different intervals, etc.) than the first plurality of values.

[0151] At block 424, processing logic provides the first value and the first plurality of values ​​to the first model. Processing logic also provides the first value and the first plurality of values ​​to the second model. In some embodiments, different and / or additional values ​​(e.g., a second value, a second plurality of values, a third plurality of values, etc.) are provided.

[0152] At block 426, processing logic receives a first plurality of outputs from the first model. Each of the first plurality of outputs is associated with a first value and one of the first plurality of values. Each of the first plurality of outputs is associated with a first characteristic of one of the first plurality of simulated substrates (e.g., a characteristic of the simulated substrate that the first model is configured to predict). In some embodiments, different and / or additional outputs (e.g., outputs associated with a second value, a second plurality of values, etc.) may be received.

[0153] At block 428, processing logic receives a second plurality of outputs from the second model. Each of the second plurality of outputs is associated with the first value and one of the first plurality of values. Each of the second plurality of outputs is associated with a second characteristic of one of the first plurality of simulated substrates (e.g., a characteristic of the simulated substrate that the second model is configured to predict). In some embodiments, different and / or additional outputs (e.g., outputs associated with the second value, the second plurality of values, etc.) may be received.

[0154] At box 429, the processing logic prepares the first plurality of outputs and the second plurality of outputs for presentation via a presentation component of the GUI. In some embodiments, additional outputs may be further prepared for presentation (e.g., outputs associated with different groups or groups of inputs). The presentation component includes two axes. The first axis of the two axes corresponds to a first property of a first feature (e.g., a feature for which the first model is configured to predict one or more values). The second axis of the two axes corresponds to a second property of a second feature (e.g., a feature for which the second model is configured to predict one or more values). Preparing the first plurality of outputs and the second plurality of outputs for presentation includes facilitating the generation of a graphic for display in the presentation component, the graphic indicating a value of the first property of the first feature and a value of the second property of the second feature for each simulated substrate of the first plurality of simulated substrates.

[0155] In some embodiments, the first property and the second property (e.g., a statistical measure of one or more feature predictions) may be the same property (e.g., may both be averages of feature predictions). In some embodiments, the first property and the second property may be different properties. In some embodiments, the first model comprises a machine learning model. In some embodiments, the second model comprises a machine learning model. In some embodiments, one or more of the models is a statistical model, a physics-based model, or the like. In some embodiments, the generated graph may be combined with Figure 4B The graphics described share one or more characteristics.

[0156] Figure 4D is a flow chart of a method 400D for generating a graph demonstrating learning of a model according to some embodiments. The method 400D may share one or more features with the methods 400B and / or 400C.

[0157] At block 430, processing logic receives a first value associated with a first input parameter of the model. The first input parameter is associated with a process recipe for processing a substrate. The first input parameter may be associated with a manufacturing parameter, a sensor reading, etc., for example.

[0158] At block 432, processing logic receives a first plurality of values. The first plurality of values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values. Each of the first plurality of values ​​is associated with a second input parameter of the model. The second input parameter is associated with a process recipe.

[0159] At block 434, processing logic provides the first value and the first plurality of values ​​to a first model. The first model may be configured to generate one or more outputs predicting performance of the simulated substrate in response to receiving a set of input values.

[0160] At box 436, the processing logic receives a first plurality of outputs from the model. Each of the first plurality of outputs is associated with a first value and one of the first plurality of values ​​(e.g., each output is an output generated by the model according to the first value of the input as the first parameter and the first plurality of values ​​as the input of the second input parameter). Each of the first plurality of outputs is associated with a first feature of one of the first plurality of simulated substrates. In some embodiments, additional inputs may be provided to the model, and additional outputs may be received from the model. In one or more input parameters, the additional inputs may be different from the initial inputs. The additional inputs may provide different multiple inputs, different single values ​​(e.g., different baseline conditions), or the like. Each output (e.g., each predicted feature, each set of predicted features, each property of a set of predicted features, or the like) may be associated with a set of inputs (e.g., one value of each input parameter). In some embodiments, an additional input group may be provided to the model, and the model may generate an additional output group (e.g., an output associated with a second plurality of simulated substrates).

[0161] At block 438, processing logic prepares the first plurality of outputs for presentation via a presentation component of the GUI. The presentation component includes two axes (two independent axes (e.g., axes associated with values ​​that are independent of each other)). A first axis of the two independent axes corresponds to a first property of a first feature. Preparing the first plurality of outputs for presentation includes facilitating generation of a graphic for display in the presentation component. The graphic indicates a value of the first property of the first feature associated with an output in the first plurality of outputs.

[0162] Figure 5A 500A is an exemplary presentation component that displays an indication of learning of one or more models (e.g., input / output mappings, associations, etc.) according to some embodiments. The presentation component 500A includes a first axis 502 and a second axis 504. These axes can be orthogonal, independent, or the like. The first axis 502 can be associated with a first feature, a first property of the first feature, or the like. For example, the presentation component 500A can display data associated with one or more machine learning models to visualize the learning of one or more models (e.g., learned associations, learned input / output mappings, or the like). The first axis 502 can indicate a value of a first property of a first feature (e.g., a statistical measure associated with one or more predicted measures of a simulated substrate). The second axis 504 can indicate a value of a second property of a second feature. In some embodiments, the first property is the same as the second property. In some embodiments, the first feature is the same as the second feature. In some embodiments, the first feature is associated with an output of the first model, and the second feature is associated with an output of the second model.

[0163] Presentation component 500A includes an indication of output space 506. The indication of output space 506 may include, for example, Figure 5A Multiple data points shown. In some embodiments, data points indicating output space 506 can be generated by providing multiple sets of input data (e.g., one set per output data point) to one or more models associated with presentation component 500A. In some embodiments, the inputs associated with the output data points can span the input space of the model, can span a portion of the input space of the model, can be randomly selected from the input space of the model (e.g., enough points can be selected to effectively / substantially span the input space of the model), or the like. In some embodiments, the indication of output space 506 can be represented in different ways (e.g., by a colored or shaded area of ​​presentation component 500A, by a portion of presentation component 500A surrounded by a shape to indicate the output space of one or more models, or the like).

[0164] The presentation component 500A may further include a first curve 508 and a second curve 510. The first curve 508 may include a plurality of data points (in Figure 5A 508). The first curve 508 may include a visual distinction between outputs associated with a set of inputs having one or more varying input values. For example, the first curve 508 may be associated with a series of groups of inputs, wherein each of the series maintains all inputs except a first input at a baseline value. The first input value may vary from a minimum value to a maximum value. The shape of the data points representing the output values ​​may indicate a progression of the associated input values ​​(e.g., the shape may range from an output value associated with the set of inputs including the lowest value of the first input to an output value associated with the set of inputs including the highest value of the first input).

[0165] The second curve 510 may include a plurality of output data points (represented as black triangles). The second curve 510 may include output data points, each of which is associated with a set of input parameter values. The second curve 510 may include output data points associated with a series of sets of inputs, such that each of the series is associated with a value of a second input parameter that transforms one or more models. The first curve 508 and the second curve 510 may intersect at an area 512 (indicated by a dashed circle) associated with a baseline set of input parameter values.

[0166] In some embodiments, more than two input parameters may be changed (e.g., presentation component 500A may include more than two curves). Each curve may be associated with a different input parameter. In some embodiments, each curve may intersect at region 512. In some embodiments, different curves may be associated with different input baseline groups (e.g., the curves may intersect in multiple regions of presentation component 500A). In some embodiments, only one curve may be presented (e.g., associated with changing one input parameter).

[0167] Figure 5B is an exemplary presentation component 500B depicting learning of one or more models according to some embodiments. In some embodiments, a user can select a set of baseline input values. The presentation component 500B includes a second curve 514 (e.g., including a series of data points each associated with a set of input parameter values, wherein each of the series includes all inputs set to a baseline value except for a second input parameter). The presentation component 500B can be associated with Figure 1 The same one or more models as the presentation component 500A of FIG. The presentation component 500B may be displayed in response to the processing logic receiving an adjustment for a set of baseline input values. The presentation component 500B may be displayed after the user inputs a set of adjusted baseline input values ​​via the GUI.

[0168] First curve 508 and second curve 514 intersect at region 516 (e.g., region 516 may indicate one or more output values ​​associated with the updated baseline input set). In some embodiments, the same range of input values ​​may be associated with the curve after adjustment for the baseline condition (e.g., curve 508 includes the same data points in presentation component 500A as in presentation component 500B). In some embodiments, the range of input values, the density of input values, the value of multiple values, or the like may change when the baseline condition changes. In some embodiments, one or more sets of input conditions may be provided to one or more models associated with presentation component 500B in response to a change in the baseline condition, a change in one or more multiple input values, or the like. Outputs from the one or more models may be displayed via presentation component 500B.

[0169] In some embodiments, one or more curves associated with multiple sets of input parameter values ​​(e.g., varying one input parameter value from a minimum value to a maximum value) may be shaped differently depending on the values ​​of other input parameter values. For example, the shape of curve 510 may be different from curve 514. Presentation components 500A and 500B can concisely display such nonlinearities in the input / output mapping of one or more models associated with the presentation component by allowing a user to change the set of baseline values.

[0170] Figure 5C is an exemplary GUI 500C according to some embodiments. The GUI includes a presentation component 520 (e.g., including Figure 5B GUI 500C further includes additional components (eg, components for presenting additional information to the user, components for receiving instructions from the user, etc.).

[0171] GUI 500C includes a space setting component 522. The space setting component 522 may include one or more options related to the space displayed in the presentation component 520 (e.g., an input space for one or more models, an output space for one or more models, or the like). The space setting component 522 may include a space editing component 524. The space editing component 524 may allow a user to adjust the input space of one or more parameters associated with one or more curves of the presentation component 520. For example, the space editing component 524 may open a window for editing one or more input value ranges for generating one or more sets of data points (e.g., curves) for the presentation component 520. For example, a minimum value, a maximum value, a difference between adjacent values, a number of values ​​provided, or the like may be adjusted by the space editing component 524.

[0172] The space setting component 522 may further include a space restriction component 526. In some embodiments, a reduced area of ​​the model space (e.g., input space, output space, etc.) may be of particular interest. The space restriction component 526 may allow a user to limit the space displayed by the presentation component 520. For example, the space restriction component 526 may allow a user to enter a percentage value. The presentation component 520 may present a portion of the maximum range of one or more varying input values ​​(e.g., corresponding to the percentage value of the input). In some embodiments, the presentation component 520 may utilize a restricted space to display the same number of data points (e.g., the number selected by the user) as in an unrestricted space (e.g., if the input space is restricted, the density of the points increases). In some embodiments, the presentation component 520 may display a different number of data points in a restricted space than in an unrestricted space. In some embodiments, in response to a user selection of a component via the space setting component 522, one or more series of inputs may be provided to one or more models.

[0173] GUI 500C may include a presentation settings component 528. The presentation settings component 528 may be used to adjust various visual settings associated with presentation component 520. Settings (e.g., data point shape), a method of visually distinguishing between outputs associated with a lowest varying input value of a plurality of input values ​​and a highest varying input value of a plurality of input values, data point and / or curve colors, spacing of data points presented (e.g., an option to present fewer data points than received from one or more models), toggling display of a feature representing a range of an output space, or the like may be associated with presentation settings component 528.

[0174] GUI 500C may further include a baseline adjustment component 530. The user may adjust the set of baseline values ​​(e.g., adjust a point in the input space around which an input parameter is adjusted, adjust an intersection of two or more curves in the output space at an output data point, etc.) using baseline adjustment component 530. Baseline adjustment component 530 may transform the value of one or more baseline input parameters.

[0175] In some embodiments, the baseline adjustment component 530 may include an input selection component 532. The input selection component may enable a user to select an input whose baseline input value is to be adjusted. The input selection component 532 may include a list, a drop-down list, a series of icons, a fillable field, an open window for input parameter selection, or the like. The baseline adjustment component 530 may further include a value selection component 534. The value selection component may allow adjustment of the value of the input parameter indicated by the input selection component 532. The value selection component may include a fillable field, a slider, one or more buttons or icons, or the like.

[0176] In some embodiments, a new graph can be generated in response to a user adjusting a baseline value associated with presentation component 520. The new graph can include the new baseline value (e.g., two or more curves can extend from the new baseline value to depict changes in output as a function of input changes to the new baseline value). In some embodiments, in response to the user adjusting the baseline value, the new value is provided to one or more machine learning models associated with generating data for presentation via presentation component 520.

[0177] GUI 500C can further include an axis adjustment component 536. Axis adjustment component 536 can adjust the axes of presentation component 520. Axis adjustment component 536 can allow a user to adjust each axis individually (e.g., can allow adjustment of a first axis and adjustment of a second axis). In some embodiments, presentation component 520 can depict a three-dimensional graph (e.g., a three-dimensional scatter plot of three dimensions of the output space of one or more models). Axis adjustment component can allow a user to adjust a third axis.

[0178] Adjustments to an axis may include selecting features to be represented by the axis (e.g., predicted features and / or measurements of a simulated substrate). In some embodiments, selections of different features may associate different models (e.g., machine learning models) with the presentation component 520. After the user selects one or more settings for an axis, one or more series of groups of input conditions may be provided to one or more models, and multiple outputs may be received from one or more models for generating a graph presented in the presentation component 520. In some embodiments, adjustments to an axis may include selecting properties of the features represented by the axis (e.g., statistical measures). Selection of one or more features and / or properties may include selecting from a list, a set of icons or buttons, a fillable field, etc. In some embodiments, the GUI 500C may display multiple graphs (e.g., multiple charts with different properties corresponding to a first axis and a second axis. Multiple charts may be used to simultaneously display additional learning of a model, input / output mapping, etc.

[0179] Figure 5D 5 is an exemplary presentation component 500D depicting model learning associated with a plurality of varying input parameters according to some embodiments. Presentation component 500D includes a first axis 540. First axis 540 may correspond to a first property (e.g., a statistical measure) of a first feature (e.g., a model output) of a simulated substrate. Second axis 542 may correspond to a second property of a second feature. The second property may be the same or different from the first property. The second feature may be the same or different from the first feature.

[0180] The presentation component 500D may include an indication of an output space 544. The indication of the output space 544 may be represented as a plurality of points (as shown), a shaded area, a bounded area, or the like. The indication of the output space 544 may be generated by supplying a random sampling of input conditions to one or more machine learning models associated with the data of the presentation component 500D. The indication of the output space 544 may be generated by supplying a systematic sampling of input conditions to one or more machine learning models associated with the data of the presentation component 500D.

[0181] The presentation component 500D includes curves 546. Each curve can represent changing one input parameter from a maximum value to a minimum value while maintaining all other input parameters at a set of baseline values. The output received when providing the set of baseline inputs to one or more models associated with the data of the presentation component 500D can be represented by the location where the curves cross. Changing a first input while maintaining other inputs at the set of baseline values ​​can produce a first curve 548, changing a second set of inputs can produce a second curve 550, changing a third set of inputs while maintaining other inputs at the set of baseline values ​​can produce a third curve 552, and so on.

[0182] The presentation component 500D can visually display the learning of one or more models. The curve of the presentation component 500D can show how the output of the model changes as the input changes. For example, a user can aim to reduce the second property of the second feature associated with the second axis 542 (as shown by the arrow) (e.g., in a new substrate manufacturing process, an updated substrate design, or the like). For example, the second property of the second feature can be the average thickness of the substrate, and a thinner substrate can be targeted. The curve 546 visually shows which input parameters have an impact on the second feature in the output space, and gives an indication of how strong the impact is on the second property of the second feature near the set of baseline values ​​(e.g., close to, in the input space, the baseline input).

[0183] The user may determine from presentation component 500D that the input parameter associated with curve 552 does not have a strong effect on the second property of the second feature (e.g., compared to other input parameters associated with other curves). For example, curve 552 may be associated with an input parameter (e.g., air pressure), and air pressure may not have a strong effect on some output results (e.g., substrate thickness). Depending on, for example, the display settings of presentation component 500D, the strength of the effect may be estimated by the length of the curve, the density of points along the curve, etc.

[0184] The user may determine from the presentation component 500D that the input parameter associated with the curve 550 has a moderate effect on the second property of the second feature (e.g., compared to other input parameters associated with other curves). For example, the curve 550 may be associated with an input parameter (e.g., temperature), and the temperature may have an effect on an output result (e.g., average substrate thickness) (e.g., learned by one or more machine learning models). The presentation component 500D provides further understanding associated with the first axis 540 to indicate that transforming the input value associated with the curve 550 will not have a strong effect on the second output property. For example, the second property of the second feature may be the average resistivity of the substrate. The presentation component 500D may visually make it clear that transforming the input value associated with the curve 550 (e.g., temperature) will have an effect on the second property of the second feature (e.g., thickness of the substrate indicated by the second axis 542), but will not have a strong effect on the first property of the first feature (e.g., resistivity).

[0185] The user can determine from the presentation component 500D that the input parameter associated with the curve 548 has a strong influence on the second property of the second feature (e.g., compared to other input parameters associated with other curves). For example, the curve 548 can be associated with an input parameter (e.g., processing time), and the processing time can have an influence on the output result (e.g., average substrate thickness).

[0186] The input associated with curve 548 also has a strong effect on the first property of the first feature (e.g., average resistivity). The user can easily see the effect that transforming the input parameter associated with curve 548 will have on both the property associated with the first axis 540 and the property associated with the second axis 542. The presentation component 500D can display nonlinear learning of one or more models (e.g., when the input parameter associated with curve 548 changes, the second property temporarily decreases and then increases). Traditional model learning presentations (e.g., bar graphs) cannot capture these nonlinearities and may only display learning adjacent to a set of baseline conditions. Using the GUI associated with the presentation component 500D, the set of baseline conditions can be transformed, and patterns of curves 546 associated with different sets of baseline conditions (e.g., baseline conditions around different portions of one of the curves displayed by the presentation component 500D) can be displayed. The user can use the displayed learning of one or more models to, for example, build an intuition about how substrate processing conditions affect substrate properties. The displayed learning can be used to visually confirm corrective actions (e.g., confirm the effect of updating a processing recipe). Displayed learning can be used to convey to the user that the model has produced logical associations between inputs and outputs (e.g., a user may be more likely to trust a model when it can be shown that the model matches the user's intuition, the model matches the user's experience, the model includes associations that are not obviously suspicious (e.g., discontinuities in one or more curves), etc.). Displayed learning can be used to indicate the reliability of the model (e.g., rapidly oscillating or discontinuous curves may indicate unreliability of the model, sampling bias or incompleteness of the training set, or the like).

[0187] Figure 6 6 is a block diagram illustrating a computer system 600 according to certain embodiments. In some embodiments, the computer system 600 can be connected (e.g., via a network (e.g., a local area network (LAN), an intranet, an extranet, or the Internet)) to other computer systems. The computer system 600 can operate in the function of a server or client computer in a client-to-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 600 can be provided by a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a network device, a server, a network router, a switch or a bridge, or any device capable of executing a set of instructions (sequential or otherwise) specifying an action to be taken by the device. In addition, the term "computer" shall include any collection of computers that execute a set (or multiple sets) of instructions, either individually or collectively, for executing any one or more of the methods described herein.

[0188] In a further aspect, the computer system 600 may include a processing device 602, a volatile memory 604 (e.g., a random access memory (RAM)), a non-volatile memory 606 (e.g., a read only memory (ROM) or an electronically erasable programmable ROM (EEPROM)), and a data storage device 618 that may communicate with each other via a bus 608.

[0189] The processing device 602 may be provided by one or more processors, for example, a general-purpose processor (e.g., a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor that implements other types of instruction sets, or a microprocessor that implements a combination of types of instruction sets) or a special-purpose processor (e.g., an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0190] The computer system 600 may further include a network interface device 622 (e.g., coupled to the network 674). The computer system 600 may also include a video display unit 610 (e.g., LCD), an alphanumeric input device 612 (e.g., keyboard), a cursor control device 614 (e.g., mouse), and a signal generating device 620.

[0191] In some embodiments, the data storage device 618 may include a non-transitory computer-readable storage medium 624 (e.g., a non-transitory computer-readable storage medium) on which instructions 626 encoding any one or more of the methods or functions described herein may be stored, the instructions 626 including instructions encoding Figure 1 Components (e.g., prediction component 114, presentation component 115, model 190, etc.) and instructions for implementing the methods described herein.

[0192] During execution of the instructions 626 by the computer system 600 , the instructions 626 may also reside completely or partially in the volatile memory 604 and / or the processing device 602 , and thus, the volatile memory 604 and the processing device 602 may also constitute machine-readable storage media.

[0193] Although the computer-readable storage medium 624 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" shall include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) for storing one or more sets of executable instructions. The term "computer-readable storage medium" shall also include tangible media capable of storing or encoding a set of instructions to be executed by a computer to cause the computer to perform any one or more of the methods described herein. The term "computer-readable storage medium" shall include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0194] The methods, components, and features described herein may be implemented by separate hardware components, or may be integrated into the functionality of other hardware components (e.g., ASICs, FPGAs, DSPs, or similar devices). In addition, the methods, components, and features may be implemented by firmware modules or functional circuits within a hardware device. In addition, the methods, components, and features may be implemented using any combination of hardware devices and computer program components or using a computer program.

[0195] Unless otherwise specifically stated, terms (e.g., "receive," "perform," "provide," "obtain," "cause," "access," "determine," "add," "use," "train," "generate," "prepare," "train," "facilitate," or the like) refer to actions and processes performed or implemented by a computer system that manipulate and transform data represented as physical (electronic) values ​​in computer system buffers and memories into other data similarly represented as physical values ​​in computer system buffers and memories or other such information storage, transmission, or display devices. In addition, the terms "first," "second," "third," "fourth," etc., as used herein, are intended as marks used to distinguish different elements and may not have a sequence meaning according to their numerical names.

[0196] The examples described herein also relate to an apparatus for performing the methods described herein. This apparatus may be specially constructed to perform the methods described herein, or may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0197] The methods and illustrative examples described herein are not inherently related to any particular computer or other device. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized equipment to perform the methods described herein and / or each of their individual functions, routines, subroutines, or operations. Examples of the structures of various such systems are set forth in the description above.

[0198] The above description is intended to be illustrative rather than restrictive. Although the present disclosure has been described with reference to specific illustrative examples and embodiments, it will be understood that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the attached claims and the full range of equivalents given by the claims.

Claims

1. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a processing device to perform operations comprising: receiving, by a processing device, a first value associated with a first input parameter of a model, wherein the first input parameter is associated with a first processing condition of a semiconductor wafer processing process; receiving, by the processing device, a first plurality of values, wherein the first plurality of values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values, and wherein each of the first plurality of values ​​is associated with a second input parameter of the model, wherein the second input parameter is associated with a second processing condition of the semiconductor wafer processing process; providing, by the processing device, the first value and the first plurality of values ​​to the model; receiving, by the processing device, a first plurality of outputs from the model, wherein each of the first plurality of outputs is associated with the first value and one of the first plurality of values, and wherein each of the first plurality of outputs is associated with a first characteristic of one of a first plurality of simulated semiconductor wafers; as well as The first plurality of outputs are prepared, by the processing device, for presentation via a presentation component associated with a graphical user interface (GUI), wherein the presentation component includes two axes, a first axis of the two axes corresponding to a first property of the first feature and a second axis of the two axes corresponding to a second property of the first feature, and wherein preparing the first plurality of outputs for presentation includes facilitating generation of a graph for display in the presentation component, the graph indicating a value of the first property of the first feature and a value of the second property of the first feature associated with each of the first plurality of outputs.

2. The non-transitory machine-readable storage medium of claim 1, wherein the model comprises a machine learning model.

3. The non-transitory machine-readable storage medium of claim 2, the operations further comprising: Receive multiple sets of input values; receiving a plurality of metrology measurements, wherein each of the plurality of metrology measurements is associated with one of the plurality of sets of input values; as well as The model is trained using the multiple sets of input values ​​and the multiple metrology measurements to generate simulated metrology measurements of a simulated semiconductor wafer, wherein training the model includes providing the multiple sets of input values ​​to the model as training inputs and providing the multiple metrology measurements to the model as target outputs.

4. The non-transitory machine-readable storage medium of claim 1 , wherein the first characteristic comprises at least one of: Semiconductor wafer thickness; Semiconductor wafer resistivity; Semiconductor wafer thin film resistors; Semiconductor wafer refractive index; Semiconductor wafer extinction coefficient; or An indication of semiconductor wafer geometry.

5. The non-transitory machine-readable storage medium of claim 1 , wherein the first property of the first feature comprises a statistical metric associated with simulated metrology measurements at a plurality of locations of the simulated semiconductor wafer, wherein the statistical metric comprises at least one of: an average of the values ​​of the first feature; the median of the values ​​of the first feature; the standard deviation of the values ​​of the first characteristic; or The uniformity of the values ​​of the first characteristic.

6. The non-transitory machine-readable storage medium of claim 1, the operations further comprising: receiving, by the processing device, a second value associated with the second input parameter of the model; receiving, by the processing device, a second plurality of values, wherein the second plurality of values ​​ranges from a lowest value in the second plurality of values ​​to a highest value in the second plurality of values, and wherein each of the second plurality of values ​​is associated with the first input parameter of the model; providing, by the processing device, the second value and the second plurality of values ​​to the model; receiving, by the processing device, a second plurality of outputs from the model, wherein each of the second plurality of outputs is associated with the second value and one of the second plurality of values, and wherein each of the second plurality of outputs is associated with the first characteristic of one of a second plurality of simulated semiconductor wafers; as well as preparing, by the processing device, the second plurality of outputs for presentation via the presentation component of the GUI, wherein preparing the second plurality of outputs for presentation includes facilitating generation of a graphic for display in the presentation component, the graphic indicating the first property of the first feature and the second property of the first feature associated with each of the second plurality of outputs.

7. The non-transitory machine-readable storage medium of claim 1 , wherein preparing the first plurality of outputs for presentation by the processing device further comprises facilitating generation of a graphic for display in the presentation component, the graphic visually distinguishing an output of the first plurality of outputs associated with the highest value of the first plurality of values ​​from an output of the first plurality of outputs associated with the lowest value of the first plurality of values.

8. The non-transitory machine-readable storage medium of claim 1, the operations further comprising: receiving, by the processing device, a second value associated with the first input parameter of the model; providing, by the processing device, the second value and the first plurality of values ​​to the model; receiving, by the processing device, a second plurality of outputs from the model, wherein each of the second plurality of outputs is associated with the second value and one of the first plurality of values, and wherein each of the second plurality of outputs is associated with a first characteristic of one of a second plurality of simulated semiconductor wafers; as well as preparing, by the processing device, the second plurality of outputs for presentation via the presentation component, wherein preparing the second plurality of outputs for presentation includes facilitating generation of a graphic for display in the presentation component, the graphic indicating a value of the first property of the first feature and a value of the second property of the first feature associated with each of the second plurality of outputs.

9. A non-transitory machine-readable storage medium for storing instructions that, when executed, cause a processing device to perform operations comprising: receiving, by the processing device, a first plurality of outputs from a first model, wherein each of the first plurality of outputs is associated with a first input value and one of the first plurality of input values, and wherein each of the first plurality of outputs is associated with a first characteristic of one of a first plurality of simulated substrates; receiving, by the processing device, a second plurality of outputs from a second model, wherein each of the second plurality of outputs is associated with the first value and one of the first plurality of values, and wherein each of the second plurality of outputs is associated with a second characteristic of one of the first plurality of simulated substrates; as well as The first plurality of outputs and the second plurality of outputs are prepared by the processing device for presentation via a presentation component of a graphical user interface (GUI), wherein the presentation component includes two axes, a first axis of the two axes corresponding to a first property of the first feature and a second axis of the two axes corresponding to a second property of the second feature, and wherein preparing the first plurality of outputs and the second plurality of outputs for presentation includes facilitating generation of a graph for display in the presentation component, the graph indicating a value of the first property of the first feature and a value of the second property of the second feature for each simulated substrate in the first plurality of simulated substrates.

10. The non-transitory machine-readable storage medium of claim 9, wherein the first model comprises a first machine learning model, and wherein the second model comprises a second machine learning model.

11. The non-transitory machine-readable storage medium of claim 10, the operations further comprising: receiving a first plurality of sets of input values; receiving a second plurality of sets of input values; receiving a first plurality of metrology measurements, wherein each of the first plurality of metrology measurements is associated with one of the first plurality of sets of input values; receiving a second plurality of metrology measurements, wherein each of the second plurality of metrology measurements is associated with one of the second plurality of sets of input values; training the first model using the first plurality of sets of input values ​​and the first plurality of metrology measurements to generate first simulated metrology measurements of a simulated substrate, wherein training the first model comprises: providing the first plurality of sets of input values ​​to the first model as training inputs and providing the first plurality of metrology measurements to the first model as target outputs; as well as The second model is trained using the second plurality of sets of input values ​​and the second plurality of metrology measurements to generate second simulated metrology measurements of the simulated substrate, wherein training the second model includes providing the second plurality of sets of input values ​​to the second model as training inputs and providing the second plurality of metrology measurements to the second model as target outputs.

12. The non-transitory machine-readable storage medium of claim 9, wherein the first characteristic comprises at least one of: Substrate thickness; Substrate resistivity; Substrate thin film resistor; Refractive index of substrate; Substrate extinction coefficient; or Indication of substrate geometry.

13. The non-transitory machine-readable storage medium of claim 9, wherein the first property of the first feature comprises a statistical metric associated with a simulated metrology measurement at a plurality of locations of one of the first plurality of simulated substrates, wherein the statistical metric comprises at least one of: an average of the values ​​of the first feature; the median of the values ​​of the first feature; the standard deviation of the values ​​of the first characteristic; or The uniformity of the values ​​of the first characteristic.

14. The non-transitory machine-readable storage medium of claim 9, wherein preparing, by the processing device, the first plurality of outputs and the second plurality of outputs for presentation further comprises: Facilitating generation of a graphic for display in the presentation component that visually distinguishes presentation data points associated with a first one of the first plurality of simulated substrates associated with the highest one of the first plurality of values ​​from presentation data points associated with a second one of the first plurality of simulated substrates associated with the lowest one of the first plurality of values.

15. The non-transitory machine-readable storage medium of claim 9, further comprising: The first input value is received by the processing device, the first input value being associated with a first input parameter of: the first model; as well as the second model, wherein the first input parameter is associated with a first processing condition of a substrate processing process; as well as The first plurality of input values ​​is received by the processing device, wherein the first plurality of input values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values, and wherein each of the first plurality of values ​​is associated with a second input parameter of: the first model; and The second model, wherein the second input parameter is associated with a second processing condition of the substrate processing process.

16. The non-transitory machine-readable storage medium of claim 15, the operations further comprising: receiving, by the processing device, a second value associated with the first model and the second input parameter of the second model; receiving, by the processing device, a second plurality of values, wherein the second plurality of values ​​ranges from a lowest value in the second plurality of values ​​to a highest value in the second plurality of values, and wherein each of the second plurality of values ​​is associated with the first input parameter of the first model and the second model; providing, by the processing device, the second value and the second plurality of values ​​to the first model; providing, by the processing device, the second value and the second plurality of values ​​to the second model; receiving, by the processing device, a third plurality of outputs from the first model, wherein each of the third plurality of outputs is associated with the second value and one of the second plurality of values, and wherein each of the third plurality of outputs is associated with the first characteristic of one of a second plurality of simulated substrates; receiving, by the processing device, a fourth plurality of outputs from the second model, wherein each of the fourth plurality of outputs is associated with the second value and one of the second plurality of values, and wherein each of the fourth plurality of outputs is associated with the second characteristic of one of the second plurality of simulated substrates; The third plurality of outputs and the fourth plurality of outputs are prepared by the processing device for presentation via the presentation component of the GUI, wherein preparing the third plurality of outputs and the fourth plurality of outputs for presentation includes facilitating generation of a graphic for display in the presentation component, the graphic indicating a value of the first property of the first feature and a value of the second property of the second feature for each simulated substrate in the second plurality of simulated substrates.

17. The non-transitory machine-readable storage medium of claim 15, the operations further comprising: receiving, by the processing device, second values ​​associated with the first model and the first input parameter of the second model; providing the second value and the first plurality of values ​​to the first model and the second model by the processing device; receiving, by the processing device, a third plurality of outputs from the first model and a fourth plurality of outputs from the second model, wherein each of the third plurality of outputs and each of the fourth plurality of outputs is associated with the second value and one of the first plurality of values, and wherein each of the third plurality of outputs is associated with the first characteristic of one of a second plurality of simulated substrates, and wherein each of the fourth plurality of outputs is associated with the second characteristic of one of the second plurality of simulated substrates; as well as The third and fourth plurality of outputs are prepared by the processing device for presentation via the presentation component, wherein preparing the third and fourth plurality of outputs for presentation includes facilitating generation of a graphic for display in the presentation component, the graphic indicating an associated value of the first property of the first feature and an associated value of the second property of the second feature for each of the second plurality of simulated substrates.

18. A method comprising: receiving, by one or more processors, a first value associated with a first input parameter of a first model, wherein the first input parameter is associated with a process recipe for processing a substrate; receiving, by the one or more processors, a first plurality of values, wherein the first plurality of values ​​ranges from a lowest value in the first plurality of values ​​to a highest value in the first plurality of values, and wherein each of the first plurality of values ​​is associated with a second input parameter of the first model, wherein the second input parameter is associated with the process recipe; providing, by the one or more processors, the first value and the first plurality of values ​​to the first model; receiving, by the one or more processors, a first plurality of outputs from the first model, wherein each of the first plurality of outputs is associated with the first value and one of the first plurality of values, and wherein each of the first plurality of outputs is associated with a first characteristic of a simulated substrate; as well as The first plurality of outputs are prepared, by the one or more processors, for presentation via a presentation component of a graphical user interface (GUI), wherein the presentation component includes two independent axes, a first axis of the two independent axes corresponding to a first property of the first feature, and wherein preparing the first plurality of outputs for presentation includes facilitating generation of a graph in the presentation component that visually displays a relationship of the outputs of the first plurality of outputs to the first property of the first feature.

19. The method according to claim 18, further comprising: providing, by the one or more processors, the first value and the first plurality of values ​​to a second model; receiving, by the one or more processors, a second plurality of outputs from the second model, wherein each of the second plurality of outputs is associated with a second feature; as well as preparing, by the one or more processors, the second plurality of outputs for presentation via the presentation component of the GUI, wherein the second of the two independent axes corresponds to a second property of the second feature, and wherein preparing the first plurality of outputs and the second plurality of outputs for presentation includes facilitating generation of a graph in the presentation component that visually displays a relationship of the first plurality of outputs and the second plurality of outputs to the first property of the first feature and the second property of the second feature.

20. The method of claim 18, further comprising: receiving, by the one or more processors, a second value associated with the second input parameter of the first model; receiving, by the one or more processors, a second plurality of values, wherein the second plurality of values ​​ranges from a lowest value in the second plurality of values ​​to a highest value in the second plurality of values, and wherein each of the second plurality of values ​​is associated with the first input parameter of the first model; providing, by the one or more processors, the second value and the second plurality of values ​​to the first model; receiving, by the one or more processors, a second plurality of outputs from the first model, wherein each of the second plurality of outputs is associated with the second value and one of the second plurality of values, and wherein each of the second plurality of outputs is associated with the first feature; as well as preparing, by the one or more processors, the second plurality of outputs for presentation via the presentation component of the GUI, wherein preparing the second plurality of outputs for presentation includes facilitating generation of a graphic in the presentation component that visually displays the first plurality of outputs and a relationship of the second plurality of outputs to the first property of the first feature.

Images