Generation of synthetic microscopy images of fabricated devices

Abstract

The method includes receiving data indicating a plurality of dimensions of a manufactured device. The method further includes providing the data to a trained machine learning model. The method further includes receiving a composite microscopic image associated with the manufactured device from the trained machine learning model, the composite microscopic image generated in consideration of the first data. The method further includes at least one of (i) outputting the composite microscopic image to a display, or (ii) performing one or more operations on the composite microscopic image.

Description

[Technical field]

[0001] The present disclosure relates to methods associated with machine learning models used to evaluate manufactured devices, such as semiconductor devices, and more particularly, to methods for generating and utilizing synthetic microscopy image data using machine learning models associated with devices processed by processing equipment. [Background technology]

[0002] A product may be produced by performing one or more manufacturing processes using manufacturing equipment. For example, a semiconductor manufacturing equipment may be used to produce a substrate by a semiconductor manufacturing process. The product is produced with certain characteristics suitable for a target application. Machine learning models are used in various process control and predictive functions associated with the manufacturing equipment. The machine learning models are trained using data associated with the manufacturing equipment. Images of the product (e.g., a manufactured device) may be taken and the images may, for example, enhance understanding of device function, failures, performance, or may be used for metrology or inspection. Summary of the Invention

[0003] The following is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure, nor is it intended to limit the scope of particular embodiments of the disclosure or the claims. Its sole purpose is to present some ideas of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] In one aspect of the disclosure, a method includes receiving data indicating a plurality of dimensions of a manufactured device. The method further includes providing the data to a trained machine learning model. The method further includes receiving a composite microscopic image associated with the manufactured device from the trained machine learning model, the composite microscopic image having been generated in consideration of the first data. The method further includes at least one of (i) outputting the composite microscopic image to a display, or (ii) performing one or more operations on the composite microscopic image.

[0005] In another aspect of the present disclosure, a method includes receiving a plurality of microscopic images of a plurality of manufactured devices. The method further includes receiving data indicating a plurality of dimensions of the plurality of manufactured devices. The method further includes training a machine learning model, where training the machine learning model includes providing the first data to the machine learning model as a training input. The training further includes providing the plurality of microscopic images to the machine learning model as a target output.

[0006] In another aspect of the present disclosure, a non-transitory machine-readable storage medium is disclosed. The storage medium has instructions stored thereon that, when executed, cause a processing device to perform operations. The operations include receiving data indicating a plurality of dimensions of a manufactured device. The operations further include providing the data to a trained machine learning model. The operations further include receiving, from the trained machine learning model, a composite microscope image associated with the manufactured device. The composite microscope image was generated in consideration of the first data. The operations further include performing at least one of (i) outputting the composite microscope image to a display, or (ii) performing one or more operations on the composite microscope image.

[0007] In the figures of the accompanying drawings, the present disclosure is illustrated by way of example and not by way of limitation. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example system architecture according to some embodiments. [Figure 2A-B] FIG. 2 is a block diagram of an example dataset generator for generating a dataset for a model, according to some embodiments. [Diagram 3] FIG. 1 is a block diagram illustrating a system for generating output data according to some embodiments. [Figure 4A-B] 1 is a flow diagram of a method associated with generating and utilizing one or more machine learning models to generate synthetic image data, according to some embodiments. [Figure 4C] 1 is a flow diagram of a method for generating and using synthetic microscope image data, according to some embodiments. [Figure 5A-B] FIG. 1 is a block diagram of an example machine learning architecture for generating synthetic data, according to some embodiments. [Figure 5C-D] 1 is a flow diagram of a method associated with training and using a machine learning model to generate synthetic image data from cartoon imagery, according to some embodiments. [Figure 5E] 1 is an exemplary cartoon image of a manufactured device according to some embodiments. [Figure 6] FIG. 1 is a block diagram illustrating a computer system according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Described herein are techniques for generating composite microscope images of manufactured products. Manufacturing equipment is used to produce products such as substrates (e.g., wafers, semiconductors). The manufacturing equipment may include manufacturing or processing chambers to isolate the substrates from the environment. The properties of the produced substrates should meet target values ​​to facilitate a particular function. Manufacturing parameters are selected to produce substrates that meet the target property values. Many manufacturing parameters (e.g., hardware parameters, process parameters, etc.) contribute to the properties of the processed substrates. The manufacturing system may control the parameters by specifying set points for the property values, receiving data from sensors located in the manufacturing chambers, and adjusting the manufacturing equipment until the sensor readings match the set points. In some embodiments, trained machine learning models are utilized to improve the performance of the manufacturing equipment.

[0010] The machine learning model may be applied in several ways in connection with the process chamber and / or the manufacturing equipment. The machine learning model may receive as input sensor data measuring the value of a property in the process chamber. The machine learning model may be configured to predict a process result, for example, a metrology result of a finished product. The machine learning model may receive as input in-situ data (e.g., reflectance spectroscopy data of a semiconductor wafer during an etching process) and / or ex-situ data (e.g., metrology data associated with the workpiece or substrate). The machine learning model may be configured to predict and control the process result, for example, predict when an etching process is complete and send an instruction to the process chamber to stop the etching operation. Additionally or alternatively, the machine learning model may be configured to estimate one or more properties (e.g., critical dimensions) of a manufactured device on the substrate or workpiece. In some embodiments, the machine learning model may receive as input metrology data of the substrate (e.g., generated after the process is completed) and / or in-situ data of the substrate (e.g., generated during the manufacturing process). This metrology data and / or in situ data may generally be data collected with or without damaging the substrate. Many metrology measurements are destructive and damage the substrate, such as those performed by cross-sectional imaging tools including cross-sectional scanning electron microscopy (XSEM) or transmission electron microscopy (TEM). The machine learning model may be configured to generate as output an estimate or prediction of one or more measurements that would normally be generated after destruction of the substrate (e.g., critical dimension (CD) measurements), a synthetic microscopic image, a prediction of the root cause of a product anomaly (e.g., a processing defect), etc. There are many others, but these are a few representative examples of the use of machine learning in relation to manufacturing equipment.

[0011] In some embodiments, predictions of product metrology values ​​may be performed (e.g., using machine learning models, models based on physical phenomena, etc.). The metrology predictions may be performed taking into account target conditions in the process chamber, measurements of conditions near the substrate, in situ (e.g., during processing) measurements of the product during processing, etc. In some embodiments, the predicted metrology measurements may include some predictions of product dimensions, such as a prediction of the thickness at the center of the substrate.

[0012] In some embodiments, metrology data of the product may be measured. Performing metrology measurements may include obtaining one or more microscopic images of the product. The microscopic images may include images captured using optical techniques, images captured using electron-based techniques (e.g., scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, etc.), or other similar techniques. In some embodiments, metrology of internal structures may be measured. Measuring internal structures of the product may include cutting a cross section of the product and taking an image of the internal structure. Benefits of performing cross-sectional microscopy of the product include imaging internal structures that are not visible under normal circumstances, providing the ability to measure (e.g., from a microscopic image) dimensions of structures that would not otherwise be measured or predicted, ensuring that predictive models maintain threshold accuracy, etc. Metrology measurements (e.g., stand-alone metrology measurements, metrology measurements performed outside of a processing chamber, etc.) may be costly and / or time-consuming to perform. Metrology measurements (e.g., cross-sectional microscopy) may damage or destroy the processed product.

[0013] In some embodiments, microscopic images may be used in training the machine learning model. For example, to train the machine learning model, dimensions measured by performing cross-sectional measurements (e.g., by SEM) may be provided as target outputs for predicting internal dimensions of processed products. In some embodiments, large amounts of data are used to train the machine learning model. Hundreds, thousands, or even more substrates may be used in training the machine learning model. The cost of performing comprehensive metrology, such as the time spent, materials consumed, products destroyed and discarded, energy used in production, and processing equipment, may be multiplied by this large amount of data collected for training.

[0014] In some embodiments, the microscope images (e.g., contrast of the microscope images) may vary due to differences in sectioning, exposure, etc. resulting from inconsistent measurement techniques, different imaging technicians, or the like. It may be difficult to perform measurements using some images, which further increases the cost of generating sufficient data to train machine learning models.

[0015] The methods and devices of the present disclosure may address one or more of these shortcomings of conventional solutions. In some embodiments, a realistic microscopic image (e.g., top view, cross-sectional image, etc.) of the processed product is generated. This image may be utilized for measurement, visualization, input or training of machine learning or other models, etc.

[0016] A machine learning model may be used to generate synthetic microscopic images of the product. In some embodiments, a large amount of historical image data may be available. For example, a large amount of historical data related to related products may be available for use in training the synthetic microscopic image generation machine learning model, including related products of different designs, older generation products, etc.; related manufacturing processes; related process recipes; etc. In some embodiments, a generative adversarial network (GAN) generator model may be configured to generate synthetic data that matches the distribution of true data, e.g., synthetic data that is statistically and structurally similar to the true microscopic images.

[0017] In some embodiments, the microscope image generator may be configured to receive as input an indication of a dimension of the product. In some embodiments, a machine learning model associated with the process chamber (e.g., a machine learning model configured to receive as input sensor data during processing and generate as output one or more indications of predicted metrology of the product) may generate an output used by the composite image generator to generate the composite microscope image. In some embodiments, in-situ metrology measurements (e.g., spectral measurements of a substrate during processing) may be used as input to the composite image generator. In some embodiments, integrated measurements (e.g., measurements taken while the substrate is not being processed but is still in a vacuum) or in-line measurements (e.g., metrology measurements from equipment coupled to the process equipment but outside the vacuum) may be utilized by the composite microscope image generator to generate the composite metrology image. In some embodiments, stand-alone metrology measurements (e.g., measurements measured in a metrology facility, measurements that are less intrusive or destructive than the target image, such as measurements that do not destroy or involve cross-sectioning of the substrate, etc.) may be utilized as input to the image generator.

[0018] In some embodiments, data used to train the generator model may be labeled with one or more attributes. An attribute may include a label that identifies one or more features of the data. The attribute information may include data that indicates a process recipe associated with the product, e.g., data that indicates a sequence of recipe operations. The attribute information may include structural information of the product, e.g., instructions for rules regarding the order and / or placement of parts of the product during imaging. The attribute information may include data that indicates the product design. The attribute information may include target characteristics of the output data, e.g., color scale, contrast values ​​and / or brightness values ​​of the target composite image, etc. The attributes may include labels that identify the state of the manufacturing system, e.g., labels of defects present in processing equipment, an indication of the time since the manufacturing equipment was installed or maintained, etc.

[0019] In some embodiments, a synthetic image generator may receive as input a cartoon structure, e.g., a cartoon structure output by a cartoon generator model. The cartoon generator model may be configured to synthesize data indicative of measurements of the product (e.g., in situ measurements, output of a predictive machine learning model, etc.) and additional product information (e.g., product design, device type, structural layer order, relationships between structural dimensions, etc.) to generate a cartoon picture of the product or device. The cartoon generator model may or may not be a trained machine learning model, or may include a trained machine learning model, or may not include a trained machine learning model. The cartoon picture may be provided as input to a synthetic microscope image generator, which may be configured to generate a realistic synthetic image incorporating data from the cartoon, e.g., a realistic synthetic image replicating structural information from the cartoon.

[0020] In some embodiments, the generation of synthetic data may include the use of a generative adversarial network (GAN). A GAN is a type of unsupervised machine learning model (e.g., a model is provided with training inputs without providing a target output during a training operation). A basic GAN includes two parts: a generator and a discriminator. The generator generates synthetic data, e.g., synthetic microscope image data. The discriminator is then provided with synthetic data and true data, e.g., data collected by a cross-sectional SEM image of a product. The discriminator attempts to label data as true or synthetic, and the generator attempts to generate synthetic data that the discriminator cannot distinguish from true data. After the generator achieves a target efficiency (e.g., after reaching a threshold portion of the output that the discriminator does not classify as synthetic data), the generator may be used to generate synthetic data for use in other applications.

[0021] Aspects of the present disclosure provide technical advantages over conventional solutions. The techniques of the present disclosure allow for the generation of accurate (e.g., similar enough to reality to draw conclusions) synthetic microscope images of a product based on several measured or predicted metrology parameters and / or one or more design attributes of the manufactured device. In some embodiments, this approach allows for prediction of device dimensions that may be measured directly, typically at a large expense, such as by using stand-alone metrology equipment, destroying the device to generate cross-sectional images, etc. In some conventional systems, generating accurate (e.g., accurate beyond a threshold) predictions may include performing stand-alone metrology (e.g., destructive cross-sectional imaging) on ​​a large number of products. This large amount of data used to generate accurate predictions may be further exacerbated by changing chamber conditions (e.g., aging and drift, part replacement, maintenance, etc.), rare target events (e.g., defect or anomaly detection), etc. In conventional systems, a large number of processing runs may be performed to generate the data used to generate the predictions. This may result in a large amount of wasted material, a large amount of chamber downtime, consumed energy, etc. Metrology images (e.g., SEM microscopy images of a product) may have significant variance between images. Composite image data that is more consistent than the measured data may be generated, for example, by training a composite image generator using a selection of similar images, configuring the generator to generate composite images having chosen contrast values, brightness values, or other similar values, etc.

[0022] In some embodiments, the metrology images are used to train machine learning models to, for example, predict defects present in a manufacturing or processing system based on the microscopic images. The use of a synthetic microscopic image generator may enable rapid and low-cost generation of large amounts of synthetic microscopic data. Attribute data may be provided to the generator to generate synthetic data including a target set of characteristics that may otherwise be difficult to obtain. For example, the generator may be configured to generate image data related to products processed using equipment in which defects are occurring. Recording image data indicative of defects may include operating processing equipment under non-ideal conditions that may increase costs, increase processing time, increase material consumed, increase energy consumed, or reduce component life, or the like. Utilizing a generator to generate synthetic microscopic images may avoid these additional expenditures.

[0023] In one aspect of the disclosure, a method includes receiving data indicating a plurality of dimensions of a manufactured device. The method further includes providing the data to a trained machine learning model. The method further includes receiving a composite microscopic image associated with the manufactured device from the trained machine learning model, the composite microscopic image having been generated in consideration of the first data. The method further includes at least one of (i) outputting the composite microscopic image to a display, or (ii) performing one or more operations on the composite microscopic image.

[0024] In another aspect of the present disclosure, a method includes receiving a plurality of microscopic images of a plurality of manufactured devices. The method further includes receiving data indicating a plurality of dimensions of the plurality of manufactured devices. The method further includes training a machine learning model, where training the machine learning model includes providing the first data to the machine learning model as a training input. The training further includes providing the plurality of microscopic images to the machine learning model as a target output.

[0025] In another aspect of the present disclosure, a non-transitory machine-readable storage medium is disclosed. The storage medium has instructions stored thereon that, when executed, cause a processing device to perform operations. The operations include receiving data indicating a plurality of dimensions of a manufactured device. The operations further include providing the data to a trained machine learning model. The operations further include receiving, from the trained machine learning model, a composite microscope image associated with the manufactured device. The composite microscope image was generated in consideration of the first data. The operations further include performing at least one of (i) outputting the composite microscope image to a display, or (ii) performing one or more operations on the composite microscope image.

[0026] 1 is a block diagram illustrating an example system 100 (example system architecture) according to some embodiments. System 100 includes client devices 120, manufacturing equipment 124, sensors 126, measurement equipment 128, a prediction server 112, and a data store 140. Prediction server 112 may be part of a prediction system 110. Prediction system 110 may further include server machines 170 and 180.

[0027] The sensors 126 may provide sensor data 142 associated with the manufacturing equipment 124 (e.g., associated with the manufacturing equipment 124 producing a corresponding product, such as a substrate). The sensor data 142 may be used to ascertain the health of the equipment and / or the health of the product (e.g., product quality). The manufacturing equipment 124 may produce the product according to a recipe or by running for a period of time. In some embodiments, the sensor data 142 may include one or more values ​​of optical sensor data, spectral data, temperature (e.g., heating device temperature), spacing (SP), pressure, high frequency radio frequency (HFRF), radio frequency (RF) match voltage, RF match current, RF match capacitor position, electrostatic chuck (ESC) voltage, actuator position, current, flow rate, power, voltage, etc. The sensor data 142 may include historical sensor data 144 and current sensor data 146. The current sensor data 146 may relate to a product currently being processed, a product recently processed, a number of products recently processed, etc. The current sensor data 146 may be used as input to a trained machine learning model, such as input to a trained machine learning model to generate the predictive data 168. The historical sensor data 144 may include data stored in association with previously produced products. The historical sensor data 144 may be used to train a machine learning model, such as model 190, synthetic data generator 174, etc. The historical sensor data 144 and / or the current sensor data 146 may include attribute data, such as labels of manufacturing equipment ID or design, ID, type and / or location of sensors, status of manufacturing equipment, such as defects present, service life, etc.

[0028] The sensor data 142 may be related to or indicative of manufacturing parameters, such as hardware parameters (e.g., hardware settings or installed components, e.g., size, type, etc.) of the manufacturing equipment 124 or process parameters (e.g., heater settings, gas flows, etc.) of the manufacturing equipment 124. Alternatively or in addition, data related to some hardware parameters and / or process parameters may be stored as manufacturing parameters 150, which may include historical manufacturing parameters (e.g., associated with historical processing runs) and current manufacturing parameters. The manufacturing parameters 150 may indicate input settings for manufacturing devices (e.g., heater power, gas flow rates, etc.). The sensor data 142 and / or manufacturing parameters 150 may be provided while the manufacturing equipment 124 is performing a manufacturing process (e.g., equipment readings while processing a product). The sensor data 142 may vary from product to product (e.g., from substrate to substrate). The substrate may have properties (film thickness, film strain, etc.) measured by metrology equipment 128, such as at a stand-alone metrology facility. Metrology data 160 may be a component of data store 140. Metrology data 160 may include historical metrology data 164 (e.g., metrology data associated with previously processed products).

[0029] In some embodiments, metrology data 160 may be provided without the use of stand-alone metrology equipment, such as in situ metrology data (e.g., measurements or surrogates of measurements collected during processing), integrated metrology data (e.g., measurements or surrogates of measurements collected while the product is in the chamber or under vacuum other than during a processing operation), in-line metrology data (e.g., data collected after the substrate is removed from vacuum), etc. Metrology data 160 may include current metrology data 166 (e.g., metrology data associated with a product currently being processed or a recently processed product).

[0030] In some embodiments, the sensor data 142, the metrology data 160, or the manufacturing parameters 150 may be processed (e.g., by the client device 120 and / or the prediction server 112). The processing of the sensor data 142 may include generating features. In some embodiments, the features are patterns (e.g., slope, width, height, peaks, etc.) of the sensor data 142, the metrology data 160, and / or the manufacturing parameters 150, or combinations of values ​​from the sensor data 142, the metrology data, and / or the manufacturing parameters (e.g., power derived from voltage and current, etc.). The sensor data 142 may include features, which may be used by the prediction component 114 to perform signal processing and / or to obtain prediction data 168 for performing corrective actions.

[0031] Each instance (e.g., set) of sensor data 142 may correspond to a product (e.g., substrate), a set of manufacturing equipment, a type of substrate produced by the manufacturing equipment, or the like. Similarly, each instance of metrology data 160 and manufacturing parameters 150 may correspond to a product, a set of manufacturing equipment, a type of substrate produced by the manufacturing equipment, or the like. The data store may further store information relating sets of different data types, e.g., information indicating that the set of sensor data, the set of metrology data, and the set of manufacturing parameters all relate to the same product, the same manufacturing equipment, the same type of substrate, etc.

[0032] In some embodiments, a processing device (e.g., via a machine learning model) may be used to generate the synthetic data 162. The synthetic data may be processed in any of the manners described above with respect to other types of data, such as by generating features, combining values, concatenating data from particular recipes, chambers or substrates, etc. The synthetic data 162 may share features with some of the metrology data 160, such as including image data and resembling microscopic images (e.g., SEM and / or TEM images) included in the metrology data 160.

[0033] In some embodiments, the prediction system 110 may generate the predicted data 168 using supervised machine learning (e.g., the predicted data 168 includes output from a machine learning model trained using labeled data, such as sensor data labeled with measurement data (which may include, for example, synthetic microscope images generated according to embodiments described herein). In some embodiments, the prediction system 110 may generate the predicted data 168 using unsupervised machine learning (e.g., the predicted data 168 includes output from a machine learning model trained using unlabeled data, where the output may include clustering results, principal component analysis, anomaly detection, etc.). In some embodiments, the prediction system 110 may generate the predicted data 168 using semi-supervised learning (e.g., the training data may include a mixture of labeled and unlabeled data, etc.).

[0034] The client device 120, the manufacturing equipment 124, the sensor 126, the measurement equipment 128, the prediction server 112, the data store 140, the server machine 170, and the server machine 180 may be coupled to each other via the network 130 to generate predictive data 168 for performing corrective actions. In some embodiments, the network 130 may provide access to cloud-based services. Operations performed by the client device 120, the prediction system 110, the data store 140, etc. may be performed by a cloud-based virtual device.

[0035] In some embodiments, network 130 is a public network that provides client device 120 with access to prediction server 112, data store 140, and other public computing devices. In some embodiments, network 130 is a private network that provides client device 120 with access to manufacturing equipment 124, sensors 126, measurement equipment 128, data store 140, and other private computing devices. Network 130 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof.

[0036] The client devices 120 may include computing devices such as personal computers (PCs), laptops, mobile phones, smartphones, tablet computers, netbook computers, networked televisions ("smart TVs"), networked media players (e.g., Blu-ray players), set-top boxes, over-the-top (OTT) streaming devices, operator boxes, etc. The client devices 120 may include a corrective action component 122. The corrective action component 122 may receive user input of instructions (e.g., via a graphical user interface (GUI) displayed via the client devices 120) related to the manufacturing equipment 124. In some embodiments, the corrective action component 122 transmits the instructions to the prediction system 110, receives output (e.g., prediction data 168) from the prediction system 110, determines a corrective action based on the output, and causes the corrective action to be implemented. In some embodiments, the corrective action component 122 obtains sensor data 142 (e.g., current sensor data 146) related to the manufacturing equipment 124 (e.g., from a data store 140, etc.) and provides the sensor data 142 (e.g., current sensor data 146) related to the manufacturing equipment 124 to the predictive system 110.

[0037] In some embodiments, the corrective action component 122 may retrieve current metrology data 166 (e.g., predicted or surrogate metrology measurements of a product being processed) and provide that data to a synthetic data generator 174. The synthetic data generator 174 may generate as an output a predicted synthetic microscope image of the product associated with the current metrology data 166. The corrective action component 122 may store this synthetic image data in the data store 140. In some embodiments, the corrective action component 122 stores data in the data store 140 for use as input to machine learning or other models (e.g., current sensor data 146 provided to model 190, current metrology data 166 provided to synthetic data generator 174, etc.), and components of the prediction system 110 (e.g., prediction server 112, server machine 170) retrieve sensor data 142 from the data store 140. In some embodiments, the prediction server 112 may store the output of the trained model 190 (e.g., the prediction data 168 ) in the data store 140 , and the client device 120 may retrieve the output from the data store 140 .

[0038] In some embodiments, a corrective action component 122 receives corrective action instructions from the predictive system 110 and causes the corrective action to be implemented. Each client device 120 may include an operating system that enables a user to perform one or more of generating, reviewing, or editing data (e.g., instructions related to the manufacturing equipment 124, corrective actions related to the manufacturing equipment 124, etc.).

[0039] In some embodiments, the metrology data 160 (e.g., historical metrology data 164) corresponds to historical property data of a product (e.g., a product processed using manufacturing parameters associated with the historical sensor data 144 and the historical manufacturing parameters of manufacturing parameters 150), and the prediction data 168 relates to predicted property data (e.g., predicted property data of a product that will be produced or a product that has been produced under conditions recorded by the current sensor data 146 and / or the current manufacturing parameters). In some embodiments, the prediction data 168 is or includes predicted metrology data (e.g., virtual metrology data, virtual composite microscope images) of a product that will be produced or a product that has been produced according to conditions recorded as the current sensor data 146, the current measurement data, the current metrology data, and / or the current manufacturing parameters. In some embodiments, the predictive data 168 is or includes an indication of any anomalies (e.g., an abnormal product, an abnormal component, an abnormal manufacturing equipment 124, an abnormal energy usage, etc.) and optionally an indication of one or more causes of those anomalies. In some embodiments, the predictive data 168 is an indication of time change or drift of certain components of the manufacturing equipment 124, sensors 126, measurement equipment 128, and the like. In some embodiments, the predictive data 168 is an indication of the end of life of components of the manufacturing equipment 124, sensors 126, measurement equipment 128, or the like. In some embodiments, the predictive data 168 is an indication of the progress of a processing operation being performed, e.g., for use in process control.

[0040] Running a manufacturing process that results in a defective product can be costly in terms of time, energy, products, components, manufacturing equipment 124, costs of identifying defects and discarding the defective product, etc. By inputting sensor data 142 (e.g., manufacturing parameters that are being or will be used to manufacture the product) into the predictive system 110, receiving an output of predicted data 168, and performing corrective actions based on the predicted data 168, the system 100 can have the technical advantage of avoiding the costs of producing, identifying, and discarding defective products. By providing a number of predicted metrology measurements to a synthetic data generator 174 and receiving a synthetic microscope image as an output, products that are not predicted to meet performance thresholds can be identified, production can be stopped, corrective actions can be performed, alerts can be sent to users, recipes can be updated, etc.

[0041] Running a manufacturing process that results in a failure of a component of the manufacturing equipment 124 can be costly in downtime, damage to the product, damage to equipment, rush-ordering replacement components, etc. By inputting sensor data 142 (e.g., manufacturing parameters that are being or will be used to manufacture the product), metrology data, measurement data, etc., receiving output of predictive data 168, and performing corrective actions based on the predictive data 168 (e.g., predicted operational maintenance, e.g., replacing, treating, cleaning, etc., of components), the system 100 may have the technical advantage of avoiding one or more of the following costs: unexpected failure of components, unscheduled downtime, lost productivity, unexpected failure of equipment, product waste, or the like. Performance of components, such as the performance of the manufacturing equipment 124, sensors 126, metrology equipment 128, and the like, may be monitored over time to provide an indication of deteriorating components.

[0042] The manufacturing parameters may be less than optimal for producing a product, and producing the product may have costly consequences such as increased resource (e.g., energy, coolant, gas, etc.) consumption, increased time to produce the product, increased component failures, increased amount of defective products, etc. By inputting indications of the measurements into the synthetic data generator 174, receiving the output of the synthetic data 162, and taking corrective action (e.g., based on the synthetic data 162) to update the manufacturing parameters (e.g., set optimal manufacturing parameters), the system 100 may have the technical advantage of using optimal manufacturing parameters (e.g., hardware parameters, process parameters, optimal design) to avoid the costly consequences of less than optimal manufacturing parameters.

[0043] The corrective action may be related to one or more of Computational Process Control (CPC), Statistical Process Control (SPC) (e.g., SPC on electronic components to determine a process under control, SPC to predict the useful life of a component, SPC for comparison to a 3 sigma graph, etc.), Advanced Process Control (APC), model-based process control, preventive operational maintenance, design optimization, manufacturing parameter updates, manufacturing recipe updates, feedback control, machine learning corrections, or the like.

[0044] In some embodiments, the corrective action includes issuing an alert (e.g., an alert to stop or not run a manufacturing process if the predictive data 168 indicates a predicted anomaly, such as an anomaly in the product, component, or manufacturing equipment 124). In some embodiments, a machine learning model is trained to monitor the progress of a processing run (e.g., monitor in situ sensor data to predict whether the manufacturing process has reached completion). In some embodiments, the machine learning model may send instructions to terminate a processing run when the model determines that the process is complete. In some embodiments, the corrective action includes providing feedback control (e.g., feedback control that modifies a manufacturing parameter in response to the predictive data 168 indicating a predicted anomaly). In some embodiments, performing the corrective action includes causing an update to one or more manufacturing parameters to be performed. In some embodiments, performing the corrective action may include retraining a machine learning model associated with the manufacturing equipment 124. In some embodiments, performing the corrective action may include training a new machine learning model associated with the manufacturing equipment 124.

[0045] The manufacturing parameters 150 may include hardware parameters (e.g., information indicative of which components are installed on the manufacturing equipment 124, information indicative of component replacement, information indicative of the age of a component, information indicative of a software version or update, etc.) and / or process parameters (e.g., temperature, pressure, flow rate, rate, current, voltage, gas flow rate, lift speed, etc.). In some embodiments, the corrective action includes performing preventive operational maintenance (e.g., replacing, treating, cleaning, etc., components of the manufacturing equipment 124). In some embodiments, the corrective action includes performing design optimization (e.g., updating manufacturing parameters, manufacturing processes, manufacturing equipment 124, etc. to optimize the product). In some embodiments, the corrective action includes updating a recipe (e.g., changing when a manufacturing subsystem goes into an idle or active mode, changing set points for various characteristic values, etc.).

[0046] Prediction server 112, server machine 170, and server machine 180 may each include one or more computing devices, such as a rack mount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a graphics processing unit (GPU), an accelerator application specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc. Operations of prediction server 112, server machine 170, server machine 180, data store 140, etc. may be performed by a cloud computing service, a cloud data storage service, etc.

[0047] The prediction server 112 may include a prediction component 114. In some embodiments, the prediction component 114 may receive current sensor data 146 and / or current manufacturing parameters (e.g., received from the client device 120 and retrieved from the data store 140) and generate an output (e.g., predicted data 168) based on the current data for performing corrective actions associated with the manufacturing equipment 124. In some embodiments, the prediction data 168 may include one or more predicted dimensional measurements of the processed product. In some embodiments, the prediction data 168 may be provided to a synthetic data generator 174 to generate synthetic microscope image data, e.g., synthetic data 162. In some embodiments, the prediction component 114 may use one or more trained machine learning models 190 to determine an output for performing corrective actions based on the current data.

[0048] Manufacturing equipment 124 may be associated with one or more machine learning models, such as model 190. The machine learning models associated with manufacturing equipment 124 may perform many tasks including process control, classification, performance prediction, etc. Model 190 may be trained using data associated with manufacturing equipment 124 or data associated with products processed by manufacturing equipment 124, such as sensor data 142 (e.g., collected by sensors 126), manufacturing parameters 150 (e.g., associated with process control of manufacturing equipment 124), metrology data 160 (e.g., generated by metrology equipment 128), etc.

[0049] One type of machine learning model that may be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network. An artificial neural network typically includes a feature representation component with a classifier or recurrent layer that maps features to a desired output space. For example, a convolutional neural network (CNN) hosts multiple layers of convolutional filters. Pooling may be performed and nonlinearities may be addressed at the lower layers, and above the lower layers, a multi-layer perceptron is typically added to map the upper layer features extracted by the convolutional layers to a decision (e.g., a classification output).

[0050] Recurrent Neural Networks (RNNs) are another type of machine learning model. Recurrent Neural Network models are designed to interpret a sequence of inputs where the inputs are intrinsically related to each other, e.g., time trace data, sequential data, etc. The output of a perceptron in an RNN is fed back as input to that perceptron to generate the next output.

[0051] Deep learning is a type of machine learning algorithm that uses a cascade of multiple layers of nonlinear processing units to perform feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks may learn in a supervised (e.g., classification) and / or unsupervised manner (e.g., pattern analysis). Deep neural networks include a hierarchy of layers, with different layers learning different levels of representations corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in an image recognition application, the raw input may be a matrix of pixels, a first representation layer may take the pixels and code the edges, a second layer may construct and code the configuration of edges, a third layer may code higher-order shapes (e.g., teeth, lips, gums, etc.), and a fourth layer may recognize the scanning mission. Notably, the deep learning process can learn on its own which features to optimally place at which level. The "deep" in "deep learning" refers to the number of layers through which data is transformed. More precisely, deep learning systems have significant credit assignment path (CAP) depth. A CAP is a chain of transformations from inputs to outputs. A CAP describes the potentially causal connections between inputs and outputs. For feedforward neural networks, the CAP depth may be the depth of the network or the number of hidden layers + 1. For recurrent neural networks, where signals may propagate through layers more than once, the CAP depth is potentially infinite.

[0052] In some embodiments, the prediction component 114 and / or the model 190 includes a generator of a generative adversarial network (GAN) trained to generate synthetic microscopic images. The machine learning model may be trained by including this target model in the generative adversarial network (GAN). The GAN juxtaposes (e.g., puts in an adversarial configuration) two (or more) machine learning models to facilitate training of the model. A simple GAN includes a generator and a discriminator. The generator is configured to generate synthetic data that resembles data from a set of true data. The discriminator is configured to classify the output from the generator as true data or synthetic data. The weights and biases of the model are adjusted to improve the data generation of the generator and the classification of the discriminator. The GAN may be configured to transform the image from one space to another, for example, to replace one characteristic of the input image with another characteristic. In embodiments, an image-to-image (e.g., pix2pix) GAN may be configured to convert a primitive drawing or cartoon into a realistic image. In an embodiment, a discriminator of the image-to-image GAN may be provided with an original (e.g., true) image and / or a synthetic image generated by a generator, and the discriminator of the image-to-image GAN may classify which of the two is true and which is synthetic.

[0053] In some embodiments, the prediction component 114 receives the current sensor data 146, the current metrology data 166, and / or the current manufacturing parameters 154, performs signal processing to decompose the current data into a set of current data, provides the set of current data as inputs to the trained model 190, and obtains an output from the trained model 190 indicative of the predicted data 168. In some embodiments, the prediction component 114 receives metrology data of the substrate (e.g., predicted metrology data based on the sensor data) and provides the metrology data to the trained model 190. For example, the current sensor data 146 may include sensor data indicative of a metrology value (e.g., a geometry value) of the substrate. The model 190 may be configured to accept data indicative of the substrate metrology value and generate as an output a predicted composite microscope image. In some embodiments, the prediction data is indicative of metrology data (e.g., a prediction of substrate quality). In some embodiments, the prediction data is indicative of the health of a component. In some embodiments, the prediction data is indicative of a progress of a process (e.g., utilized to terminate a processing operation).

[0054] In some embodiments, the various models discussed in connection with model 190 (e.g., supervised machine learning models, unsupervised machine learning models, etc.) may be combined into one model (e.g., an ensemble model) or may be separate models.

[0055] Data may be passed bidirectionally between several separate models included in models 190, synthetic data generator 174, and prediction component 114. In some embodiments, some or all of these operations may instead be performed by different devices, such as client device 120, server machine 170, server machine 180, etc. Those skilled in the art will understand that variations in data flow, which components perform which processes, which data is provided to which models, and the like, are within the scope of this disclosure.

[0056] The data store 140 may be a memory (e.g., random access memory), a drive (e.g., hard drive, flash drive), a database system, a cloud-accessible memory system, or another type of component or device capable of storing data. The data store 140 may include multiple storage components (e.g., multiple drives or multiple databases) that may be across multiple computing devices (e.g., multiple server computers). The data store 140 may store sensor data 142, manufacturing parameters 150, metrology data 160, synthetic data 162, and forecast data 168.

[0057] The sensor data 142 may include historical sensor data 144 and current sensor data 146. The sensor data may include time tracking of sensor data throughout the duration of the manufacturing process, association of data with physical sensors, pre-processed data such as averages and composite data, and data indicating sensor performance over time (i.e., across many manufacturing processes). The manufacturing parameters 150 and metrology data 160 may include similar features, such as historical metrology data 164 and current metrology data 166. The historical sensor data 144, historical metrology data 166, and historical manufacturing parameters may be historical data (e.g., at least a portion of these data may be used to train the model 190). The current sensor data 146, current metrology data 166 may be current data (e.g., at least a portion of the historical data that is input to the learning model 190) for which predictive data 168 is generated (e.g., to perform corrective actions). The synthetic data 162 may include synthetic data that resembles a synthetic image generated by the synthetic data generator 174, such as a scanning electron microscope (SEM) image, a transmission electron microscope (TEM) image, or other similar image type.

[0058] In some embodiments, the prediction system 110 further includes a server machine 170 and a server machine 180. The server machine 170 includes a dataset generator 172 that can generate datasets (e.g., a set of data inputs and a set of target outputs) for training, validating, and / or testing a model 190, including one or more machine learning models. Some operations of the dataset generator 172 are described in detail below with respect to FIGS. 2A-B and 4A. In some embodiments, the dataset generator 172 may divide the historical data (e.g., the historical sensor data 144, the historical manufacturing parameters, the historical metrology data 164) into a training set (e.g., 60 percent of the historical data), a validation set (e.g., 20 percent of the historical data), and a testing set (e.g., 20 percent of the historical data).

[0059] In some embodiments, the prediction system 110 generates (e.g., via the prediction component 114) multiple sets of features. For example, a first set of features may correspond to a first set of types of sensor data (e.g., from a first set of sensors, a first combination of values ​​from the first set of sensors, a first pattern of values ​​from the first set of sensors) corresponding to each of the data sets (e.g., a training set, a validation set, and a test set), and a second set of features may correspond to a second set of types of sensor data (e.g., from a second set of sensors different from the first set of sensors, a second combination of values ​​different from the first combination, a second pattern different from the first pattern) corresponding to each of the data sets.

[0060] The server machine 170 may include a synthetic data generator 174. The synthetic data generator 174 may include one or more trained machine learning models, models based on physical phenomena, models based on rules, or the like. In an embodiment, the synthetic data generator 174 is or includes a generator of a GAN, such as a generator of an image-to-image GAN. The synthetic data generator 174 may be trained to generate synthetic microscope images from input data. In an embodiment, the input data is or includes a simple line or cartoon drawing of a cross-sectional side view of a device or structure. The synthetic data generator 174 may be trained using metrology data 160, such as metrology data 160 collected by metrology instrument 128. The synthetic data generator 174 may be configured to generate synthetic data, such as synthetic microscope image data. The synthetic data 162 may resemble historical metrology data 164. The synthetic data 162 may be used to train a machine learning model 190 to generate predictive data 168, for example, for performing corrective actions. The dataset generator 172 may combine the measured data 160 and the synthetic data 162 to generate datasets, such as training, testing, and validation datasets.

[0061] In some embodiments, the synthetic data generator 174 may be configured to generate a synthetic microscope image of the manufactured product. During a training operation, the synthetic data generator 174 may be provided with a true microscope image (e.g., an image of a manufactured device collected by a microscope system such as a SEM or TEM system), and the synthetic data generator 174 may be configured to generate a synthetic image that resembles the true image. In some embodiments, during training, the synthetic data generator 174 may be provided with an indication of a dimension of the substrate. For example, during a training operation, an output of an in-situ metrology system may be provided to the synthetic data generator 174. The synthetic data generator 174 may be configured to accept one or more indications of metrology (e.g., a list of measurements from a metrology system, a list of predicted measurements from a model configured to predict substrate dimensions, etc.) and generate one or more synthetic microscope images as an output. In some embodiments, the synthetic data generator 174 may be provided with an indication of the metrology of the manufactured product in the form of a cartoon image of the product. In some embodiments, the synthetic data generator 174 may include a cartoon generator.

[0062] In some embodiments, the output from the synthetic data generator 174 may be utilized for performance analysis (e.g., predicted substrate performance, analysis of substrate processing system performance, etc.). In some embodiments, the output from the synthetic data generator 174 may be utilized to select substrates for further investigation, e.g., used to flag potentially defective substrates for further metrology. In some embodiments, the output from the synthetic data generator 174 may be provided to another model, e.g., another machine learning model. This second model may be configured to accept one or more metrology images associated with the substrate. This second model may be configured to perform defect determination, performance estimation, corrective action recommendations, etc., based on the microscopic image (e.g., the synthetic microscopic image generated by the synthetic data generator 174).

[0063] In some embodiments, historical data is provided to the machine learning model 190 as training data. In some embodiments, synthetic data 162 is provided to the machine learning model 190 as training data. In some embodiments, this historical and / or synthetic sensor data may be or include microscopic image data. The type of data provided will vary depending on the application of the machine learning model. For example, the machine learning model may be trained by providing the model with historical sensor data 144 as training inputs and corresponding metrology data 160 as target outputs. In some embodiments, a large amount of data may be used to train the model 190. For example, sensor and metrology data for hundreds of substrates may be used. In some embodiments, a fairly small amount of data is available to train the model 190, for example, to train the model 190 to recognize rare events such as equipment failures, or to train the model 190 to generate predictions for newly seasoned or maintained chambers. The synthetic data 162 may be generated by the synthetic data generator 174 to augment the available real data (e.g., data generated by the metrology equipment 128) in the training model 190.

[0064] The server machine 180 includes a training engine 182, a validation engine 184, a selection engine 185, and / or a test engine 186. The engines (e.g., training engine 182, validation engine 184, selection engine 185, and test engine 186) may refer to hardware (e.g., circuitry, dedicated logic circuitry, programmable logic circuitry, microcode, processing devices, etc.), software (e.g., instructions running on a processing device, general-purpose computer system, or dedicated machine), firmware, microcode, or a combination thereof. The training engine 182 may be capable of training the model 190 and / or the synthetic data generator 174 using one or more sets of features associated with the training set from the dataset generator 172. The training engine 182 may generate multiple trained models 190, where each trained model 190 corresponds to a different set of features of the training set (e.g., sensor data from a different set of sensors). For example, a first trained model may be trained using all features (e.g., X1-X5), a second trained model may be trained using a first subset of features (e.g., X1, X2, X4), and a third trained model may be trained using a second subset of features (e.g., X1, X3, X4, and X5), where the second subset of features may overlap with the first subset of features. The dataset generator 172 may receive the output of the trained models (e.g., synthetic data 162 from the synthetic data generator 174), assemble the data into training, validation, and testing datasets, and use those datasets to train a second model (e.g., a machine learning model configured to output predictive data, corrective actions, etc.).

[0065] The validation engine 184 may be capable of validating the trained models 190 using a corresponding set of validation set features from the dataset generator 172. For example, a first trained machine learning model 190 trained using a first set of training set features may be validated using a first set of validation set features. The validation engine 184 may determine the accuracy of each of the trained models 190 based on the corresponding set of validation set features. The validation engine 184 may discard trained models 190 that have an accuracy that does not meet a threshold accuracy. In some embodiments, the selection engine 185 may be capable of selecting one or more trained models 190 that have an accuracy that meets a threshold accuracy. In some embodiments, the selection engine 185 may be capable of selecting the trained model 190 with the highest accuracy among the trained models 190.

[0066] The testing engine 186 may be capable of testing the trained models 190 using a corresponding set of test set features from the dataset generator 172. For example, a first trained machine learning model 190 trained using a first set of training set features may be tested using a first set of test set features. The testing engine 186 may determine the trained model 190 that has the highest accuracy among all of the trained models based on the test set.

[0067] In the case of a machine learning model, the model 190 may refer to a model artifact generated by the training engine 182 using a training set that includes data inputs and corresponding target outputs (correct answers for each corresponding training input). In an embodiment, the training set includes synthetic microscope images generated by the synthetic data generator 174. Patterns in the data set that map the data inputs to the target outputs (correct answers) can be found, and the machine learning model 190 is provided with a mapping that captures these patterns. The machine learning model 190 may use one or more of the following: support vector machines (SVM), radial basis functions (RBF), clustering, supervised machine learning, semi-supervised machine learning, unsupervised machine learning, k-nearest neighbor algorithms (k-NN), linear regression, random forests, neural networks (e.g., artificial neural networks, recurrent neural networks), and the like. The synthetic data generator 174 may include one or more machine learning models, which may include one or more of these same types of models (e.g., artificial neural networks).

[0068] In some embodiments, one or more machine learning models 190 may be trained using historical data (e.g., historical sensor data 144). In some embodiments, models 190 may have been trained using synthetic data 162 or a combination of historical and synthetic data.

[0069] In some embodiments, the synthetic data generator 174 may be trained using historical data. For example, the synthetic data generator 174 may be trained to generate synthetic data 162 using historical measurement data 164. In some embodiments, the synthetic data generator 174 may include a generative adversarial network (GAN). A GAN includes at least a generator and a discriminator. The generator attempts to generate data (e.g., time trace sensor data) similar to the input data (e.g., true sensor data). The discriminator attempts to distinguish the true data from the synthetic data. Training a GAN includes the generator becoming skilled at generating data similar to the true sensor data and the discriminator becoming skilled at distinguishing the true data from the synthetic data. A trained GAN includes a generator configured to generate synthetic data that includes many features of the true data used to train the GAN. In some embodiments, the input data may be labeled with one or more attributes, such as information about a tool, sensor, product, or product design associated with the input data. In some embodiments, the generator may be configured to generate synthetic data having a set of attributes, such as synthetic data related to target processing operations, target processing equipment defects, target metrology configurations (e.g., contrast, cross-sectional or top-view images, brightness, etc.), or the like.

[0070] Generating and utilizing the synthetic data 162 has significant technical advantages over other methods. In some embodiments, measurements of one or more dimensions of a device are generated. Some measurements may be made using inexpensive non-destructive methods. For example, in-situ reflectance spectrum data may correlate with etch depth, as well as integrated or in-line metrology may provide one or more dimensions of a produced device. In some embodiments, target dimension measurements, such as measurements of dimensions of internal structures of a product or device, are not easily obtained. Measuring target dimensions directly may be costly in terms of time, for example because the measurements may be performed in stand-alone metrology equipment, or may be costly in terms of materials, for example, destructive to the product. Measurements that are difficult to obtain by inexpensive conventional methods may be obtained by performing measurements of the synthetic data 162, for example, by calculating measurements of the product from one or more synthetic microscope images. The synthetic data 162 generated by the synthetic data generator 174 may have high accuracy. Thus, the synthetic data generator 174 may generate synthetic data that provides information about the internal structures of the substrate that can only be obtained by destructive means.

[0071] The synthetic data 162 may further provide technical advantages when used to train additional models, such as model 190. In some embodiments, large amounts of data (e.g., data from hundreds of substrates) may be used to train machine learning models, models based on physical phenomena, and the like. Generating such amounts of data may be costly, for example, in terms of consumed raw materials, process gases, energy, time, equipment wear, etc. Synthetic data generator 174 may be used to quickly and inexpensively generate large amounts of image data that may be used to train models.

[0072] In some embodiments, true microscopic images may vary in unpredictable or adverse ways. For example, different images may have different characteristics, such as different contrast, brightness, clarity, etc. This may be due to operator error, microscopy procedures, etc. The synthetic data 162 may be tuned to exhibit desired characteristics. In some embodiments, training data may be selected (e.g., by a user, an algorithm) to train the synthetic data generator 174 to exhibit desired characteristics. In some embodiments, the generator may be provided with attribute data describing characteristics of the images, and the generator may be configured to generate images according to a target set of attributes and / or characteristics.

[0073] The prediction component 114 may provide current data to the model 190 and may execute the model 190 on the inputs to obtain one or more outputs. For example, the prediction component 114 may provide current sensor data 146 to the model 190 and may execute the model 190 on the inputs to obtain one or more outputs. The prediction component 114 may be able to determine (e.g., extract) predicted data 168 from the output of the model 190. The prediction component 114 may determine (e.g., extract) confidence data from the output that indicates a confidence that the predicted data 168 is an accurate predictor of a process associated with the current sensor data 146 and / or input data for a product produced or to be produced at current manufacturing parameters using the manufacturing equipment 124. The prediction component 114 or the corrective action component 122 may use this confidence data to determine whether to cause a corrective action associated with the manufacturing equipment 124 to be performed based on the predicted data 168.

[0074] The confidence data may include or indicate a confidence that the prediction data 168 is an accurate prediction for a product or component associated with at least a portion of the input data. In one example, the confidence is a real number between 0 and 1, inclusive, where 0 indicates no confidence that the prediction data 168 is an accurate prediction for a product processed according to the input data or a component health of a component of the manufacturing equipment 124, and 1 indicates absolute confidence that the prediction data 168 accurately predicts a characteristic of a product processed according to the input data or a component health of a component of the manufacturing equipment 124. In response to the confidence data indicating a confidence below a threshold level for a predetermined number of instances (e.g., a percentage of instances, a frequency of instances, a total number of instances, etc.), the prediction component 114 may retrain the trained model 190 (e.g., based on the current sensor data 146, current manufacturing parameters, etc.). In some embodiments, retraining may include generating one or more data sets (eg, via the data set generator 172) utilizing historical and / or synthetic data.

[0075] For purposes of illustration and not limitation, aspects of the disclosure describe training one or more machine learning models 190 using historical data (e.g., historical sensor data 144, historical manufacturing parameters) and synthetic data 162, and inputting current data (e.g., current sensor data 146, current manufacturing parameters 154, and current metrology data 164) into the one or more trained machine learning models to determine predicted data 168. In other embodiments, heuristic models, physics-based models, or rule-based models are used (e.g., without the use of trained machine learning models) to determine predicted data 168. In some embodiments, such models may be trained using historical and / or synthetic data. In some embodiments, the models may be retrained utilizing a combination of true historical data and synthetic data. The predictive component 114 may monitor historical sensor data 144, historical manufacturing parameters, and metrology data 160. Any of the information described with respect to data inputs 210A-B of FIGS. 2A-B may be monitored by the heuristic models, physics-based models, or rule-based models, or may be otherwise used.

[0076] In some embodiments, the functionality of client device 120, prediction server 112, server machine 170, and server machine 180 may be provided by fewer machines. For example, in some embodiments, server machines 170 and 180 may be combined into a single machine, and in other embodiments, server machine 170, server machine 180, and prediction server 112 may be combined into a single machine. In some embodiments, client device 120 and prediction server 112 may be combined into a single machine. In some embodiments, the functionality of client device 120, prediction server 112, server machine 170, server machine 180, and data store 140 may be performed by a cloud-based service.

[0077] In general, functions described as being performed by client device 120, prediction server 112, server machine 170, and server machine 180 in one embodiment may be performed on prediction server 112 in other embodiments, where appropriate. Additionally, functions attributed to particular components may be performed by different or multiple components operating together. For example, in some embodiments, prediction server 112 may determine corrective actions based on prediction data 168. In another example, client device 120 may determine prediction data 168 based on output from a trained machine learning model.

[0078] Additionally, the functionality of a particular component may be performed by different or multiple components working together. One or more of prediction server 112, server machine 170, or server machine 180 may be accessed as a service offered to other systems or devices through an appropriate application programming interface (API).

[0079] In embodiments, a "user" may be expressed as a single individual. However, other embodiments of the present disclosure encompass a "user" being an entity controlled by multiple users and / or automated sources. For example, a collection of individual users combined into a group of administrators may be considered a "user."

[0080] Embodiments of the present disclosure may be applied to data quality assessment, feature enhancement, model evaluation, virtual metrology (VM), predictive maintenance (PdM), marginal optimization, process control or the like.

[0081] 2A-B show block diagrams of example dataset generators 272A-B (e.g., dataset generator 172 of FIG. 1 ) that generate datasets for training, testing, validation, etc. of a model (e.g., model 190 of FIG. 1 ), according to some embodiments. Each dataset generator 272 may be part of server machine 170 of FIG. 1 . In some embodiments, several machine learning models associated with manufacturing equipment 124 may be trained, used, and maintained (e.g., within a manufacturing facility). Each machine learning model may be associated with one dataset generator 272, multiple machine learning models may share one dataset generator 272, etc.

[0082] 2A illustrates a system 200A that includes a dataset generator 272A for generating a dataset for one or more supervised models (e.g., synthetic data generator 174 of FIG. 1). The dataset generator 272A may generate the dataset (e.g., data inputs 210A, target outputs 220A) using historical data and / or design rule data. In some embodiments, a dataset generator similar to the dataset generator 272A may be utilized to train an unsupervised machine learning model, e.g., the target output 220A may not be generated by the dataset generator 272A.

[0083] The dataset generator 272A may generate a dataset for training, testing, and validating a model. In some embodiments, the dataset generator 272A may generate a dataset for a machine learning model. In some embodiments, the dataset generator 272A may generate a dataset for training, testing, and / or validating a generator model configured to generate synthetic microscope image data. The machine learning model is provided with a set of historical metrology data 264A and / or a set of design rule data 265A as data inputs 210A. The machine learning model may be configured to accept metrology and design rule data as input data and generate synthetic microscope image data as output.

[0084] In some embodiments, the dataset generator 272A may be configured to generate a dataset for training, testing, validation, etc. of a generative adversarial network (GAN). In some embodiments, the dataset generator 272A may be used to generate a dataset for an image-to-image (e.g., pix-2-pix) GAN. In some embodiments, design rule data may be provided to train the machine learning model. In some embodiments, metrology data and design rule data may be provided to another model, which generates and provides to the machine learning model a composite of information from both metrology data and design rule data. In some embodiments, the composite model may generate a predicted cartoon image of a manufactured product, for example, a predicted cartoon image of a manufactured product incorporating known metrology measurements and various design rules. With reference to FIG. 2A, assume that the dataset generator provides both metrology data and design rule data to the model. For example, the model may be considered an ensemble model incorporating a cartoon generator and an image-to-image GAN.

[0085] The dataset generator 272A may be used to generate data for any type of machine learning model that takes metrology and / or design rule data as input. The dataset generator 272A may be used to generate data for a machine learning model that generates predicted metrology data for a substrate. The dataset generator 272A may be used to generate data for a machine learning model configured to provide process control instructions. The dataset generator 272A may be used to generate data for a machine learning model configured to identify product anomalies and / or process equipment defects.

[0086] In some embodiments, a dataset generator 272A generates a dataset (e.g., a training set, a validation set, a test set) that includes one or more data inputs 210A (e.g., training inputs, validation inputs, test inputs). The data inputs 210A may be provided to the training engine 182, the validation engine 184, or the test engine 186. This dataset may be used to train, validate, or test a model (e.g., the synthetic data generator 174 of FIG. 1).

[0087] In some embodiments, the data input 210A may include one or more sets of data. As an example, the system 200A may generate a set of sensor data that may include one or more of sensor data from one or more types of sensors, combinations of sensor data from one or more types of sensors, patterns from sensor data from one or more types of sensors, and / or composite versions thereof.

[0088] In some embodiments, data input 210A may include one or more sets of data. As an example, system 200A may generate a set of historical metrology data that may include one or more of metrology data for a set of dimensions of a device (e.g., including height and width of the device, but not including optical data or surface roughness, etc.), metrology data derived from one or more types of sensors, a combination of metrology data derived from one or more types of sensors, a pattern from metrology data, etc. The sets of data input 210A may include data describing different manufacturing aspects, e.g., a combination of metrology data and sensor data, a combination of metrology data and manufacturing parameters, a combination of some metrology data, some manufacturing parameter data and some sensor data, etc.

[0089] In some embodiments, the data set generator 272A may generate a first data input corresponding to the first set of historical metrology data 264A and / or the first set of design rule data 265A for training, validating or testing a first machine learning model. The data set generator 272A may generate a second data input corresponding to the second set of historical metrology data 264B and / or the second set of design rule data 265B for training, validating or testing a second machine learning model.

[0090] In some embodiments, the dataset generator 272A generates a dataset (e.g., training set, validation set, test set) that includes one or more data inputs 210A (e.g., training inputs, validation inputs, test inputs) and may include one or more target outputs 220A corresponding to the data inputs 210A. The dataset may further include mapping data that maps the data inputs 210A to the target outputs 220A. In some embodiments, the dataset generator 272A may generate data for training a machine learning model configured to output realistic synthetic microscope image data by generating a dataset that includes output microscope image data 268. The data inputs 210A may also be referred to as "features," "attributes," or "information." In some embodiments, the dataset generator 272A may provide the dataset to the training engine 182, the validation engine 184, or the testing engine 186, where the dataset is used to train, validate, or test a machine learning model (e.g., the synthetic data generator 174, one of the machine learning models included in the models 190, the ensemble model 190, etc.).

[0091] System 200B including dataset generator 272B (e.g., dataset generator 172 of FIG. 1) generates a dataset for one or more machine learning models (e.g., model 190 of FIG. 1). Dataset generator 272B may use historical data to generate a dataset (e.g., data input 210B). An exemplary dataset generator 272B is configured to generate a dataset for a machine learning model configured to take predicted microscope image data as input and generate anomaly prediction data as output. Similar dataset generators (or similar operations of dataset generator 272B) may be utilized for machine learning models configured to perform different functions, e.g., a machine learning model configured to receive sensor data and predicted metrology data as input, a machine learning model configured to receive target metrology data (e.g., target microscope image) as input and generate estimated conditions or processing operation recipes as output that may produce a device consistent with the input target data, etc. Dataset generator 272B may share features and / or functionality with dataset generator 272A.

[0092] The dataset generator 272B may generate a dataset for training, testing, and validating a machine learning model. The machine learning model is provided with a set of historical synthetic microscopy data 262A (e.g., output synthetic data from the synthetic data generator 174, output from a model trained using a dataset from the dataset generator 272A, etc.) as a data input 210B. The machine learning model may include two or more separate models (e.g., the machine learning model may be an ensemble model). The machine learning model may be configured to generate output data indicative of the performance of the process chamber, such as an indication of an anomaly present in the processing equipment. In some embodiments, training may not include providing a target output to the machine learning model. The dataset generator 272B may generate a dataset for training an unsupervised machine learning model, such as a model configured to receive synthetic microscopy data as an input and generate clustered data, outlier detection data, anomaly detection data, etc. as an output.

[0093] In some embodiments, the dataset generator 272B generates a dataset (e.g., a training set, a validation set, a test set) that includes one or more data inputs 210B (e.g., training inputs, validation inputs, test inputs). The data inputs 210B may also be referred to as "features," "attributes," or "information." In some embodiments, the dataset generator 272B may provide the dataset to the training engine 182, validation engine 184, or testing engine 186, where the dataset is used to train, validate, or test a machine learning model (e.g., model 190 of FIG. 1). Some embodiments of generating a training set are further described with respect to FIG. 4A.

[0094] In some embodiments, the dataset generator 272B may generate a first data input corresponding to the first set of historical sensor data 244A for training, validating or testing a first machine learning model, and the dataset generator 272A may generate a second data input corresponding to the second set of historical sensor data 244B for training, validating or testing a second machine learning model.

[0095] The data input 210B for training, validating, or testing a machine learning model may include information for a particular manufacturing chamber (e.g., of a particular substrate manufacturing equipment). In some embodiments, the data input 210B may include information for a particular type of manufacturing equipment, e.g., manufacturing equipment sharing a particular characteristic. The data input 210B may include data related to a certain type of device, e.g., device's intended function, design, device produced with a particular recipe, etc. Training a machine learning model based on one type of equipment, device, recipe, etc. may enable the trained model to generate plausible synthetic sensor data in several settings (e.g., for several different facilities, products, etc.).

[0096] In some embodiments, following generating the dataset and using the dataset to train, validate or test a machine learning model, the model may be further trained, validated or tested, or tuned (e.g., weights or parameters, such as connection weights of a neural network, associated with the model's input data).

[0097] FIG. 3 is a block diagram illustrating a system 300 for generating output data (e.g., synthetic data 162 of FIG. 1 ) according to some embodiments. In some embodiments, the system 300 may be used with a machine learning model configured to generate synthetic microscope image data (e.g., synthetic data generator 174 of FIG. 1 ). In some embodiments, the system 300 may be used with a machine learning model to determine corrective actions associated with manufacturing equipment. In some embodiments, the system 300 may be used with a machine learning model to determine defects in manufacturing equipment. In some embodiments, the system 300 may be used with a machine learning model to cluster or classify substrates. The system 300 may be used with a machine learning model associated with a manufacturing system that has different functionality than those listed above.

[0098] At block 310, the system 300 (e.g., a component of the prediction system 110 of FIG. 1 ) performs data splitting (e.g., via the data set generator 172 of the server machine 170 of FIG. 1 ) of data used in training, validating, and / or testing the machine learning model. In some embodiments, the training data 364 includes historical data, such as historical metrology data, historical design rule data, historical classification data (e.g., classification of whether a product meets a performance threshold), historical microscope image data, etc. In some embodiments, the training data 364 may include synthetic microscope image data, e.g., synthetic microscope image data generated by the synthetic data generator 174 of FIG. 1 , when, for example, a synthetic microscope image generated by a trained machine learning model is utilized to train a second machine learning model. The training data 364 may undergo data splitting at block 310 to generate the training set 302, the validation set 304, and the test set 306. For example, the training set may be 60% of the training data, the validation set may be 20% of the training data, and the test set may be 20% of the training data.

[0099] The generation of the training set 302, validation set 304, and test set 306 may be tailored to a particular application. For example, the training set may be 60% of the training data, the validation set may be 20% of the training data, and the test set may be 20% of the training data. The system 300 may generate multiple sets of features for each of the training, validation, and test sets. For example, if the training data 364 includes sensor data, which includes sensor data from 20 sensors (e.g., sensor 126 of FIG. 1) and features derived from 10 manufacturing parameters (e.g., manufacturing parameters corresponding to the same process run as the sensor data from these 20 sensors), the sensor data may be divided into a first set of features including sensors 1-10 and a second set of features including sensors 11-20. Additionally, the manufacturing parameters may be divided into multiple sets, such as a first set of manufacturing parameters including parameters 1-5 and a second set of manufacturing parameters including parameters 6-10. The target inputs may be split into multiple sets, the target outputs may be split into multiple sets, both the target inputs and the target outputs may be split into multiple sets, or neither the target inputs nor the target outputs may be split into multiple sets. Multiple models may be trained on different sets of data.

[0100] At block 312, the system 300 performs model training (e.g., via the training engine 182 of FIG. 1 ) using the training set 302. Training of machine learning models and / or training of models based on physical phenomena (e.g., digital twins) may be accomplished with a supervised learning method, which involves feeding a training dataset of labeled inputs through the model, observing its output, defining an error (by measuring the difference between the output and the label value), and tuning the weights of the model such that the error is minimized using techniques such as deep gradient descent and backpropagation. In many applications, repeating this process across many labeled inputs of the training dataset gives a model that can generate correct outputs when presented with inputs different from those present in the training dataset. In some embodiments, training of the machine learning model may be accomplished in an unsupervised manner, e.g., no labels or classifications may be provided during training. Unsupervised models may be configured to perform anomaly detection, outcome clustering, and the like.

[0101] For each training data item in the training dataset, the training data item may be input to a model (e.g., a machine learning model). The model may then process the input training data item (e.g., several measured dimensions of a manufactured device, a cartoon picture of a manufactured device, etc.) to generate an output. The output may include, for example, a synthetic microscope image. This output may be compared to the label of the training data item (e.g., an actual microscope image of the device associated with the measured dimensions).

[0102] Processing logic may then compare the generated output (e.g., a synthetic image) with the labels (e.g., actual images) included in the training data items. Processing logic determines an error (i.e., a classification error) based on the difference between the output and the labels. Processing logic adjusts one or more weights and / or values ​​of the model based on this error.

[0103] When training the neural network, an error term or delta may be determined for each node of the artificial neural network. Based on this error, the artificial neural network adjusts one or more of the parameters of the artificial neural network (weights for one or more inputs of a node) for one or more of the nodes of the artificial neural network. The parameters may be updated in a backpropagation manner, with the nodes in the top layer being updated first, followed by the nodes in the next layer, and so on. The artificial neural network includes multiple layers of "neurons", each layer receiving as inputs values ​​from neurons in the previous layer. The parameters for each neuron include weights associated with the values ​​received from each of the neurons in the previous layer. Adjusting the parameters may therefore include adjusting weights assigned to each of the inputs to one or more neurons in one or more layers of the artificial neural network.

[0104] System 300 may train multiple models using multiple sets of features from training set 302 (e.g., a first set of features from training set 302, a second set of features from training set 302, etc.). For example, system 300 may train models to generate a first trained model using a first set of features in the training set (e.g., sensor data from sensors 1-10, metrology measurements 1-10, etc.) and to generate a second trained model using a second set of features in the training set (e.g., sensor data from sensors 11-20, metrology measurements 11-20, etc.). In some embodiments, the first trained model and the second trained model may be combined to generate a third trained model (e.g., which may be a better predictor or synthetic data generator than the first or second trained models, alone). In some embodiments, the sets of features used in comparing models may overlap (e.g., the first set of features may be sensor data from sensors 1-15 and the second set of features may be sensors 5-20). In some embodiments, hundreds of models may be generated, including models with various permutations of features and combinations of models.

[0105] At block 314, the system 300 performs model validation (e.g., via validation engine 184 of FIG. 1 ) using the validation set 304. The system 300 may validate each of the trained models using a corresponding set of features in the validation set 304. For example, the system 300 may validate a first trained model using a first set of features in the validation set (e.g., sensor data from sensors 1-10 or metrology measurements 1-10) and a second trained model using a second set of features in the validation set (e.g., sensor data from sensors 11-20 or metrology measurements 11-20). In some embodiments, the system 300 may validate hundreds of models (e.g., models with various permutations of features, combinations of models, etc.) generated at block 312. At block 314, the system 300 may determine the accuracy of each of the one or more trained models (e.g., via model validation) and may determine whether one or more of the trained models have an accuracy that meets a threshold accuracy. In response to determining that none of the trained models have an accuracy that meets the threshold accuracy, flow returns to block 312, where the system 300 performs model training using a different set of features from the training set. In response to determining that one or more of the trained models have an accuracy that meets the threshold accuracy, flow proceeds to block 316. The system 300 may discard trained models that have an accuracy lower than the threshold accuracy (e.g., based on a validation set).

[0106] At block 316, the system 300 performs model selection (e.g., via selection engine 185 of FIG. 1 ) to determine which model of the one or more trained models that meet the threshold accuracy has the highest accuracy (e.g., selected model 308 based on the check of block 314). In response to determining that two or more of the trained models that meet the threshold accuracy have the same accuracy, flow may return to block 312, where the system 300 performs model training to determine the trained model with the highest accuracy using a further refined training set corresponding to the further refined set of features.

[0107] At block 318, the system 300 performs model testing (e.g., via the test engine 186 of FIG. 1 ) using the test set 306 to test the selected model 308. The system 300 may test the first trained model using a first set of features in the test set (e.g., sensor data from sensors 1-10) to determine that the first trained model meets a threshold accuracy (e.g., based on the first set of features of the test set 306). In response to the accuracy of the selected model 308 not meeting the threshold accuracy (e.g., the selected model 308 is too well-fitted to the training set 302 and / or the validation set 304 and cannot be applied to other data sets, such as the test set 306), flow proceeds to block 312, where the system 300 performs model training (e.g., retraining) using a different training set (e.g., sensor data from a different sensor) corresponding to a different set of features. In response to determining that the selected model 308 has an accuracy that meets the threshold accuracy based on the test set 306, flow proceeds to block 320. At least in block 312, the model may learn patterns in the training data to make predictions or generate synthetic data, and in block 318, the system 300 may apply the model to the remaining data (e.g., the training set 306) to test predictions or generate synthetic data.

[0108] At block 320, the system 300 uses the trained model (e.g., the selected model 308) to receive current data 322 (e.g., current metrology data 166 of FIG. 1, such as measurements from an in-situ metrology device) and determines (e.g., extracts) output data 324 (e.g., synthetic data 162 of FIG. 1) from the output of the trained model. Corrective action may be taken in conjunction with the manufacturing equipment 124 of FIG. 1 in light of the output data 324. In some embodiments, the current data 322 may correspond to the same type of features of the historical data used to train the machine learning model. In some embodiments, the current data 322 may correspond to a subset of the type of features of the historical data used to train the selected model 308 (e.g., the machine learning model may be trained using some metrology measurements and may be configured to generate output based on a subset of the metrology measurements).

[0109] In some embodiments, the operation of using the trained model in block 320 may not include providing the current data 322 to the selected model 308. In some embodiments, the selected model 308 may be configured to generate synthetic microscopic image data. Training may include providing true microscopic image data to the machine learning model. The training data (e.g., training set 302) may include attribute data. The attribute data may include information that labels the training data, such as an indication of which tool is associated with the data, a sensor type and ID, an indication of the service life of the tool (e.g., time since the tool was installed, time since the previous maintenance event, etc.), an indication of defects or impending defects in manufacturing equipment that may be reflected in the training data, product type or design, target characteristics of the output image, etc. Using the selected model 308 may include providing instructions to the model to generate synthetic microscopic image data. Using the selected model 308 may include providing one or more attributes. The generated data may be adapted to one or more attributes, for example, synthetic data may be generated that resembles data from a particular tool, data collected when defects are present in manufacturing equipment, data collected from a particular product design, image data at target contrast or brightness levels, etc.

[0110] In some embodiments, the performance of a machine learning model trained, validated, and tested by system 300 may deteriorate. For example, a manufacturing system associated with the trained machine learning model may undergo gradual or sudden changes. As a result of the changes in the manufacturing system, the performance of the trained machine learning model may deteriorate. A new model may be generated to be used to replace the degraded machine learning model. The new model may be generated by modifying the old model by retraining, generating a new model, etc.

[0111] In some embodiments, one or more of operations 310-320 may be performed in various orders and / or together with other operations not presented and described herein. In some embodiments, one or more of operations 310-320 may not be performed. For example, in some embodiments, one or more of data partitioning of block 310, model validation of block 314, model selection of block 316, or model testing of block 318 may not be performed.

[0112] 3 illustrates a system configured to train, validate, test, and use one or more machine learning models. The machine learning models are configured to accept data (e.g., set points provided to manufacturing equipment, sensor data, measurement data, etc.) as input and provide data (e.g., prediction data, corrective action data, classification data, etc.) as output. The partition, train, validate, select, test, and use blocks of the system 300 may be similarly performed to train a second model using a different type of data. Additionally, retraining may be performed using current data 322 and / or additional training data 346.

[0113] 4A-C are flow diagrams of methods 400A-C associated with training and utilizing a machine learning model, according to certain embodiments. Methods 400A-C may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic circuits, programmable logic circuits, microcode, processing devices, etc.), software (e.g., instructions executed on a processing device, a general-purpose computer system, or a dedicated machine), firmware, microcode, or combinations thereof. In some embodiments, methods 400A-C may be performed in part by prediction system 110. Method 400A may be performed in part by prediction system 110 (e.g., server machine 170 and dataset generator 172 of FIG. 1, dataset generators 272A-B of FIG. 2A-B). Prediction system 110 may use method 400A to generate a dataset for performing at least one of training, validating, or testing a machine learning model according to embodiments of the present disclosure. Methods 400B-C may be executed by prediction server 112 (e.g., prediction component 114) and / or server machine 180 (e.g., training, validation, and testing operations may be executed by server machine 180). In some embodiments, a non-transitory machine-readable storage medium stores instructions that, when executed by a processing device (e.g., a processing device of prediction system 110, a processing device of server machine 180, a processing device of prediction server 112, etc.), cause the processing device to perform one or more of methods 400A-C.

[0114] For ease of explanation, methods 400A-C are illustrated and described as a series of operations. However, operations in accordance with the present disclosure may be performed in various orders and / or simultaneously, as well as with other operations not shown and described herein. Moreover, not all illustrated operations may be performed to implement methods 400A-C in accordance with the disclosed subject matter. Furthermore, those skilled in the art will understand and appreciate that methods 400A-C may also be represented as a series of interrelated states via a state diagram or events.

[0115] 4A is a flow diagram of a method 400A for generating a dataset for a machine learning model, according to some embodiments. Referring to FIG. 4A, in some embodiments, at block 401, processing logic performing the method 400A initializes a training set T to be an empty set.

[0116] At block 402, processing logic generates a first data input (e.g., a first training input, a first validation input) that may include one or more of sensor, manufacturing parameters, metrology data, etc. In some embodiments, this first data input may include a first set of features for a type of data and the second data input may include a second set of features for a type of data (e.g., as described with respect to FIG. 3). In some embodiments, the input data may include historical data and / or synthetic data.

[0117] In some embodiments, at block 403, the processing logic optionally generates a first target output for one or more of these data inputs (e.g., a first data input). In some embodiments, the input includes one or more metrology measurements and the target output is a microscope image. In some embodiments, the input includes a cartoon image of the device (e.g., generated using a combination of metrology measurements and one or more design rules) and the target output is a microscope image. In some embodiments, the first target output is prediction data. In some embodiments, the input data may be in the form of sensor data and the target output may also be a list of components that are likely to be defective, as in the case of a machine learning model configured to identify a faulty manufacturing system. In some embodiments, no target output is generated (e.g., an unsupervised machine learning model that can group or find correlations in input data without needing to provide a target output).

[0118] At block 404, processing logic optionally generates mapping data indicating an input / output mapping. This input / output mapping (or mapping data) may relate to a data input (e.g., one or more of the data inputs described herein), a target output for the data input, and an association between the data input and the target output. In some embodiments, such as those associated with machine learning models in which no target output is provided, block 404 may not be performed.

[0119] In some embodiments, at block 405, processing logic adds the mapping data generated at block 404 to the dataset T.

[0120] At block 406, processing logic branches based on whether the dataset T is sufficient for at least one of training, validating, and / or testing a machine learning model, such as synthetic data generator 174 or model 190 of FIG. 1. If so, execution proceeds to block 407; if not, execution returns to block 402. It should be noted that in some embodiments, whether the dataset T is sufficient may be determined simply based on the number of inputs in the dataset, and in some embodiments based on the number of inputs in the dataset that are mapped to outputs, and in some other embodiments, whether the dataset T is sufficient may be determined based on one or more other criteria (e.g., measures of diversity of data examples, accuracy, etc.) in addition to or instead of the number of inputs.

[0121] At block 407, the processing logic provides the dataset T (e.g., to the server machine 180) for training, validating, and / or testing the machine learning model 190. In some embodiments, the dataset T is a training set, and the dataset T is provided to the training engine 182 of the server machine 180 to perform training. In some embodiments, the dataset T is a validation set, and the dataset T is provided to the validation engine 184 of the server machine 180 to perform validation. In some embodiments, the dataset T is a test set, and the dataset T is provided to the test engine 186 of the server machine 180 to perform testing. For example, in the case of a neural network, the input values ​​of a given input / output mapping (e.g., the numerical values ​​associated with the data inputs 210A) are input to the neural network, and the output values ​​of the input / output mapping (e.g., the numerical values ​​associated with the target outputs 220A) are stored in the output nodes of the neural network. The connection weights of the neural network are then adjusted according to a learning algorithm (e.g., backpropagation, etc.), and the procedure is repeated for the remaining input / output mappings of the dataset T. After block 407, the model (e.g., model 190) may be at least one of trained using the training engine 182 of the server machine 180, validated using the validation engine 184 of the server machine 180, or tested using the testing engine 186 of the server machine 180. The trained model may be implemented by the prediction component 114 (of the prediction server 112) to generate predicted data 168 for performing signal processing, to generate synthetic data 162, or to perform corrective actions related to the manufacturing equipment 124.

[0122] FIG. 4B is a flow diagram of a method 400B for generating and utilizing synthetic microscope image data, according to some embodiments. At block 410 of method 400B, processing logic receives first data. The first data includes an indication of a plurality of dimensions of a manufactured device. The first data may be generated from measured or predicted metrology data associated with a product (e.g., a manufactured device). The first data may be generated based on metrology measurements taken in a non-destructive measurement process. In some embodiments, a manufacturing system (e.g., used to produce a manufactured device) may be associated with one or more trained machine learning models. The trained machine learning models may be configured to accept sensor data, manufacturing parameters, process operation recipe specifications, or the like, as inputs and generate one or more predicted measurements of a manufactured device as outputs. The first data may be derived from these predicted measurements.

[0123] In some embodiments, the manufacturing system may include an in-situ metrology system (e.g., a component configured to determine or infer metrology measurements of a product undergoing processing). The in-situ metrology may include, for example, spectral reflectance data of a substrate and may include processing of the data by a machine learning model. The first data may be derived from the in-situ metrology data. In some embodiments, the manufacturing system may include an integrated metrology system (e.g., a component configured to measure or infer measurements of a manufactured device after or between processing operations without exposing the device to ambient atmosphere). The integrated metrology may include optical measurements, spectral data, etc. The first data may be derived from the integrated metrology measurements. In some embodiments, the manufacturing system may include an in-line metrology system (e.g., a metrology system coupled to a processing portion of the manufacturing system). The first data may be derived from the in-line metrology data.

[0124] In some embodiments, measured or predicted metrology data may be provided to the cartoon generator. Exemplary cartoon generator outputs and cartoon generator operations are discussed in more detail in connection with FIG. 5E. The cartoon generator may receive predicted or measured data indicating dimensions of the fabricated device and data indicating design rules for the fabricated device. The design rules may include the placement of structures in the system, e.g., a mask may be located above a gate. The design rules may include the expected shape of the structures, e.g., a spacer may be expected to be rectangular and a gate may be expected to be trapezoidal. The design rules may include relationships between structures, e.g., the width of a structure may be the width of the space between two spacers and a gate. The design rules may include other property information, such as the composition of the structures. The design rules may include other instructions that aid in generating instructions for the location, size, properties, or shape of the structures in the fabricated device.

[0125] The cartoon generator may incorporate design rule data, as well as metrology and / or sensor data (e.g., measured data, data predicted by machine learning models, etc.). The cartoon generator may generate cartoon images (e.g., unrealistic simplified images, synthetic primitive images, etc.) that indicate predicted locations and sizes of structures of the fabricated device. In some embodiments, the cartoon generator may be provided with data from multiple processing operations (e.g., several processing operations related to the same fabricated device) to refine the predictions and the generated images of the device. In some embodiments, the cartoon generator may incorporate design rules to infer additional measurements not provided by the metrology system, for example, the cartoon generator may infer the size of internal structures from the design rules and measurements of external structures. In some embodiments, the cartoon may present views of the fabricated device that may only be obtained by direct imaging by destructive means, for example, cross-sections. The cartoon generator may receive metrology data generated by non-destructive means, for example, optical measurements of the exterior surfaces of the fabricated device.

[0126] At block 412, the processing logic provides the first data to a trained machine learning model. The model may be configured to accept the first data indicative of one or more metrology measurements of the manufactured product and generate a composite microscopic image of the product. At block 414, the processing logic receives a composite microscopic image associated with the manufactured device from the trained machine learning model. The composite microscopic image was generated in consideration of the first data. The machine learning model may be configured to generate a composite image similar to a true image (e.g., an image of the manufactured device captured by a microscopy instrument) used to train the model. In some embodiments, training the machine learning model may include providing cross-sectional image data, top view image data, side view image data, or another viewing angle image data to the machine learning model. In some embodiments, training the machine learning model may include providing scanning electron microscope images, transmission electron microscope images, or another microscopy image to the machine learning model.

[0127] In some embodiments, the trained machine learning model may include a neural network. In some embodiments, the trained machine learning model may include a generator model. In some embodiments, the trained machine learning model may include a generator of a generative adversarial network (GAN). In some embodiments, the machine learning model may be an image-to-image GAN.

[0128] At block 416, processing logic performs one or more actions in response to receiving the composite microscope image. The actions may include outputting the composite microscope image on a display (e.g., issuing an alert to a user). The actions may also include performing one or more operations on the composite microscope image.

[0129] The operations performed on the synthetic microscope image may include measuring features of the synthetic microscope image. The machine learning model that generates the synthetic image may be configured to generate a realistic image associated with the manufactured device. In some embodiments, accurate measurements of structures may be generated from the synthetic microscope image, similar to measurements that would be taken from a true image of the device. Measurements of features, structures, distances, etc. in the synthetic microscope image may be used to calculate dimensions of the manufactured device. In some embodiments, the operations performed on the synthetic microscope image may include inputting the synthetic microscope image into another trained machine learning model, which may output predictions, estimates, etc. based on the synthetic microscope image.

[0130] In some embodiments, corrective action may be taken in consideration of the composite microscope image, measurements of the image, calculated dimensions of the manufactured device, output of a trained machine learning model processing the composite microscope image, etc. In some embodiments, the corrective action may include scheduling maintenance, e.g., corrective or preventive maintenance. In some embodiments, the corrective action may include updating a process recipe, a cleaning recipe, a seasoning recipe, etc. In some embodiments, the corrective action may include providing an alert to a user, e.g., providing an alert to a user if an anomaly exists in the input data or output image to the machine learning model.

[0131] In some embodiments, the method further includes training a machine learning model to generate plausible microscopic images. Additional details regarding the training and operation of an exemplary image generator are further discussed in connection with Figures 4C and 5C.

[0132] FIG. 4C is a flow diagram of a method 400C for generating and using synthetic microscope image data, according to some embodiments. At block 420, processing logic provides historical data including metrology data (e.g., in-situ metrology data, in-line metrology data, etc.) and historical microscope image data (e.g., cross-sectional SEM image data) as training inputs for training a first machine learning model. This first machine learning model may be a GAN. This first machine learning model may be an image-to-image GAN. The input data may be provided with one or more associated attributes, such as data indicating tool ID, sensor ID, process recipe, service life, characteristics of target output image, etc. The GAN may utilize a discriminator model to train the generator model. The metrology data may include data from many types of sensors associated with a manufacturing system, a process chamber, etc. In some embodiments, the metrology data may include in-situ metrology data, in-line metrology data, integrated metrology data, or other such types of data.

[0133] At block 421, the processing logic provides data to the trained first machine learning model indicating one or more attributes of the target synthetic microscope image data. The provided attributes may reflect one or more process or image conditions of interest, for example, one or more conditions under which a large amount of synthetic data is generated. The trained first machine learning model may be provided with a set of random or pseudorandom numbers. The random input may be used as a seed for generating the synthetic data. At block 422, the processing logic receives an output from the trained first machine learning model. The output includes the synthetic microscope image data. The output data may further include data indicating one or more associated attributes. The operations of blocks 421 and 422 may share characteristics with the operations of FIG. 4B.

[0134] At block 423, the processing logic provides this output from the trained first machine learning model as a training input for training a second machine learning model. In some embodiments, this output data is provided along with historical microscope image data, e.g., true microscope image data collected from one or more manufactured products. In some embodiments, synthetic data may be utilized to augment the true data. The processing logic provides data indicative of at least one attribute of the one or more attributes as a target output for training the second machine learning model. In some embodiments, synthetic data is generated to train a model to predict defects, anomalies, or predicted performance of manufacturing equipment that has been in operation for a period of time. In some embodiments, synthetic data may be generated that is related to a particular tool, sensor, product design, etc. Such information may be provided as attribute data to train the second machine learning model.

[0135] At block 424, the processing logic provides the current microscope image data to the trained second machine learning model. The trained second machine learning model may be configured to accept the microscope image data as input and generate predicted data as output, e.g., predicted metrology data, predicted anomalies of a product, predicted defects of a manufacturing system, predicted progress of a processing operation, etc. This second machine learning model may receive as input synthetic (e.g., predicted) microscope image data, true microscope image data, etc. At block 425, the processing logic receives the predicted data output from the trained second machine learning model. At block 426, corrective action is performed in light of the output from the trained second machine learning model. This corrective action may include scheduling maintenance of processing equipment, updating a process recipe, sending an alert to a user, etc.

[0136] 5A-B are diagrams illustrating a process and architecture for training and operating a generative adversarial network, according to some embodiments. FIG. 5A illustrates a simple GAN 500A. During training, input data 502 is provided to a discriminator 508. The discriminator 508 is configured to distinguish whether the input data 502 is real data or synthetic data. The discriminator 508 is trained until it achieves acceptable accuracy. The accuracy parameter may be tuned based on the application, for example, based on the amount of available training data.

[0137] In some embodiments, input data 502 (e.g., drawn from the same data set as the data used to train the discriminator 508) may be provided to the generator 506 to train it to generate plausible synthetic data. The generator 506 may be provided with a random input, such as noise 504, e.g., a fixed-length vector of pseudorandom values. The generator 506 uses this random input as a seed to generate synthetic data (e.g., synthetic microscope images). The generator 506 provides this synthetic data to the discriminator 508. The discriminator 508 is further provided with additional input data 502 (e.g., true data drawn from the same set as the data used to train the discriminator 508). The discriminator 508 attempts to distinguish the input data 502 from the synthetic data provided by the generator 506.

[0138] The discriminator 508 provides classification results (e.g., whether each data set provided to the discriminator 508 was labeled as true or synthetic) to the classification validation module 510. The classification validation module 510 determines whether one or more data sets were correctly labeled by the discriminator 508. Both the discriminator 508 and the generator 506 are provided with feedback data indicating the labeling accuracy. Both the generator 506 and the discriminator 508 are updated in light of the information received from the classification validation module 510. The generator 506 is updated to generate synthetic data that better reproduces the characteristics of the input data 502, e.g., to generate synthetic data that is more frequently labeled as true data by the discriminator 508. The discriminator 508 is updated to improve its accuracy in distinguishing true data from synthetic data. The training process may be repeated until the generator 506 reaches an accuracy threshold, e.g., until a sufficiently large portion of the data generated by the generator 506 is not correctly classified by the discriminator 508.

[0139] 5B is a block diagram illustrating an exemplary GAN 500B operational process for generating synthetic microscope image data, according to some embodiments. In some embodiments, the exemplary GAN 500B may include many of the features discussed in connection with FIG. 5A.

[0140] In some embodiments, the GAN 500B includes a set of generators 520 and a set of discriminators 530. In some embodiments, the discriminators 530 are trained by providing input data 536 to the discriminators 530. The discriminators 530 are configured to distinguish between real data and synthetic data. The generators 520 may be configured to generate synthetic data. The generators 520 may be provided with noise 512, e.g., random or pseudorandom input, as a seed.

[0141] In some embodiments, the GAN 500B may include multiple generators 520 and / or multiple discriminators 530. The discriminator 530 may be configured to accept output data from different generators or sets of generators. In some embodiments, the generator 520 may be configured to generate attribute data via an attribute generator 522 and generate associated data (e.g., synthetic microscope image data) via a feature generator 526. In some embodiments, the feature generator 526 is configured to generate normalized data (e.g., synthetic microscope data having intensity values ​​ranging from 0 to 1), and the min / max generator is configured to generate minimum and maximum values ​​for that data. In some embodiments, an approach that separates the min / max generator from the feature generator 526 may improve the performance of the generator 520.

[0142] In some embodiments, noise 512 may be provided to the attribute generator 522 and / or the feature generator 526. In some embodiments, each generator in the generators 520 may be provided with a different set of noise (e.g., a different set of random inputs). In some embodiments, the output of the attribute generator 522 (e.g., synthetic attribute data) may be provided to an auxiliary discriminator 532. The auxiliary discriminator 532 may determine whether the combination of attribute values ​​is likely to be associated with real data. A preliminary determination may be performed to conserve processing power in generating and / or discriminating synthetic data from the feature generator 526. The output of the generators 520 may all be provided to a discriminator 534. The discriminator 534 may distinguish real data, including attribute data, feature (e.g., microscope image) data, etc., from synthetic data. In some embodiments, the min / max generator may be an optional feature, e.g., GAN 500B may be configured to normalize data from feature generator 526 with the min / max values ​​or to generate data values ​​by feature generator 526.

[0143] In some embodiments, target attributes 523 may be provided to the generator 520. The target attributes may define characteristics of the target microscopic image to be generated. In some embodiments, the target attributes may include features of the image to be generated, such as image quality, contrast, brightness, etc. In some embodiments, the target features include target metrology measurements, one or more target design rules, etc. In some embodiments, the target measurements and / or design rules may be provided to the generator 520 by providing a cartoon image (such as the one shown in FIG. 5E) to the generator 520. The generator 520 may be configured to generate a realistic synthetic microscopic image that shares features in common with the provided attributes (e.g., includes a shape similar to the provided cartoon).

[0144] In some embodiments, the feature generator 526 may include a machine learning generator model designed to generate image data, e.g., synthetic microscopic image data of a manufactured product. In some embodiments, the GAN 500B may include a conditional GAN. In a conditional GAN, during training, the synthetic and true images may be provided to a discriminator, which may be configured to determine whether the synthetic image is an acceptable representation of the true image. In some embodiments, the generator may be further updated with an additional loss function. In some embodiments, the output of the generator (e.g., generated from a cartoon image associated with the manufactured product) may be compared to a true microscopic image of the associated product. In some embodiments, the images may be compared pixel-by-pixel, feature-by-feature, or in other similar manners. The generator may be penalized for the difference between the input and the output based on the sum of the absolute value of the error (e.g., an L1 loss function), the sum of the squared errors (e.g., an L2 loss function), etc.

[0145] In some embodiments, the synthetic data may include a target shape or pattern. For example, future simulated imagery, e.g., future simulated imagery for training a second machine learning model, may include an abnormal structure in one or more imaged products. In some embodiments, the feature generator 526 may accept instructions to facilitate the generation of synthetic data including the target shape or pattern. A range or distribution of positions (e.g., spatial positions), values, shapes, etc. may be provided to the feature generator 526. The feature generator 526 may generate data having the target shape or pattern represented according to a distribution of characteristics, e.g., an abnormal structure may appear in many sets of synthetic data within one position range reaching one height range and having one width range.

[0146] In some embodiments, the synthetic data (e.g., data output from generator 520) may be utilized to train one or more machine learning models. In some embodiments, the synthetic data may be utilized to train a machine learning model configured for event detection, e.g., a machine learning model configured to determine whether the synthetic image data is within normal variation or indicative of a system anomaly. In some embodiments, the synthetic data may be utilized to generate a robust model, where synthetic data may be generated that has a higher noise level than the true data, and a machine learning model trained with the synthetic data may be able to provide useful outputs for a wider variety of inputs than a model trained only on the true data. In some embodiments, the synthetic data may be utilized to test a model for robustness. In some embodiments, the synthetic data may be utilized to generate a model for anomaly detection and / or classification. In some embodiments, the synthetic data may be provided as a training input to train a machine learning model, and one or more attribute data (e.g., attribute data indicative of a system defect) may be provided as a target output to train a machine learning model. In some embodiments, the attribute data may include an indication of the service life of the manufacturing system, such as the time since the system was installed, the number of products produced since the system was installed, the time since the last maintenance event, or the number of products produced since the last maintenance event.

[0147] FIG. 5C is a flow diagram of a method for training a machine learning model (e.g., a GAN) to generate realistic synthetic microscope images, according to some embodiments. At block 540, a true image (e.g., a microscope image of a manufactured device taken using a microscope technique) is generated. The image can be generated using any microscope technique, e.g., scanning electron microscopy, transmission electron microscopy, etc. The image can be an image of a device or structure, including a cross-sectional image, a top-view image, etc. In some embodiments, the images used for training may be selected for use because they exhibit target characteristics, e.g., contrast, clarity, sharpness, etc. In some embodiments, one or more characteristics of the image (e.g., contrast, sharpness, etc.) may be classified as attributes.

[0148] At block 542, one or more critical dimensions (CD) are measured from the microscope image. The measured dimensions may include the height of the structure, the width of the structure, etc. The measurements may include determining the edges of the structure (e.g., using machine learning image processing models), determining the distance from one edge of the structure to the opposite edge on the image, and calculating the size of the imaged device from the size of the image.

[0149] At block 544, the measured CDs and the design rules (from block 543) are provided to a cartoon generator. The cartoon generator is configured to synthesize the CDs and the design rules to generate a cartoon image of the manufactured device. The cartoon image may include both the measured CDs and the design rules information. In some embodiments, the cartoon image presents accurate dimensions in a simplified picture. An exemplary cartoon image is shown in FIG. 5E.

[0150] At block 546, the cartoon is provided to a synthetic image generator to be trained. The synthetic image generator may be included in a GAN. The synthetic image generator may be included in a conditional GAN. The synthetic image generator may be included in an image-to-image (e.g., pix2pix) GAN. The synthetic image generator generates a synthetic image at block 548.

[0151] At block 552, the CD is measured from the composite image. In some embodiments, the CD may be the same as the CD measured at block 542. In some embodiments, the composite image may resemble the true image, and a similar CD may be measurable from the two images. The operations of block 552 may share characteristics in common with the operations of block 542.

[0152] At block 554, additional loss terms are included in the analysis. For example, the true and synthetic images may be compared pixel by pixel. In some embodiments, an L1 loss function may be applied to the synthetic image, e.g., a penalty may be calculated to aid in training a machine learning model. Data indicating the differences between the synthetic and true images (e.g., loss terms, measured CD differences, etc.) may be provided to the image generator. The generator may be updated (e.g., weights between neurons in a neural network may be adjusted) to improve the similarity between the true and synthetic images.

[0153] FIG. 5D illustrates a flow diagram of a method 500D for generating a composite microscope image using a trained image generator based on machine learning, according to some embodiments. At block 560, a number of critical dimensions (CDs) associated with a manufactured product are provided to processing logic. The provided CDs may be predicted CDs, e.g., generated by a machine learning model associated with a system that processes the manufactured product. The provided CDs may be measured during processing of the product, e.g., by an in-situ metrology device. The provided CDs may be measured during or after processing operations, e.g., by an in-line, integrated, or stand-alone metrology system. In some embodiments, the provided product CDs may be generated using non-destructive means, e.g., optical measurements, predictions based on sensor data, etc.

[0154] At block 562, the product CD and the design rules (from block 561) are provided to a cartoon generator, which may have functionality similar to that of the cartoon generator of block 544 of FIG. 5C. The cartoon generator generates as output a cartoon image associated with the manufactured device. The cartoon image may indicate the provided measurements (e.g., CD) and the provided design rules. At block 564, the cartoon image is provided to a composite image generator.

[0155] The synthetic image generator of block 564 may include components such as a GAN (e.g., one or more generators of GANs), a conditional GAN, an image-to-image GAN, a neural network, or other machine learning models, and is configured to accept a cartoon image indicating the structure of a fabricated device and generate as an output a realistic synthetic microscope image 566 of the device.

[0156] 5E shows an exemplary cartoon image 500E of a fabricated device provided as input to a synthetic image generator according to some embodiments. The cartoon image reproduces some information that may have been determined from a microscopic image of the device, e.g., from a SEM or TEM image. Image 500E shows similar structures that may be seen when performing cross-sectional TEM imaging of a logic device. Cartoons showing other views, other devices, other angles, etc. are within the scope of this disclosure.

[0157] Cartoon image 500E includes several structures. A device structure stands on pedestal 570. The device may include a gate 572. The gate may be surrounded by spacers 574, and a mask 576 may be on top of the gate. A deposition material 578 may be disposed on top of mask 576. Other devices, designs, etc. are within the scope of this disclosure.

[0158] The cartoon generator may be a trained machine learning model or may be a drawing module that uses one or more rules to generate a synthetic cartoon (e.g., simple line drawing) image, or may include a trained machine learning model or may include a drawing module that uses one or more rules to generate a synthetic cartoon (e.g., simple line drawing) image. The cartoon generator may be provided with some design rules that are used to approximate the shape of the fabricated device. The design rules may be determined based on the intended device characteristics, characteristics measured by a metrology system, etc. In some embodiments, the design rules may be approximations of the dimensions of the device. For example, the two-dimensional cross-section of the mask 576 in an SEM image of the device may be roughly trapezoidal and may have variations from an exact trapezoid due to the physics of the deposition / etch, non-uniformity of the deposition / etch, effects from adjacent structures, etc. The cartoon generator may be configured to approximate the true cross-sectional shape of the mask with a square that is located above the gate 572 and has a top edge that coincides with the top edge of the spacer 574. Similarly, the cartoon generator may approximate other structures as being simple shapes when in reality they may be more complex, for example, the deposit 578 may be approximated as a semicircle, the spacer 574 may be approximated as a parallelogram or rectangle, and the pedestal 570 may be approximated as a rectangle.

[0159] The cartoon generator may receive some measurements of the device from a manufacturing system, a metrology system, etc. The cartoon generator may also receive some design rules. For example, the cartoon generator may be provided with a set of rules regarding the relative placement of structures, such as a mask 576 having the same width as the gate 572 may be placed above the gate 572 as a square structure, and in the generated cartoon, the top deposit 578 may be a hemisphere with a radius coinciding with the top edge of the spacer 574 and the mask 576. The design rules may be approximations of the true structure shapes. In some embodiments, the synthetic image generator may convert the approximation shapes into more realistic shapes. For example, the cartoon generator may be configured to represent the pedestal 570 as a rectangle, and the synthetic image generator may be trained (based on supplied microscope image training data) to generate images with an etched pedestal (e.g., as shown in image 500E).

[0160] The cartoon generator may combine the provided metrology measurements with the design rules to generate a cartoon image. Metrology data may be available for some dimensions of a device and not for others. The cartoon generator may use the design rules to fill in the measurements that are not provided. For example, a metrology may provide a measurement of a structure, say height 580. Synthesizing this height and known design rules, the cartoon generator may generate an image including all internal structures, for example by estimating (or incorporating another metrology measurement that defines) the width, placing a semicircle representing a silicon dioxide deposit on top of the structure, and generating an image of the entire device by incorporating the intended design rules (e.g., relative widths of masks / gates and spacers, basic shape and placement of structures, etc.).

[0161] In some embodiments, the cartoon image 500E may be provided to a machine learning image generator (e.g., an image-to-image GAN generator). The machine learning model may generate a composite microscope image of the fabricated device as an output. In some embodiments, metrology measurements not taken during / after processing, metrology measurements not provided to the cartoon generator, etc. may be calculated from this composite image. For example, successive width measurements at various locations of the structure may be measured from this composite image, and the etch depth of the pedestal 570 may be quantified.

[0162] FIG. 6 is a block diagram illustrating a computer system 600 according to some embodiments. In some embodiments, the computer system 600 may be connected to other computer systems (e.g., via a network such as a local area network (LAN), an intranet, an extranet, or the Internet). The computer system 600 may operate in the capacity of a server or a client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 600 may be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch or bridge, or any other device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the device. Furthermore, the term "computer" is intended to include a collection of computers that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methods described herein.

[0163] In an additional aspect, computer system 600 may include a processing device 602, a volatile memory 604 (e.g., random access memory (RAM)), a non-volatile memory 606 (e.g., read-only memory (ROM) or electrically erasable programmable ROM (EEPROM)), and a data storage device 618, which may communicate with each other via a bus 608.

[0164] The processing device 602 may be provided by one or more processors, such as a general-purpose processor (e.g., a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a microprocessor implementing other types of instruction sets, or a combination of types of instruction sets), or a specialized processor (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0165] Computer system 600 may further include a network interface device 622 (e.g., coupled to a network 674). Computer system 600 may further include a video display unit 610 (e.g., an LCD), an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse), and a signal generating device 620.

[0166] In some embodiments, the data storage device 618 may include a non-transitory computer-readable storage medium 624 (e.g., a non-transitory machine-readable medium) having stored thereon instructions 626 encoding one or more of the methods or functions described herein, including instructions encoding the components of FIG. 1 (e.g., the prediction component 114, the corrective action component 122, the model 190, etc.) and instructions for performing the methods described herein.

[0167] The instructions 626 may further reside, completely or partially, within the volatile memory 604 and / or within the processing device 602 during execution of the instructions 626 by the computer system 600, and thus the volatile memory 604 and the processing device 602 may also constitute machine-readable storage media.

[0168] While the computer-readable storage medium 624 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" is intended to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of executable instructions. The term "computer-readable storage medium" is further intended to include any tangible medium that can store or encode a set of instructions for execution by a computer that causes the computer to perform one or more of the methods described herein. The term "computer-readable storage medium" is intended to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0169] The methods, components, and features described herein may be implemented by discrete hardware components or may be integrated into the functionality of other hardware components, such as an ASIC, FPGA, DSP, or similar device. Additionally, the methods, components, and features described herein may be implemented by firmware modules or by functional circuits within a hardware device. Additionally, the methods, components, and features described herein may be implemented in any combination of hardware devices and computer program components, or in a computer program.

[0170] Unless otherwise indicated, terms such as "receive," "perform," "provide," "obtain," "perform," "access," "determine," "add," "use," "train," "reduce," "generate," "correct," or other similar terms, refer to operations and processes performed or implemented by a computer system that manipulate and convert data in computer system registers and memory, represented as physical (electronic) quantities, into other data in the computer system memory or registers, or other such information storage, transmission, or display devices, also represented as physical quantities. Furthermore, terms such as "first," "second," "third," "fourth," etc., as used herein, are meant as markers to distinguish between different elements and may not have an ordinal meaning based on the numerical designation of those terms.

[0171] The examples described herein further relate to apparatus for carrying out the methods described herein. The apparatus may be specially constructed for carrying out the methods described herein, or may include a general-purpose computer system that is selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0172] The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform the methods described herein and / or each of the individual functions, routines, subroutines or operations of those methods. Example structures for these various systems are provided in the description above.

[0173] The above description is intended to be illustrative and not limiting. Although the present disclosure has been described with reference to certain illustrative examples and embodiments, it will be recognized that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. receiving first data indicating a plurality of dimensions of a manufactured device; providing the first data to a trained machine learning model; receiving, from the trained machine learning model, a synthetic microscopy image associated with the manufactured device, the synthetic microscopy image having been generated in consideration of the first data; and At least one of (i) outputting the composite microscope image to a display; or (ii) performing one or more operations on the composite microscope image. The method includes:

2. receiving second data indicating a subset of the plurality of dimensions indicated by the first data; receiving third data indicating one or more design rules for the manufactured device; and providing the second data and the third data to a model configured to generate the first data taking into account the second data and the third data; The method of claim 1 further comprising:

3. the first data includes a composite primitive image; the second data includes forecast data generated by a manufacturing system associated with the manufactured device; The method of claim 2.

4. The method of claim 3 , wherein the second data comprises metrology data generated by a non-destructive measurement process.

5. The method of claim 1 , wherein the trained machine learning model comprises a generative adversarial network generator.

6. The method of claim 1 , wherein the composite microscope image comprises a composite image of a cross-section of the fabricated device, the composite image resembling a microscope image produced by an electron-based microscopy technique.

7. performing the one or more operations on the composite microscope image, measuring a feature of the composite microscopic image; and calculating dimensions of the fabricated device based on the measurements of the features of the composite microscope image; The method of claim 1 , comprising:

8. performing corrective action in consideration of the calculated dimensions of the manufactured device, the corrective action comprising: Schedule maintenance, Update the process recipe, or Providing alerts to users The method of claim 7, comprising one or more of:

9. receiving a plurality of microscopic images, each microscopic image of the plurality of microscopic images being a microscopic image of one of a plurality of manufactured devices; receiving first data indicating a plurality of dimensions of the plurality of manufactured devices; and training a machine learning model to generate a composite microscopic image using the first data and the plurality of microscopic images, where training the machine learning model includes providing the first data to the machine learning model as a training input and providing the plurality of microscopic images to the machine learning model as a target output. The method includes:

10. receiving second data, the second data being based on metrology measurements of the plurality of manufactured devices, the metrology measurements being generated by one or more non-destructive measurement processes; and providing the second data to a model to generate the first data, the first data including a plurality of composite primitive images; The method of claim 9 further comprising:

11. The method of claim 9 , wherein training the machine learning model comprises training a generative adversarial network.

12. 10. The method of claim 9, wherein the plurality of microscopic images comprises cross-sectional images of the fabricated device, the microscopic images being generated by an electron-based imaging method.

13. A non-transitory machine-readable storage medium having instructions stored thereon that, when executed, receiving first data indicating a plurality of dimensions of a manufactured device; providing the first data to a trained machine learning model; receiving, from the trained machine learning model, a synthetic microscopy image associated with the manufactured device, the synthetic microscopy image having been generated in consideration of the first data; and At least one of (i) outputting the composite microscope image to a display; or (ii) performing one or more operations on the composite microscope image. A non-transitory machine-readable storage medium that causes a processing device to perform operations including

14. The operation further comprises: receiving second data indicating a subset of the plurality of dimensions indicated by the first data; receiving third data indicating one or more design rules for the manufactured device; and providing the second data and the third data to a model configured to generate the first data taking into account the second data and the third data; 14. The non-transitory machine-readable storage medium of claim 13, comprising:

15. the first data includes a composite primitive image; the second data includes forecast data generated by a manufacturing system associated with the manufactured device; 15. The non-transitory machine-readable storage medium of claim 14.

16. 15. The non-transitory machine-readable storage medium of claim 14, wherein the second data comprises metrology data generated by a non-destructive measurement process.

17. 14. The non-transitory machine-readable storage medium of claim 13, wherein the trained machine learning model comprises a generator of a generative adversarial network.

18. 14. The non-transitory machine-readable storage medium of claim 13, wherein the synthetic microscope image comprises a synthetic cross-sectional scanning electron microscope image.

19. performing the one or more operations on the composite microscope image, measuring a feature of the composite microscopic image; and calculating dimensions of the fabricated device based on the measurements of the features of the composite microscope image; 14. The non-transitory machine-readable storage medium of claim 13, comprising:

20. The operations further include performing corrective action in consideration of the calculated dimensions of the manufactured device, the corrective action comprising: Schedule maintenance, Update the process recipe, or Providing alerts to users 20. The non-transitory machine-readable storage medium of claim 19, comprising one or more of:

Images