Indexing and Modeling for Semiconductor Defect Image Retrieval

Abstract

The subject matter herein can be implemented in, among other things, a method, a system, and a computer-readable storage medium. The method can include a processing device storing a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The method further includes receiving first image data including one or more image frames exhibiting a first substrate processing defect. The method further includes determining a first feature vector corresponding to the first image data. The method further includes determining a selected feature vector from the plurality of feature vectors based on a proximity between the first feature vector and each of the selected feature vectors from the plurality of feature vectors. The method further includes determining second image data including one or more image frames corresponding to the selected embedding vector from the plurality of embedding vectors, and performing an action based on determining the second image data.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION

[0001] Embodiments herein relate generally to modeling for indexing and retrieval of semiconductor defect images. More particularly, embodiments herein relate to image retrieval of semiconductor defect images using a novel combination of deep learning and vector search techniques. [Background technology]

[0002]

[0002] In manufacturing, for example, in the production of semiconductor devices, manufacturing yield depends on device quality, which can be measured directly using metrology tools or indirectly by monitoring sensors on process equipment. This information is collected at various points in the product manufacturing lifecycle. When manufacturing engineers need to identify issues with process tools or the resulting products, they must perform a laborious and costly process of analyzing many data points (e.g., metrology data for numerous samples with various measured parameters). For example, when an engineer is notified of a potential problem with a product, the engineer must review the corresponding metrology data to find any alarming characteristics of the product. A common method for identifying metrology violations is to use image analysis.

[0003]

[0003] In the semiconductor industry, images for failure analysis between process steps are generated from a variety of tools that identify the location, characterization, and classification of defects stored in various databases. A barrier to defect detection / classification is searching the databases for indexed data that helps characterize the substrate processing defects corresponding to the image processing. In conventional systems, retrieval of defect information is often limited by text queries, which are subjective, manual, tedious, indirect, inefficient, and can be limited by the evaluator's knowledge, background, relevant experience, and specific interests. Summary of the Invention

[0004] A method, system, and computer-readable medium (CRM) for facilitating modeling for indexing and retrieval of semiconductor defect images. In some embodiments, a method performed by a processing device can include storing a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The method further includes receiving first image data including one or more image frames exhibiting a first substrate processing defect. The method further includes determining a first feature vector corresponding to the first image data. The method further includes determining a selected feature vector of the plurality of feature vectors based on a proximity between the first feature vector and each of the selected feature vectors of the plurality of feature vectors. The method further includes determining second image data including one or more image frames corresponding to the selected embedding vector of the plurality of embedding vectors, and performing an action based on determining the second image data.

[0005] In some embodiments, a method may include receiving, by a processing device, a first image frame indicative of a substrate processing defect. The method further includes generating a second image frame by cropping a first region of the first image frame. The method further includes generating a third image frame by cropping a second region of the first image frame, the first region including a second region. The method further includes using the second image frame as input to a first machine learning (ML) model. The method further includes obtaining one or more outputs of the first ML model, the one or more outputs indicative of a first feature vector corresponding to the second image frame. The method further includes using the third image frame as input to a second ML model and obtaining one or more outputs of the second ML model indicative of a second feature vector corresponding to the third image frame. The method further includes updating one or more parameters of at least one of the first ML model or the second ML model based on a comparison of the one or more outputs of the first ML model and the one or more outputs of the second ML model.

[0006] In some embodiments, a non-transitory machine-readable storage medium includes instructions that, when executed by a processing device, cause the processing device to perform operations. The operations may include storing, in a data storage device, a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The operations may further include receiving first image data including one or more image frames exhibiting the first substrate processing defect. The operations may further include determining a first feature vector corresponding to the first image data. The operations may further include determining a selected feature vector of the plurality of feature vectors based on a proximity between the first feature vector and each selected feature vector of the plurality of feature vectors. The operations may further include determining second image data including one or more image frames corresponding to a selected embedding vector of the plurality of embedding vectors. The operations may further include performing an action based on determining the second image data.

[0007] Aspects and embodiments of the present disclosure will become more fully understood from the following detailed description and the accompanying drawings, which are intended to illustrate aspects and embodiments by way of example, and not by way of limitation. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary system architecture in which embodiments of the present disclosure may operate. [Figure 2] FIG. 1 is a block diagram illustrating a substrate defect image indexing and retrieval system in which embodiments of the present disclosure may operate. [Figure 3] FIG. 1 is a block diagram of a defect sizing system according to an aspect of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a process for training a machine learning model to generate an output, according to an aspect of the present disclosure. [Figure 5]1 illustrates a model training workflow and a model application workflow for indexing and retrieval of substrate defect images according to an aspect of the present disclosure. [Figure 6A] 1 illustrates a model architecture for indexing and retrieval of substrate defect images according to aspects of the present disclosure. [Figure 6B] 1 illustrates a model architecture for indexing and retrieval of substrate defect images according to aspects of the present disclosure. [Figure 7] 1 shows a flow diagram of an exemplary method for indexing and retrieving substrate defect images, according to some embodiments of the present disclosure. [Figure 8] 1 illustrates a block diagram of an exemplary computing device that operates in accordance with one or more aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009]

[0016] Embodiments described herein relate to indexing and retrieval of semiconductor defect images. In manufacturing, for example, in the production of semiconductor devices, product quality can be measured directly using metrology tools and indirectly by monitoring sensors on process equipment. This information is collected at various points in the product manufacturing lifecycle. When manufacturing engineers need to identify issues with process tools or the resulting products, they must perform the laborious and costly process of analyzing many data points (e.g., metrology data for numerous samples with various measured parameters). For example, when an engineer is notified of a potential problem with a product, the engineer must review the corresponding metrology data to find any alarming characteristics of the product. One form of metrology data that is analyzed is through the use of substrate imaging, such as through the use of an electron microscope (e.g., a scanning electron microscope).

[0010]

[0017] The semiconductor industry generates images for failure analysis and troubleshooting between process steps from a variety of tools that identify defect location, characterization, and classification, all stored in various databases. A barrier to defect detection and classification is locating indexed data within the databases that can help characterize the corresponding substrate processing defects. In traditional systems, defect information retrieval is often limited by text queries, which are subjective, manual, tedious, indirect, and inefficient and can be limited by the evaluator's knowledge, background, relevant experience, and specific interests. Furthermore, traditional systems cannot handle new or out-of-distribution (OOD) data, as well as images containing multiple defect combinations. Furthermore, traditional substrate defect analysis lacks the flexibility to target specific areas of the substrate or selectively remove the effects of other defects that may prevent a specific defect from being classified.

[0011]

[0018] Traditional defect classification algorithms often require learning the manufacturing process, including substrate device processing parameters. Images, such as microscopic images of defects, often require years of engineer experience to understand the symptoms in the image (e.g., defect size, orientation, shape, texture, morphology, and type) and then determine one or more anomalies in the substrate processing procedure or substrate processing equipment based on the symptoms (e.g., where the defect occurred, how it occurred (e.g., how the defect was generated or moved, etc.)). Furthermore, overlapping defects make identifying them even more difficult.

[0012]

[0019] Aspects and implementations of the present disclosure address these and other shortcomings of existing techniques by providing a framework capable of indexing and searching digital images of semiconductor-based particles on substrates based on image content and / or defect size, with options to focus on or ignore specific regions of defects in addition to using region-specific filters. The present disclosure leverages computer learning modeling to determine image representations and build a repository of images and / or image feature representations. In some embodiments, the present disclosure provides a search mechanism that uses image features extracted using computational modeling (e.g., a deep learning-based Visual Transformer (ViT) model). In some embodiments, the present disclosure provides options to crop and / or ignore (e.g., mask) specific portions of an image. In some embodiments, the present disclosure provides a module that extracts and makes available size information (e.g., magnification, image scaling, defect size, etc.) for identifying similar images and / or defects.

[0013]

[0020] The proposed solution leverages learning models (e.g., ViT) to extract good representations of defect images. It uses an index optimized for vector similarity search to quickly search for similar images. The proposed solution effectively reduces the difficulty of processing new data because the index and database can be updated with new data. The proposed solution provides faster and more focused results than traditional text-based solutions. Furthermore, image-based search solutions are not limited by defect keywords, class labels, or other organizational knowledge required by traditional systems. Furthermore, the proposed solution can perform dynamic searches that can acquire, process, and index new information, and also provides the ability to handle multiple defects within the same image. Furthermore, the proposed solution improves image representations of substrate defects while also providing the flexibility to target specific regions of an image (e.g., by using cropping and masking functions).

[0014]

[0021] A method, system, and computer-readable medium (CRM) for facilitating modeling for indexing and retrieval of semiconductor defect images. In an exemplary embodiment, a method performed by a processing device can include storing a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The method further includes receiving first image data including one or more image frames exhibiting a first substrate processing defect. The method further includes determining a first feature vector corresponding to the first image data. The method further includes determining a selected feature vector from the plurality of feature vectors based on a proximity between the first feature vector and each of the selected feature vectors from the plurality of feature vectors. The method further includes determining second image data including one or more image frames corresponding to the selected embedding vector from the plurality of embedding vectors, and performing an action based on determining the second image data.

[0015]

[0022] In an exemplary embodiment, a method may include receiving, by a processing device, a first image frame indicative of a substrate processing defect. The method further includes generating a second image frame by cropping a first region of the first image frame. The method further includes generating a third image frame by cropping a second region of the first image frame, the first region including a second region. The method further includes using the second image frame as an input to a first machine learning (ML) model. The method further includes obtaining one or more outputs of the first ML model, the one or more outputs indicative of a first feature vector corresponding to the second image frame. The method further includes using the third image frame as an input to the second ML model and obtaining one or more outputs of the second ML model indicative of a second feature vector corresponding to the third image frame. The method further includes updating one or more parameters of at least one of the first ML model or the second ML model based on a comparison of the one or more outputs of the first ML model and the one or more outputs of the second ML model.

[0016]

[0023] In an exemplary embodiment, a non-transitory machine-readable storage medium includes instructions that, when executed by a processing device, cause the processing device to perform operations. The operations may include storing, in a data storage device, a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The operations may further include receiving first image data including one or more image frames exhibiting the first substrate processing defect. The operations may further include determining a first feature vector corresponding to the first image data. The operations may further include determining a selected feature vector of the plurality of feature vectors based on a proximity between the first feature vector and each selected feature vector of the plurality of feature vectors. The operations may further include determining second image data including one or more image frames corresponding to a selected embedding vector of the plurality of embedding vectors. The operations may further include performing an action based on determining the second image data.

[0017]

[0024] 1 is a block diagram illustrating an exemplary system architecture 100 in which embodiments of the present disclosure may operate. As shown in FIG. 1, the system architecture 100 includes a manufacturing system 102, a metrology system 110, a client device 150, a data store 140, a server 120, and a machine learning system 170. The machine learning system 170 may be part of the server 120. In some embodiments, one or more components of the machine learning system 170 may be fully or partially integrated into the client device 150. The manufacturing system 102, the metrology system 110, the client device 150, the data store 140, the server 120, and the machine learning system 170 may each be hosted by one or more computing devices, including a server computer, a desktop computer, a laptop computer, a tablet computer, a notebook computer, a personal digital assistant (PDA), a mobile communication device, a mobile phone, a handheld computer, a cloud server, a cloud-based system (e.g., a cloud service device, a cloud network device, or a similar computing device).

[0018]

[0025] The manufacturing system 102, the metrology system 110, the client device 150, the data store 140, the server 120, and the machine learning system 170 may be coupled to each other (e.g., to perform the methods described herein) via a network 160. In some embodiments, the network 160 is a private network that provides each element of the system architecture 100 with access to each other and to other privately available computing devices. The network 160 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, and / or any combination thereof. In some embodiments, the network 160 is a cloud-based network capable of performing cloud-based functions (e.g., providing cloud service capabilities to one or more devices in the system). Alternatively or additionally, any element of the system architecture 100 may be integrated or coupled without the use of the network 160.

[0019]

[0026] Client device 150 may be or include any personal computer (PC), laptop, mobile phone, tablet computer, netbook computer, network-connected television ("smart TV"), network-connected media player (e.g., Blu-ray player), set-top box, over-the-top (OOT) streaming device, operator box, etc. Client device may be capable of performing cloud-based operations (e.g., with server 120, data store 140, manufacturing system 102, machine learning system 170, metrology system 110, etc.). Client device 150 may include browser 152, application 154, and / or other tools as described and executed by other systems in system architecture 100. In some embodiments, the client device 150 may be able to access the manufacturing system 102, the metrology system 110, the data store 140, the server 120, and / or the machine learning system 170 and communicate (e.g., send and / or receive) metrology data, processed data (e.g., augmented image data, embedding vectors, etc.), process result data (e.g., critical dimension (CD) data, thickness data), and / or representations of the inputs and outputs of various process tools (e.g., the imaging tool 114, the data preparation tool 116, the image augmentation tool 124, the embedding tool 126, the search tool 128, the defect tool 130, and / or the search component 194) at various processing stages of the system architecture 100 described herein.

[0020]

[0027] As shown in FIG. 1, the manufacturing system 102 includes process tools 104, process procedures 106, and a process controller 108. The process controller 108 can coordinate the operation of the process tools 104 to perform one or more process procedures 106. For example, various process tools can include specialized chambers such as etch chambers, deposition chambers (including chambers for atomic layer deposition, chemical vapor deposition, sputtering chambers, physical vapor deposition, or plasma versions thereof), anneal chambers, implantation chambers, plating chambers, treatment chambers, etc. In another example, a machine can incorporate a sample transport system (e.g., a selective compliance assembly robotic arm (SCARA) robot, transfer chambers, front-opening up pods (FOUPs), side storage pods (SSPs), etc.) for transporting samples between machines and process steps.

[0021]

[0028] The process steps 106 (sometimes called process recipes or process steps) can include various specifications for performing operations by the process tool 104. For example, the process steps 106 can include process specifications such as the duration of activation of the process operation, the process tools used for the operation, machine (e.g., chamber) temperatures, flow rates, pressures, etc., the sequence of deposition, etc. In another example, the process steps can include transferring instructions for transporting a sample to a further process step or for measurement by the metrology system 110.

[0022]

[0029] The process controller 108 may include devices designed to manage and coordinate the actions of the process tool 104. In some embodiments, the process controller 108 is associated with a process recipe, or a set of process steps 106 instructions, that when applied in a designed manner results in a desired process result for the substrate process. For example, a process recipe may be associated with processing a substrate to produce a target process result (critical dimension, thickness, uniformity criteria, etc.).

[0023]

[0030] As shown in FIG. 1 , the metrology system 110 includes an imaging tool 114 and a data preparation tool 116. The imaging tool 114 can include various sensors for measuring process results (e.g., critical dimensions, thickness, uniformity, etc.) within the manufacturing system 102. For example, the imaging tool 114 can include a scanning tunneling microscope (STM) or a scanning electron microscope (SEM). In another example, wafers processed within one or more processing chambers can be used to measure critical dimensions. The imaging tool 114 can also include devices that measure process results of substrates processed using the manufacturing system. For example, process results such as critical dimensions, thickness measurements (e.g., film layers from etching, deposition, etc.) of substrates processed according to a process recipe and / or actions performed by the process controller 108 can be evaluated. These measurements can also be used to measure chamber conditions throughout the substrate processing procedure.

[0024]

[0031] The data preparation tool 116 can include process methods for extracting features and / or generating synthetic / engineered data associated with the data measured by the imaging tool 114. In some embodiments, the data preparation tool 116 can identify correlations, patterns, and / or anomalies in the metrology data or process performance data. For example, the data preparation tool 116 can perform feature extraction, where the data preparation tool 116 uses a combination of measurement data to determine whether a criterion has been met. For example, the data preparation tool 116 can analyze multiple data points of related parameters to determine whether abrupt changes occurred during a substrate processing procedure across multiple processing chambers. In some embodiments, the data preparation tool 116 performs normalization across various sensor data associated with different conditions of the processing chambers. Normalization can include processing the incoming sensor data to appear similar across the various chambers and sensors used to acquire the data.

[0025]

[0032] In some embodiments, the data preparation tool 116 can perform one or more of a process control analysis, a univariate limit violation analysis, or a multivariate limit violation analysis on the metrology data (e.g., data acquired by the imaging tool 114). For example, the data preparation tool 116 can perform statistical process control (SPC) by using statistics-based methods to monitor and control the process controller 108. For example, SPC can improve the efficiency and accuracy of substrate processing procedures (e.g., by identifying data points that are within and / or outside of control limits).

[0026]

[0033] In some embodiments, the extracted features, generated synthetic / engineered data, and statistical analysis can be used in connection with machine learning system 170 (e.g., to train, validate, and / or test machine learning model 190). Additionally and / or alternatively, data preparation tool 116 can output data to server 120 for use by any of image augmentation tool 124, embedding tool 126, search tool 128, and defect tool 130.

[0027]

[0034] The data store 140 may be a memory (e.g., random access memory), a drive (e.g., a hard drive, a flash drive), a database system, a cloud-based system, or another type of component or device capable of storing data. The data store 140 may store one or more historical data 142, including storing an image repository 144 of previously processed images (e.g., images of substrate defects) and corresponding vectorized image features and metadata 146. The vectorized image data may include feature vectors or embedded data representing imaging data (e.g., image frames, images acquired using the imaging tool 114).

[0028]

[0035] The server 120 may include one or more computing devices, such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc. The server 120 may include an image augmentation tool 124, an embedding tool 126, a search tool 128, and / or a defect tool 130. The server 120 may include a cloud server or a server capable of performing one or more cloud-based functions. For example, operations of one or more of the image augmentation tool 124, the embedding tool 126, the search tool 128, and / or the defect tool 130 may be provided to a remote device (e.g., a client device 150) using a cloud environment.

[0029]

[0036] The image enhancement tool 124 receives image data (e.g., image frames) indicative of substrate processing defects and performs image enhancement on the received image data. In some embodiments, the image enhancement tool 124 includes processing logic that processes selected received image frames, identifies image features, and performs a filtering procedure that removes image frames that do not meet certain criteria. For example, images of substrate defects may be included in an image set that includes graphs, data tables, metrology-related images, etc. The filtering procedure identifies which images are indicative of substrate defects and removes images that do not directly indicate substrate processing defects.

[0030]

[0037] In some embodiments, the image augmentation tool 124 performs a cropping procedure on one or more of the received image frames. Cropping refers to the removal of unwanted outer regions from a photographic or illustrative image. This process typically involves removing portions of the image's peripheral areas to remove unwanted debris from the image, improve the image's framing, change the aspect ratio, or emphasize or separate a subject from its background. This can be performed using image editing software or an algorithm that simulates an image editing procedure. For example, the image augmentation tool 124 can receive one or more selected image frames (e.g., from the measurement system 110 and / or the client device 150) and crop one or more image frames according to the selection (e.g., generate a second image frame based on cropping the first image frame according to the selection).

[0031]

[0038] In some embodiments, the image augmentation tool 124 performs a masking procedure on one or more of the received image frames. Image masking is a technique used to isolate various portions of an image. For example, image masking can include photocompositing multiple images, hiding all or part of an image, applying selective adjustments, creating crops (such as removing backgrounds), adjusting transparency, etc. For example, the image augmentation tool 124 can receive one or more selected image frames (e.g., from the measurement system 110 and / or the client device 150) and mask one or more image frames according to the selection (e.g., generating a second image frame based on cropping a first image frame according to the selection).

[0032]

[0039] The embedding tool 126 receives image data including one or more image frames (e.g., from the metrology system 110) and determines embedded data representing the image data. The embedding tool 126 includes process methods for extracting features and / or generating synthetic / engineered data associated with the data measured by the imaging tool 114 in the form of feature data (e.g., feature vectors). In some embodiments, the embedding tool 116 can identify correlations, patterns, and / or anomalies in the metrology data or process performance data. An embedding is a relatively low-dimensional space into which a high-dimensional representation (e.g., an image) can be transformed. The embedded data (e.g., feature vectors) capture the semantics of the received image frames. The embedding tool 126 outputs the embedded data (e.g., for use by the search tool 128). The output of the embedding layer can be further passed to other machine learning techniques, such as clustering, k-nearest neighbor analysis, etc.

[0033]

[0040] The search tool 128 receives the embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) that correspond to previously processed images (e.g., image repositories). The search tool 128 performs a proximity search between the received feature embeddings and one or more feature embeddings in the vectorized image features and metadata 146 of the historical data 142.

[0034]

[0041] In some embodiments, the search tool 128 uses a vector search and / or nearest neighbor method to determine the set of nearby images in the image repository 144. The search tool may identify the vector that is closest (e.g., most similar) to the received and / or provided feature vector.

[0035]

[0042] In some embodiments, the search tool 128 scans the historical data 142 to retrieve the image repository 144 and the vectorized image features and metadata 146. The search tool 128 can use an index of vectorized image features to quickly analyze the feature vectors. In some embodiments, the search tool utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0036]

[0043] The defect tool 130 receives one or more similar image frames indicative of substrate processing defects. The defect tool 130 identifies substrate processing defects based on the selected similar image frames. The defect tool 130 identifies instances of anomalies in the manufacturing process based on a comparison of the current image with each of the selected similar image frames. In some embodiments, the defect tool 138 receives similar from the pattern mapping tool 137 and identifies instances of anomalies based on the sample patterns.

[0037]

[0044] The defect tool 130 can acquire failure mode and effects analysis (FMEA) data. The FMEA data can include a list of known issues and root causes for a particular piece of equipment, each with associated known symptoms. The defects identified by the defect tool 138 and / or similar images received are applied to the list of known issues to generate a report identifying common causes of the defects. For example, the defect tool 130 can determine the defects and identify tools, machines, or operations in the manufacturing process that correspond to the identified defects.

[0038]

[0045] In some embodiments, the defect tool 138 can be used in conjunction with the process dependency data to identify tools, machines, or processes operated upstream of the current machine operation being performed on the current sample (e.g., an operation step that occurred before the current manufacturing step in the same manufacturing process). For example, the current sample may have recently undergone a first operation by a first machine. In some embodiments, the defect tool 130 can utilize a combination of the process dependency data and the failure mode and effects analysis data to examine past operations of the sample, such as a second operation by a second machine or tool.

[0039]

[0046] Once an instance of an anomaly is identified, the defect tool 130 may proceed by modifying at least one of the machine operation or process implementation associated with the instance of anomaly and / or providing a graphical user interface (GUI) that presents a visual indicator of the machine or process associated with the instance of anomaly. The GUI may be transmitted over the network 160 and displayed on the client device 150. In some embodiments, modifying the machine operation or process implementation may include transmitting instructions to the manufacturing execution system 102 to modify the process tool 104, the process procedure 106, and / or the process controller 108.

[0040]

[0047] As mentioned above, some embodiments of the image augmentation tool 124, the embedding tool 126, the search tool 128, and / or the defect tool 130 can use machine learning models to perform the described methods. The associated machine learning models can be generated (e.g., trained, validated, and / or tested) using the machine learning system 170. The following exemplary description of the machine learning system 170 is described in the context of using the machine learning system 170 to generate the machine learning model 190 associated with the embedding tool 126. However, it should be noted that this description is merely an example. Similar processing hierarchies and methods can be used to generate and execute the machine learning models associated with the image augmentation tool 124, the embedding tool 126, the search tool 128, and / or the defect tool 130, which can use the machine learning models to perform the described methods.

[0041]

[0048] The machine learning system 170 can include one or more computing devices, such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a cloud computer, a cloud server, a system stored on one or more clouds, etc. The machine learning system 170 can include an embedding component 194 and a search component 196. In some embodiments, the embedding component 194 can receive as input one or more image frames indicative of substrate processing defects and use the historical data 142 in conjunction with the trained machine learning model 190 to determine feature embeddings corresponding to the image frames. In some embodiments, the search component 196 can use the trained machine learning model 190 to search the image repository 144 using the vectorized image features and metadata 146.

[0042]

[0049] In some embodiments, machine learning system 170 further includes server machine 172 and server machine 180. Server machines 172 and 180 may be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a cloud computer, a cloud server, a system stored on one or more clouds, etc.), a data store (such as a hard disk, a memory database, etc.), a network, a software component, or a hardware component.

[0043]

[0050] The server machine 172 may include a dataset generator 174 that can generate datasets (e.g., a set of data inputs and a set of target outputs) for training, validating, or testing a machine learning model. The dataset generator 174 may divide the historical data 142 into a training set (e.g., 60% of the historical data or another portion of the historical data), a validation set (e.g., 20% of the historical data or another portion of the historical data), and a test set (e.g., 20% of the historical data). In some embodiments, the dataset generator 174 generates multiple sets of training data. For example, one or more sets of training data may include each dataset (e.g., a training set, a validation set, a test set).

[0044]

[0051] Server machine 180 includes a training engine 182, a validation engine 184, and a testing engine 186. Training engine 182 may be capable of training machine learning model 190 using one or more images, vectorized image features and metadata 146, and / or historical process result data 148 from historical data 142 (from data store 140). In some embodiments, machine learning model 190 may be trained using one or more outputs of data preparation tool 116, image augmentation tool 124, embedding tool 126, search tool 128, and / or defect tool 130. For example, machine learning model 190 may be a hybrid machine learning model that uses image data and / or embedded features such as feature extraction, mechanistic modeling, and / or statistical modeling. Training engine 182 may generate multiple trained machine learning models 190, each corresponding to a different feature set for each training set.

[0045]

[0052] The validation engine 184 can determine the accuracy of each of the trained machine learning models 190 based on the corresponding feature set of each training set. The validation engine 184 can discard trained machine learning models 190 with accuracies that do not meet a threshold accuracy. The test engine 186 can determine the trained machine learning model 190 with the highest accuracy among all the trained machine learning models based on the test set (and, optionally, the validation set).

[0046]

[0053] In some embodiments, training data is provided to train the machine learning model 190 such that the trained machine learning model can receive new inputs having new image data indicative of new substrate processing defects. The new outputs can represent new feature embeddings (e.g., feature vectors). In some embodiments, the training data may be further used such that the new outputs further include selected similar feature vectors corresponding to images of similar substrate processing defects.

[0047]

[0054] The machine learning model 190 may refer to a model created by the training engine 182 using a training set that includes data inputs and corresponding target outputs (image frames and corresponding vectorized image features and metadata). Patterns can be found in the data set that map the data inputs to the target outputs (e.g., identifying relationships between portions of sensor data and resulting chamber status), and the machine learning model 190 is provided with a mapping that captures these patterns. The machine learning model 190 may use one or more of logistic regression, parsing, decision trees, or support vector machines (SVMs). The machine learning may consist of single-level linear or nonlinear operations (e.g., SVMs) or may be a neural network.

[0048]

[0055] The embedding component 194 can provide current data (e.g., image frames showing substrate processing defects) as input to the trained machine learning model 190 and run the trained machine learning model 190 on the input to obtain one or more outputs including a set of vectorized image features and metadata. The embedding component 194 can identify confidence data from the output indicating the confidence of the predicted vectorized image features and metadata. In one non-limiting example, the confidence is a real number between 0 and 1, where 0 indicates no confidence in one or more chamber statuses and 1 represents absolute confidence in the chamber statuses.

[0049]

[0056] For purposes of illustration and not limitation, aspects of the present disclosure describe training a machine learning model and using the trained machine learning model using information about historical data 142. In other embodiments, heuristic or rule-based models are used to determine chamber status.

[0050]

[0057] In some embodiments, the functionality of client device 150, server 120, data store 140, and machine learning system 170 may be provided by fewer machines than shown in Figure 1. For example, in some embodiments, server machines 172 and 180 may be combined into a single machine, and in other embodiments, server machines 172, 180, and 192 may be combined into a single machine. In some embodiments, machine learning system 170 may be provided in whole or in part by server 120.

[0051]

[0058] In general, functions described in one embodiment as being performed by client device 150, data store 140, metrology system 110, manufacturing system 102, and machine learning system 170 may also be performed on server 120 in other embodiments, where appropriate. Additionally, functions assigned to particular components may be performed by different components or multiple components working in conjunction.

[0052]

[0059] In embodiments, a "user" may be represented as a single individual. However, other embodiments of the present disclosure encompass a "user" being an entity controlled by multiple users and / or automated sources. For example, a set of individual users federated as a group of administrators may be considered a "user."

[0053]

[0060] 2 is a block diagram illustrating a substrate defect image indexing and retrieval system 200 in which embodiments of the present disclosure may operate. The substrate defect image indexing and retrieval system 200 may include aspects and / or features of the system architecture 100.

[0054]

[0061] 2, the substrate defect indexing and retrieval system 200 receives an input image 202. The input image may include one or more image frames showing substrate processing defects. The input image 202 may include an image of a substrate process result involving a substrate processing defect. For example, the input image 202 may include an image from a scanning tunneling microscope (STM) or a scanning electron microscope (SEM).

[0055]

[0062] As shown in FIG. 2, the substrate defect indexing and retrieval system 200 includes image enhancement logic, including cropping logic 204, masking logic 206, and size detection logic 208. The cropping logic 204 performs a cropping procedure on one or more of the received image frames. Cropping refers to the removal of unwanted outer regions from a photographic or illustrative image. This process typically involves removing portions of the image's peripheral areas to remove unwanted artifacts from the image, improve the image's framing, change the aspect ratio, or emphasize or separate a subject from its background. This can be performed using image editing software or an algorithm that simulates an image editing procedure. For example, the cropping logic 204 can receive a selection of one or more image frames and crop one or more image frames according to the selection (e.g., generate a second image frame based on cropping a first image frame according to the selection).

[0056]

[0063] The masking logic 206 performs a masking procedure on one or more of the received image frames. Image masking is a technique used to isolate various portions of an image. For example, image masking can include photocompositing multiple images, hiding all or part of an image, applying selective adjustments, creating crops (such as removing a background), adjusting transparency, etc. For example, the masking logic 206 can receive a selection of one or more image frames and mask one or more image frames according to the selection (e.g., generating a second image frame based on cropping a first image frame according to the selection).

[0057]

[0064] The size detection logic 208 processes the input image 202 and extracts content from the input image 202 that indicates an image scaling factor associated with the input image 202. The image scaling factor indicates the relative size of a depiction in the image to the size of the image. For example, the image scaling factor may include an image scale, a magnification factor, a depicted size of a depicted object, etc. The size detection logic 208 extracts size identification information (e.g., text) and determines the size of the depicted defect. The defect size is then passed to one or more downstream processes (e.g., embedding logic 210, search logic 212, defect detection logic 214, etc.). More information regarding the size detection logic 208 is provided in connection with FIG. 3.

[0058]

[0065] As shown in FIG. 2 , the substrate defect indexing and retrieval system 200 includes embedding logic 210. The embedding logic 210 receives image data, including an input image 202. The embedding logic 210 includes process methods for extracting features and / or generating synthesized / engineered data associated with data measured by an imaging tool in the form of feature data (e.g., feature vectors). In some embodiments, the embedding logic 210 can identify correlations, patterns, and / or anomalies in metrology data or process performance data. An embedding is a relatively low-dimensional space into which a high-dimensional representation (such as an image) can be transformed. The embedded data (e.g., feature vectors) capture the semantics of the received image frames. The embedding logic 210 outputs the embedded data (e.g., for use by the search tool 128). The output of the embedding layer can be further passed to other machine learning techniques, such as clustering, k-nearest neighbor analysis, etc.

[0059]

[0066] 2, the substrate defect indexing and retrieval system 200 includes search logic 212. The search logic 212 receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) that correspond to previously processed images (e.g., image repositories). The search logic 212 performs a proximity search between the received feature embedding and one or more feature embeddings of vectorized image features and metadata of previously processed images of other substrate processing defects.

[0060]

[0067] In some embodiments, the search logic 212 uses a vector search and / or nearest neighbor method to determine the set of nearby images in the image repository. The search tool can identify the vector that is closest (e.g., most similar) to the received and / or provided feature vector.

[0061]

[0068] In some embodiments, the search logic 212 scans the data structure to retrieve the image repository and corresponding vectorized image features and metadata. The search logic 212 can use vectorized image feature indexing methods to quickly analyze the feature vectors. In some embodiments, the search tool utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0062]

[0069] 2 , the substrate defect indexing and retrieval system 200 includes defect detection logic 214. The defect detection logic 214 receives one or more similar image frames indicative of substrate processing defects. The defect detection logic 214 identifies the substrate processing defects based on the selected similar image frames. The defect detection logic 214 identifies instances of anomalies in the manufacturing process based on a comparison of the current image with each of the selected similar image frames. In some embodiments, the defect detection logic 214 receives similar images from the search logic 212 and identifies instances of anomalies based on the similar images.

[0063]

[0070] The defect detection logic 214 can obtain failure mode and effects analysis (FMEA) data. The FMEA data can include a list of known issues and root causes for a particular piece of equipment, each with associated known symptoms. The defects identified by the defect detection logic 214 and / or similar images received are applied to the list of known issues to generate a report identifying common causes of the defects. For example, the defect detection tool 214 can determine the defects and identify tools, machines, or operations in the manufacturing process that correspond to the identified defects.

[0064]

[0071] In some embodiments, the defect detection logic 214 can use the process dependency data to identify tools, machines, or processes operated upstream of the current machine operation being performed on the current sample (e.g., an operation step that occurred before the current manufacturing step in the same manufacturing process). For example, the current sample may have recently undergone a first operation by a first machine. In some embodiments, the defect detection tool 214 can utilize a combination of the process dependency data and the failure mode and effects analysis data to examine past operations of the sample, such as a second operation by a second machine or tool.

[0065]

[0072] Once an instance of an anomaly is identified, the defect tool 130 may proceed by modifying at least one of the machine operation or process implementation associated with the instance of the anomaly and / or providing a graphical user interface (GUI) 216 that presents a visual indicator of the machine or process associated with the instance of the anomaly. The GUI may be transmitted over the network 160 and displayed on the client device 150. In some embodiments, modifying the machine operation or process implementation may include transmitting instructions to the manufacturing execution system 102 to modify process entities of the manufacturing execution system 102 (e.g., the process tool 104, the process step 106, and / or the process controller 108 in FIG. 1 ).

[0066]

[0073] In some embodiments, the search logic 212 outputs selected vectorized image features that are similar to the input image 202 or selected images with similar vectorized image features. Images can also be sent directly to the GUI 216 without determining specific defects. For example, the GUI 216 can include a location to display images that are determined to be similar to the input image 202.

[0067]

[0074] FIG. 3 illustrates a block diagram of a defect sizing system 300 according to an embodiment of the present disclosure. One or more features described in connection with FIG. 3 may be performed by size detection logic 208 of FIG. 2. As shown in FIG. 3, defect sizing system 300 may include receiving an input image 302, line masking logic 304, line removal logic 306, text recognition logic 308, and / or post-processing logic 310. Input image 302 may include one or more image frames showing substrate processing defects. Input image 302 may include an image of substrate process results involving substrate processing defects. For example, input image 302 may include an image from a scanning tunneling microscope (STM) or a scanning electron microscope (SEM).

[0068]

[0075] The line masking logic 304 determines the location of edges within the image frame. The line masking logic may determine the boundary or global edge of the image frame. For example, some images may include vertical or horizontal lines at the edge of the image frame. In some embodiments, the line masking logic 304 performs gamma-corrected thresholding to identify one or more edges within the image frame. For example, vertical and / or horizontal lines within the image frame may be used to identify data stored within the image, such as text overlaid on the image. In some embodiments, the line masking logic 304 performs localization using segmentation to generate a mask for repair of the image frame.

[0069]

[0076] The line removal logic 306 receives line masking data from the line masking logic 304. The line removal tool applies image enhancement techniques (cropping, masking, etc.) to portions of the image frame based on the detected lines. For example, the image may contain a brand logo or other artificial mark that is identified using the line masking logic 304. The line removal logic 306 can apply a mask to remove the lines detected by the line masking logic 304.

[0070]

[0077] The text recognition logic 308 identifies text within an image frame. For example, the image may include a relative image scaling factor, such as a scale indicating distance displayed on the image. The text recognition logic 308 can identify the numerical value and units of a given image scaling factor (e.g., magnification, relative drawing distance, etc.). The text recognition logic 308 can separate the text from the rest of the input image 302.

[0071]

[0078] The post-processing logic 310 performs actions based on the recognized text. For example, the post-processing logic may perform data cleaning and standardization, which may include delineating regions of the input image 302 identified by the text recognition logic 308. The post-processing logic 310 may further include making the size data available to other processes in the system. In some embodiments, the post-processing logic 310 determines the size of the defect based on the scaling and provides the defect size (e.g., as metadata) to further embedding and / or classification procedures associated with the input image 302.

[0072]

[0079] 4 is a block diagram illustrating a process 400 for training a machine learning model to generate an output, according to certain embodiments. Process 400 includes receiving training data in the form of input images 402. Input images 402 may include one or more image frames showing substrate processing defects. Input images 402 may include images of substrate process results involving substrate processing defects. For example, input images 402 may include images from a scanning tunneling microscope (STM) or a scanning electron microscope (SEM).

[0073]

[0080] As shown in FIG. 4, the process 400 includes cropping extension of an input image 402. A copy of the input image 402 may undergo local cropping 404, and a copy of the input image 402 may undergo global cropping 406. The local cropping may be associated with a selected portion of the input image 402 that is located within the selected portion of the input image 402. The locally cropped image is used as an input to a student model 408. The globally cropped image is used as an input to a teacher model 410. The student model outputs a student model prediction 412. The student model prediction 412 and / or the teacher model prediction 414 include a profile (e.g., a distribution) indicating different embedding vectors corresponding to the input image 402 and a corresponding confidence (e.g., probability) associated with each output.

[0074]

[0081] During operation, an exemplary training procedure may include processing logic receiving a first image frame (e.g., input image 402) indicative of a substrate processing defect. The processing logic further generates a second image frame by cropping a first region of the first image frame (e.g., global cropping 406). The processing logic further generates a third image frame by cropping a second region of the first image frame (e.g., local cropping 404). The first region (associated with global cropping 406) includes the second region (associated with local cropping 404). The processing logic uses the second image frame as input to a first machine learning (ML) model (e.g., teacher model 410). The processing logic obtains one or more outputs of the first ML model. The one or more outputs (teacher model predictions 414) indicate a first feature vector corresponding to the second image frame. The processing logic uses the third image frame as input to a second ML model (e.g., student model 408). Processing logic obtains one or more outputs of the second ML model (e.g., student model predictions 412). The one or more outputs indicate a second feature vector corresponding to the third image frame. Processing logic updates one or more parameters of at least one of the first ML model or the second ML model based on a comparison of the one or more outputs of the first ML model and the one or more outputs of the second ML model (e.g., model tuning 416).

[0075]

[0082] In some embodiments, the student models 408 may be machine learning models similar to the trained teacher models 410 but including fewer layers and / or nodes than each of the trained teacher models 410, resulting in a more compressed machine learning model. In some embodiments, multiple student models 408 may be trained, with each student model 408 trained to predict embedding vectors for a different subset of input images 402 within the cluster of input images for which the teacher model 410 is trained to output error predictions. In some embodiments, a different student model 408 may be trained for each cluster of input images 402 and / or substrate processing defects. Each teacher model may then be used to train multiple student models.

[0076]

[0083] In some embodiments, model tuning 416 includes determining an error or result of a loss function, such as a ranking loss, that is back-propagated to the student model 408. The ranking loss represents the difference between the probability of error (from the student model) and the perceived true probability (from the teacher model). For example, if the teacher model 410 determines that a first prediction has a first probability and the student model 408 determines that the first prediction has a second probability, the error is related to the difference between the two probabilities. In some embodiments, the ranking loss function may be a categorical cross-entropy function, a Kullback-Leibler divergence function, or any suitable loss function.

[0077]

[0084] In some embodiments, training can be performed by inputting input images 402 into the machine learning model one by one. In some embodiments, after one or more training rounds, processing logic can determine whether a stopping criterion has been met. The stopping criterion can be a target accuracy level, a target number of processed images from the training dataset, a target amount of change in parameters relative to one or more previous data points, a combination thereof, and / or other criteria. In one embodiment, the stopping criterion is met when at least a minimum number of data points have been processed and the loss value has stabilized and / or stopped decreasing. The loss value can represent the sum of the errors of the machine learning model. For example, the loss value can represent the sum of the differences between the modeled values ​​and the actual values. In one embodiment, the stopping criterion is met when the accuracy of the machine learning model stops improving. If the stopping criterion is not met, additional training is performed. If the stopping criterion is met, training can be completed. Once the machine learning model is trained, the model can be tested using a reserved portion of the training dataset.

[0078]

[0085] 5 illustrates a model training workflow 505 and a model application workflow 517 for substrate process outcome prediction according to aspects of the present disclosure. In some embodiments, the model training workflow 505 executes on a server that may or may not include a process defect image indexing and retrieval application, and the trained model is provided to the process defect image indexing and retrieval application, which may execute the model application workflow 517. The model training workflow 505 and the model application workflow 517 may be performed by processing logic executed by a processor of a computing device (e.g., server 120 of FIG. 1 ). One or more of these workflows 505, 517 may be implemented, for example, by a processing device implemented with one or more machine learning modules and / or other software and / or firmware executing on the processing device.

[0079]

[0086] The model training workflow 505 trains one or more machine learning models (e.g., regression models, boosted regression models, principal component analysis models, deep learning models, vision transformers) to perform one or more tasks, such as determining, predicting, or modifying, associated with a process outcome predictor (e.g., feature extraction, image search). The model application workflow 517 applies one or more trained machine learning models to perform tasks, such as determining and / or tuning, for image data (e.g., image frames indicative of substrate processing defects). The one or more machine learning models can receive the image data (e.g., image frames indicative of substrate processing defects).

[0080]

[0087] Various machine learning outputs are described herein. Specific numbers and arrangements of machine learning models are described and illustrated. However, it should be understood that the number and types of machine learning models used, and the arrangements of such machine learning models, can be varied to achieve the same or similar end results. Thus, the arrangements of machine learning models described and illustrated are merely examples and should not be construed as limiting.

[0081]

[0088] In some embodiments, one or more machine learning models are trained to perform one or more of the following tasks: Each task may be performed by a separate machine learning model; Alternatively, a single machine learning model may perform each task or a subset of tasks; Additionally or alternatively, different machine learning models may be trained to perform different combinations of tasks; In one example, one or a few machine learning models may be trained, where the trained machine learning (ML) model is a single shared neural network with multiple shared layers and multiple higher-level individual output layers, each output layer outputting a different prediction, classification, identification, etc. Tasks that the one or more trained machine learning models may be trained to perform include: a. Feature Extractor 567—As previously described, the feature extractor receives image data (e.g., raw and / or enhanced image frames showing substrate processing defects). The feature extractor includes process methods for extracting features and / or generating synthesized / engineered data associated with the data measured by the imaging tool in the form of feature data (e.g., feature vectors). In some embodiments, the feature extractor can identify correlations, patterns, and / or anomalies in metrology data or process performance data. An embedding is a relatively low-dimensional space into which a high-dimensional representation (such as an image) can be transformed. The embedding data (e.g., feature vectors) captures the semantics of the received image frames. b. Image Acquirer 564 - The image acquirer receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., image repository). The image acquirer performs a proximity search between the received feature embeddings and one or more feature embeddings of the vectorized image features and metadata. In some embodiments, the image acquirer uses a vector search and / or nearest neighbor method to determine a set of nearby images (e.g., within a threshold proximity). In some embodiments, the image acquirer utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0082]

[0089] To perform training, processing logic inputs a training dataset 536 to one or more untrained machine learning models (see FIG. 4 for training details). Before inputting the first input to the machine learning models, the machine learning models may be initialized. Processing logic trains the untrained machine learning models based on the training dataset to generate one or more trained machine learning models that perform the various operations described above.

[0083]

[0090] Once the one or more trained machine learning models 538 are generated, they may be stored in model storage 545 and added to the process defect image indexing and retrieval application. The process defect image indexing and retrieval application may use one or more trained ML models 538 and additional processing logic to implement an automatic mode in which manual entry of information by a user is minimized, and in some cases, eliminated.

[0084]

[0091] In the model application workflow 517, according to one embodiment, input data 562 (e.g., image frames showing substrate processing defects) may be used as input to a feature extractor 567, which may include a trained machine learning model. Based on the input data 562, the feature extractor 567 outputs feature data 569 and metadata representing the input data 562. The feature data 569 is input to an image acquirer 564, which may include a trained machine learning model. Based on the feature data 569, the image acquirer 564 identifies one or more other images (e.g., similar image data 566) that are identified as being similar to the input data 562 via the feature data 569.

[0085]

[0092] 6A-6B illustrate a model architecture 600 for indexing and retrieval of substrate defect images, according to an embodiment of the present disclosure. Model architecture 600 can be used to run one or more ML models. In general, model architecture 600 consists of an embedding layer 608, an encoder 614, and a final head classifier 616. First, the image is divided into non-overlapping patches. Each patch is considered a separate token by the architecture. For an image of size c×h×w (h is height, w is width, and c represents the number of channels), patches are extracted in each dimension as p×p. This results in: Sequence of patches (x1, x2, …, x) in TIFF2025539375000002.tif9170 n ) is formed. In some embodiments, the patch size p is chosen as 16x16 or 32x32.

[0086]

[0093] As shown in FIG. 6A , at 604, an input image 602 is divided (e.g., flattened) into individual image patches. For example, the input image 602 is divided into a fixed number of uniformly sized patches or embedding tokens. The input image may be converted (e.g., flattened) into a sequence of token embeddings 606 that indicate the content of the image patches. In some embodiments, the model architecture uses a constant latent vector size n throughout all its layers, and the patches are flattened and mapped to n dimensions by a trainable linear projection using a linear embedding layer 608.

[0087]

[0094] Before inputting the sequence of patches into the encoder 614, it linearly projects them into a vector of model dimension d using the learned embedding matrix. The embedded representation is concatenated with a learnable classification token that is used to perform the classification task. The embedded image patches are viewed by the Transformer as a set of patches without any notion of order. To keep the spatial arrangement of the patches the same as in the original image, positional information 610 is encoded and added to the patch representation 612 (e.g., a linear embedding that represents the content of the corresponding image patch). The resulting embedded sequence of patches for token 0 is represented as follows: TIFF2025539375000003.tif7170

[0088]

[0095] The resulting sequence of embedded patches z0 is passed to a Transformer encoder 614. As shown in Figure 6B, the encoder 614 consists of L identical layers, each with two main subcomponents: (1) a multi-head self-attention block (MSA) 656 and (2) a fully connected feedforward dense block (MLP) 660. Each of the encoder's two subcomponents employs a residual skip connection and is preceded by a normalization layer (LN) (e.g., normalization layer 654 and normalization layer 658). The final layer of the encoder 614 takes the first element of the sequence and passes it to an external head classifier to determine the Predict the class label represented by TIFF2025539375000004.tif6170.

[0089]

[0096] The MSA block 656 in the encoder 614 is the central component of the Transformer. The MSA block 656 determines the relative importance of a single patch embedding relative to other embeddings in the sequence. This block has four layers: a linear layer, a self-attention layer, a concatenation layer that concatenates the outputs of multiple attention heads, and a final linear layer. At a high level, attention can be represented by an attention weight, which is calculated by taking the weighted sum of all values ​​in the sequence z. The MSA block employs an attention function that maps a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. The output is calculated as a weighted sum of values, and the weight assigned to each value is calculated by a fitness function between the query and the corresponding key. The results of all attention heads are concatenated and projected to the target dimension through a feedforward layer with learnable weights. The MLP 616 performs class prediction 618 based on the data received from the encoder 614.

[0090]

[0097] 7 illustrates a flow diagram of an exemplary method 700 for indexing and retrieving substrate defect images according to some embodiments of the present disclosure. Method 700 is performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., running on a general-purpose computer system or a dedicated machine), or any combination thereof. In one embodiment, the method is performed using server 120 and trained machine learning model 190 of FIG. 1 , although in other embodiments, one or more blocks of FIG. 7 may be performed by one or more other machines not shown.

[0091]

[0098] The method 700 may include receiving image data (e.g., related to a substrate processing defect) and processing the received image data using a trained machine learning model 190. The trained model may be configured to generate a feature embedding for the image based on the image data. The method 700 further identifies similar images that may exhibit similar defects.

[0092]

[0099] At block 702, processing logic stores, in a data storage device, a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. At block 704, processing logic receives first image data including one or more image frames indicative of the first substrate processing defect. The first image data may include one or more image frames indicative of the substrate processing defect. The input images may include images of substrate process results involving the substrate processing defect. For example, the input images may include images from a scanning tunneling microscope (STM) or a scanning electron microscope (SEM).

[0093]

[0100] In some embodiments, processing logic receives a first selection of a first image frame of the first image data and generates a second image frame by cropping a region of the first image frame based on the first selection. A first feature vector is determined using the second image frame. In some embodiments, processing logic receives a first selection of a first image frame of the first image data. Processing logic masks a region of the first image frame based on the first selection to generate a second image frame. A first feature vector is determined using the second image frame.

[0094]

[0101] In some embodiments, processing logic extracts a selected portion of text from a first image frame of the first image data. The processing logic further determines an image scaling factor associated with the first image frame. The processing logic further determines a size associated with the first substrate processing defect based on the image scaling factor. A selected feature vector of the plurality of feature vectors is determined using the size.

[0095]

[0102] At block 706, processing logic determines a first feature vector corresponding to the first image data. In some embodiments, processing logic further divides a first image frame of the first image data into a set of image patches corresponding to the first image frame. Processing logic further determines a set of linear embeddings corresponding to the content of each image patch in the set of image patches. Processing logic further determines a set of positional embeddings corresponding to the relative position of each image patch in the set of image patches. The first feature vector is determined based on the set of linear embeddings and the set of positional embeddings.

[0096]

[0103] At block 708, processing logic determines a selected feature vector from the plurality of feature vectors based on the proximity between the first feature vector and each of the selected feature vectors from the plurality of feature vectors. At block 710, processing logic determines second image data including one or more image frames corresponding to the selected embedding vector from the plurality of embedding vectors. Processing logic receives the embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., image repository). Processing logic performs a proximity search between the received feature embeddings and one or more feature embeddings from the vectorized image features and metadata. In some embodiments, processing logic uses a vector search and / or nearest neighbor solving method to determine a set of nearby images (e.g., within a threshold proximity). In some embodiments, processing logic utilizes Euclidean distance and / or cosine similarity to determine the distance between the feature vectors.

[0097]

[0104] At block 712, processing logic optionally performs an action based on the determination of the second image data. In some embodiments, processing logic optionally prepares the second image data for display on a graphical user interface (GUI). For example, the second image data is a set of image frames similar to the first image frames and showing similar substrate processing defects. In another example, the second image data can be displayed on the GUI by displaying an indication of the first substrate processing defect.

[0098]

[0105] In some embodiments, the processing logic optionally alters operation of the process chamber and / or processing tool based on the second image data. For example, the processing logic may determine a substrate processing defect based on the second image data. The processing logic may further determine an instance of a manufacturing process anomaly associated with the first substrate processing defect. The processing logic may further send instructions (e.g., perform corrective actions associated with manufacturing process equipment) to one or more process controllers to alter one or more operation of a processing device associated with the instance of the anomaly (e.g., modify a process recipe and / or process parameters, terminate substrate processing in one or more process tools and / or process chambers, initiate preventative maintenance associated with one or more process chambers and / or process tools, etc.).

[0099]

[0106] 8 illustrates a block diagram of an exemplary computing device 800 that operates in accordance with one or more aspects of the present disclosure. In various examples, various components of computing device 800 may represent various components of client device 150, measurement system 110, server 120, data store 140, and machine learning system 170 shown in FIG.

[0100]

[0107] The exemplary computing device 800 can be connected to other computer devices within a LAN, an intranet, an extranet, and / or the Internet. The computing device 800 can operate as a server in a client-server network environment. The computing device 800 may be a personal computer (PC), a set-top box (STB), a server, a network router, a switch, a bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be performed by that device. Furthermore, although only a single exemplary computing device is illustrated, the term "computer" is intended to include a collection of computers that individually or collectively execute a set of instructions (or multiple sets) to perform any one or more of the methodologies described herein.

[0101]

[0108] The exemplary computing device 800 may include a processing device 802 (also referred to as a processor or CPU), a main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory 806 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 818), which may communicate with each other via a bus 830.

[0102]

[0109] Processing device 802 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, processing device 802 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 802 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. According to one or more aspects of the present disclosure, processing device 802 may be configured to execute instructions implementing method 700 shown in FIG. 7.

[0103]

[0110] The exemplary computing device 800 may further include a network interface device 808 that may be communicatively coupled to a network 820. The exemplary computing device 800 may further include a video display 810 (e.g., a liquid crystal display (LCD), touch screen, or cathode ray tube (CRT)), an alphanumeric input device 812 (e.g., a keyboard), a cursor control device 814 (e.g., a mouse), and an audio signal generating device 816 (e.g., a speaker).

[0104]

[0111] The data storage device 818 may include a machine-readable storage medium (or, more specifically, a non-transitory machine-readable storage medium) 828 having stored thereon one or more sets of executable instructions 822. According to one or more aspects of the present disclosure, the executable instructions 822 may include executable instructions associated with performing the method 700 shown in FIG.

[0105]

[0112] The executable instructions 822 may also reside, completely or at least partially, within the main memory 804 and / or within the processing device 802 during execution by the exemplary computing device 800, with the main memory 804 and the processing device 802 also constituting computer-readable storage media. The executable instructions 822 may also be transmitted or received over a network via the network interface device 808.

[0106]

[0113] Although computer-readable storage medium 828 is shown in FIG. 8 as a single medium, the term "computer-readable storage medium" should be interpreted to include a single medium or multiple media (e.g., centralized or distributed databases, and / or associated caches and servers) that store one or more sets of operating instructions. The term "computer-readable storage medium" is also intended to include any medium that can store or encode a set of instructions for execution by a machine, causing the machine to perform any one or more of the methods described herein. Accordingly, the term "computer-readable storage medium" is intended to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0107]

[0114] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. These steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0108]

[0115] It should be borne in mind, however, that all these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise indicated, and as will be apparent from the following description, descriptions using terms such as "identify," "determine," "store," "adjust," "cause," "return," "compare," "create," "stop," "load," "copy," "send," "replace," "execute," and the like throughout the description will be understood to refer to operations and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the computer system's registers and memory into other data also represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0109]

[0116] Examples of the present disclosure also relate to apparatus for performing the methods described herein. This apparatus may be specially constructed for the required purposes, or it may be a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such computer program may be stored on a computer-readable storage medium, including, but not limited to, any type of disk, including optical disks, compact disk read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic disk storage media, optical storage media, flash memory devices, other types of machine-accessible storage media, or any type of medium suitable for storing electronic instructions, each connected to a computer system bus.

[0110]

[0117] The methods and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, although it may be convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be set forth in the description below. Moreover, the scope of the present disclosure is not limited to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the present disclosure.

[0111]

[0118] It should be understood that the above description is illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. While the present disclosure describes particular examples, it will be understood that the disclosed systems and methods are not limited to the examples described herein, but may be practiced with modification within the scope of the appended claims. Accordingly, the specification and drawings should be considered in an illustrative, and not a restrictive, sense. The scope of the present disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. storing, in a data storage device, a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects; receiving, by a processing device, first image data comprising one or more image frames indicative of a first substrate processing defect; determining, by the processing device, a first feature vector corresponding to the first image data; determining, by the processing device, a selected feature vector of the plurality of feature vectors based on a proximity between the first feature vector and each of the selected feature vectors of the plurality of feature vectors; determining, by the processing device, second image data including one or more image frames corresponding to selected embedding vectors from a plurality of embedding vectors; performing, by the processing device, an action based on determining the second image data; A method comprising:

2. 10. The method of claim 1, further comprising: identifying, by the processing device, the first substrate processing defect based on the second image data, wherein the action is further based on identifying the first substrate processing defect.

3. identifying, by the processing device, an instance of a manufacturing process anomaly associated with the first substrate processing defect; causing, by the processing device, the performance of a corrective action associated with manufacturing process equipment based on the instance of an anomaly; The method of claim 2 further comprising:

4. The method of claim 1 , further comprising preparing, by the processing device, one or more image frames of the second image data for display on a graphical user interface (GUI).

5. receiving, by the processing device, a first selection of first image frames of the first image data; generating, by the processing device, a second image frame by cropping a region of the first image frame based on the first selection; 2. The method of claim 1, further comprising: wherein the first feature vector is determined using the second image frame.

6. receiving, by the processing device, a first selection of first image frames of the first image data; generating, by the processing device, a second image frame by masking an area of ​​the first image frame based on the first selection; 2. The method of claim 1, further comprising: wherein the first feature vector is determined using the second image frame.

7. extracting, by the processing device, a selected portion of text from a first image frame of the first image data; determining, by the processing device, an image scaling factor associated with the first image frame; determining, by the processing device, a size associated with the first substrate processing defect based on the image scaling factor; 2. The method of claim 1, further comprising: wherein the selected feature vector of the plurality of feature vectors is determined further using the size.

8. dividing, by the processing device, a first image frame of the first image data into a set of image patches corresponding to the first image frame; determining, by the processing device, a set of linear embeddings corresponding to the content of each image patch of the set of image patches; determining, by the processing device, a set of position embeddings corresponding to a relative position of each image patch of the set of image patches; 2. The method of claim 1, further comprising: wherein the first feature vector is determined based on the set of linear embeddings and the set of positional embeddings.

9. receiving, by a processing device, a first image frame indicative of a substrate processing defect; generating, by the processing device, a second image frame by cropping a first region of the first image frame; generating, by the processing device, a third image frame by cropping a second region of the first image frame, the first region including the second region; using the second image frame as an input to a first machine learning (ML) model; obtaining one or more outputs of the first ML model, the one or more outputs indicative of a first feature vector corresponding to the second image frame; using the third image frame as input to a second ML model; obtaining one or more outputs of the second ML model, the one or more outputs indicative of a second feature vector corresponding to the third image frame; updating one or more parameters of at least one of the first ML model or the second ML model based on a comparison of the one or more outputs of the first ML model and the one or more outputs of the second ML model; A method comprising:

10. the one or more outputs of the first ML model further indicate a confidence associated with the first feature vector; The method of claim 9 , wherein the one or more outputs of the second ML model further indicate a confidence level associated with the second feature vector.

11. segmenting the second image frame into a set of image patches corresponding to the second image frame using the first ML model; determining a set of linear embeddings corresponding to the content of each image patch of the set of image patches using the first ML model; determining a set of position embeddings using the first ML model, each position embedding corresponding to a position of a corresponding image patch in the set of image patches; 10. The method of claim 9, further comprising: wherein the first feature vector is determined based on the set of linear embeddings and the set of positional embeddings.

12. the one or more outputs of the first ML model include: (i) a first set of predictions; and (ii) a first set of probabilities each corresponding to a prediction in the first set of predictions; 10. The method of claim 9, wherein the one or more outputs of the second ML model include: (i) a second set of predictions; and (ii) a second set of probabilities each corresponding to a prediction in the second set of predictions.

13. The method of claim 9 , wherein at least one of the first ML model or the second ML model includes a Vision Transformer (ViT).

14. 1. A non-transitory machine-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations, the operations including: storing, in a data storage device, a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects; receiving first image data including one or more image frames indicative of a first substrate processing defect; determining a first feature vector corresponding to the first image data; determining a selected feature vector of the plurality of feature vectors based on a proximity between the first feature vector and each of the selected feature vectors of the plurality of feature vectors; determining second image data including one or more image frames corresponding to selected embedding vectors of the plurality of embedding vectors; and performing an action based on determining the second image data.

15. The operation is 15. The non-transitory machine-readable storage medium of claim 14, further comprising: identifying the first substrate processing defect based on the second image data, wherein the action is further based on identifying the first substrate processing defect.

16. The operation is identifying an instance of a manufacturing process anomaly associated with the first substrate processing defect; triggering execution of a corrective action associated with the manufacturing process equipment based on said instance of anomaly; 20. The non-transitory machine-readable storage medium of claim 15, further comprising:

17. The operation is 15. The non-transitory machine-readable storage medium of claim 14, further comprising preparing one or more image frames of the second image data for display on a graphical user interface (GUI).

18. The operation is receiving a first selection of a first image frame of the first image data; generating a second image frame by at least one of cropping and masking an area of ​​the first image frame based on the first selection; 15. The non-transitory machine-readable storage medium of claim 14, further comprising: wherein the first feature vector is determined using the second image frame.

19. The operation is extracting a selected portion of text from a first image frame of the first image data; determining an image scaling factor associated with the first image frame; determining a size associated with the first substrate processing defect based on the image scaling factor; 15. The non-transitory machine-readable storage medium of claim 14, further comprising: wherein the selected feature vector of the plurality of feature vectors is determined further using the size.

20. The operation is Dividing a first image frame of the first image data into a set of image patches corresponding to the first image frame; determining a set of linear embeddings corresponding to the content of each image patch of said set of image patches; determining a set of position embeddings corresponding to the position of each image patch of the set of image patches; 15. The non-transitory machine-readable storage medium of claim 14, further comprising: wherein the first feature vector is determined based on the set of linear embeddings and the set of positional embeddings.

Images