Modeling for indexing and semiconductor defect image retrieval

Abstract

The subject matter of this specification can be implemented, particularly in methods, systems, and computer-readable storage media. A method may include a processing device storing a plurality of feature vectors, which represent image frames corresponding to various substrate processing defects of a previously processed image. The method further includes receiving first image data including one or more image frames indicating first substrate processing defects. The method further includes determining a first feature vector corresponding to the first image data. The method further includes determining the selection of the plurality of feature vectors based on proximity between the first feature vector and each of the selections of the plurality of feature vectors. The method further includes determining second image data including one or more image frames corresponding to the selection of the plurality of embedding vectors, and performing an action based on the determination of the second image data.

Description

Technical Field

[0001] Embodiments of this specification generally relate to modeling for semiconductor defect image indexing and retrieval. More specifically, embodiments of this specification relate to image search on semiconductor defect images using a novel combination of deep learning and vector search techniques. Prior Art

[0002] In manufacturing, such as semiconductor device manufacturing, fab yield depends on device quality, which can be measured directly using metrology tools and indirectly through sensors monitoring process equipment. This information is collected at various times during the product manufacturing lifecycle. When a manufacturing engineer needs to identify a problem with a process tool or end product, he or she must undergo a laborious and expensive process of analyzing numerous data points (e.g., metrology data from many samples with various measurement parameters). For example, when an engineer is notified of a potential product problem, he or she must review the corresponding metrology data to identify the product's alarming characteristics. A common method for identifying metrology violations is through image analysis.

[0003] The semiconductor industry generates images from various tools for fault analysis between processing steps, identifying defect locations, characteristics, and classifications stored in various databases. A barrier to defect detection / classification is searching the database for indexed data that characterizes substrate processing defects corresponding to the image processing. Searching for defect information in conventional systems is often limited by text queries, which can be subjective, manual, tedious, indirect, inefficient, and limited by the evaluator's knowledge, background, relevant experience, and specific interests. Summary of the Invention

[0004] A method, system, and computer-readable media (CRM) facilitate modeling for semiconductor defect image indexing and retrieval. In some embodiments, a method performed by a processing device may include storing a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The method further includes receiving first image data comprising one or more image frames indicative of a first substrate processing defect. The method further includes determining a first feature vector corresponding to the first image data. The method further includes determining a selection of the plurality of feature vectors based on proximity between the first feature vector and each of a selection of the plurality of feature vectors. The method further includes determining second image data comprising one or more image frames corresponding to a selection of the plurality of embedding vectors, and performing an action based on determining the second image data.

[0005] In some embodiments, a method may include a processing device receiving a first image frame indicating a substrate processing defect. The method further includes generating a second image frame by cropping a first region of the first image frame. The method further includes generating a third image frame by cropping a second region of the first image frame, wherein the first region includes the second region. The method further includes using the second image frame as input to a first machine learning (ML) model. The method further includes obtaining one or more outputs of the first ML model, the one or more outputs indicating a first feature vector corresponding to the second image frame. The method further includes using the third image frame as input to a second ML model and obtaining one or more outputs of the second ML model indicating a second feature vector corresponding to the third image frame. The method further includes updating one or more parameters of at least one of the first ML model or the second ML model based on a comparison between the one or more outputs of the first ML model and the one or more outputs of the second ML model.

[0006] In some embodiments, a non-transitory machine-readable storage medium includes instructions that, when executed by a processing device, cause the processing device to perform operations. The operations may include storing a plurality of feature vectors in a data storage device, the feature vectors representing previously processed image frames corresponding to various substrate processing defects. The operations may further include receiving first image data, the first image data including one or more image frames indicating a first substrate processing defect. The operations may further include determining a first feature vector corresponding to the first image data. The operations may further include determining a selection of the plurality of feature vectors based on proximity between the first feature vector and each of a selection of the plurality of feature vectors. The operations may further include determining second image data, the second image data including one or more image frames corresponding to a selection of the plurality of embedding vectors. The operations may further include performing an action based on the determination of the second image data. Simple diagram description

[0007] Aspects and embodiments of the present disclosure will be more fully understood from the detailed description given below and the accompanying drawings, which are intended to illustrate aspects and embodiments by way of example and not limitation.

[0008] FIG. 1 is a block diagram illustrating an example system architecture in which embodiments of the present disclosure may operate.

[0009] FIG. 2 is a block diagram illustrating a substrate defect image indexing and retrieval system in which embodiments of the present disclosure may operate.

[0010] FIG. 3 is a block diagram of a defect size determination system according to aspects of the present disclosure.

[0011] FIG. 4 is a block diagram illustrating a process for training a machine learning model to generate an output according to aspects of the present disclosure.

[0012] FIG. 5 illustrates a model training workflow and a model application workflow for substrate defect image indexing and retrieval according to aspects of the present disclosure.

[0013] 6A-6B illustrate a model architecture for substrate defect image indexing and retrieval according to aspects of the present disclosure.

[0014] FIG. 7 depicts a flow chart of an example method for substrate defect image indexing and retrieval according to some embodiments of the present disclosure.

[0015] FIG. 8 depicts a block diagram of an example computing device operating in accordance with one or more aspects of the present disclosure. Implementation Method

[0016] Embodiments described herein relate to semiconductor defect image indexing and retrieval. In manufacturing, such as semiconductor device manufacturing, product quality can be measured directly using metrology tools and indirectly by monitoring sensors on process equipment. This information is collected at various times during the product manufacturing lifecycle. When a manufacturing engineer needs to identify a problem with a process tool or end product, he or she must undergo a laborious and expensive process of analyzing a large number of data points (e.g., metrology data from many samples with various measurement parameters). For example, when an engineer is notified of a potential problem with a product, the engineer must review the corresponding metrology data to identify alarming features of the product. One form of analyzing metrology data is through the use of substrate imaging, such as, for example, through the use of an electron microscope (e.g., a scanning electron microscope).

[0017] The semiconductor industry generates images from various tools for fault analysis and troubleshooting between processing steps, identifying defect locations, characteristics, and classifications stored in various databases. A barrier to defect detection / classification is searching the database for indexed data that helps characterize substrate processing defects corresponding to the image processing. Searching for defect information in conventional systems is often limited by text queries, which can be subjective, manual, tedious, indirect, inefficient, and limited by the evaluator's knowledge, background, relevant experience, and specific interests. Conventional systems are further unable to handle images with new or out-of-distribution (OOD) data and combinations of multiple defects. Conventional substrate defect analysis also lacks the flexibility to target specific areas of the substrate and selectively remove the effects of other defects that may interfere with a given defect classification.

[0018] Conventional defect classification algorithms often require learning manufacturing process parameters, such as substrate device processing parameters. Defect images (such as microscopic images) often require years of experience for engineers to understand the symptoms within the images (e.g., defect size, orientation, shape, texture, morphology, type, etc.) and, based on these symptoms, determine one or more anomalies in the substrate processing process or substrate processing equipment, such as where the defect originated and how it was caused (e.g., how the defect was generated or transported). Furthermore, the challenge of defect identification is exacerbated when multiple defects overlap.

[0019] In addition to employing domain-specific filters, aspects and embodiments of the present disclosure address these other shortcomings of the prior art by providing a framework for indexing and searching digital images of semiconductor-based particles on substrates based on image content and / or defect size, with the option to focus on or ignore specific areas of defects. The present disclosure utilizes computer learning modeling to determine image representations and build a repository of image and / or feature representations of images. In some embodiments, the present disclosure provides a search mechanism that uses image features extracted using computational modeling (e.g., a deep learning-based Visual Transformer (ViT) model). In some embodiments, the present disclosure provides options for cropping and / or ignoring (e.g., masking) certain portions of an image. In some embodiments, the present disclosure provides a module for extracting size information (e.g., magnification, image scaling, defect size, etc.) and making it available for identifying similar images and / or defects.

[0020] The proposed solution utilizes a learning model (e.g., ViT) to extract a good representation of defect images. It employs an index optimized for vector similarity searches to quickly retrieve similar images. The proposed solution effectively alleviates the difficulty of processing new data, as the index and database can be updated with new data. Compared to conventional text-based solutions, the proposed solution provides faster and more focused results. Furthermore, the image-based search solution is not limited by defect keywords, category labels, and / or other institutional knowledge required in conventional systems. The proposed solution further enables dynamic searches, where new information can be retrieved, processed, and indexed, as well as the ability to handle multiple defects in the same image. In addition to providing the flexibility to target specific areas of an image (e.g., using cropping and / or masking features), the proposed solution further provides improved image representations of substrate defects.

[0021] A method, system, and computer-readable medium (CRM) facilitate modeling for semiconductor defect image indexing and retrieval. In an exemplary embodiment, a method performed by a processing device may include storing a plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects. The method further includes receiving first image data comprising one or more image frames indicative of a first substrate processing defect. The method further includes determining a first feature vector corresponding to the first image data. The method further includes determining a selection of the plurality of feature vectors based on proximity between the first feature vector and each of a selection of the plurality of feature vectors. The method further includes determining second image data comprising one or more image frames corresponding to a selection of the plurality of embedding vectors, and performing an action based on determining the second image data.

[0022] In an example embodiment, a method may include a processing device receiving a first image frame indicating a substrate processing defect. The method further includes generating a second image frame by cropping a first region of the first image frame. The method further includes generating a third image frame by cropping a second region of the first image frame, wherein the first region includes the second region. The method further includes using the second image frame as input to a first machine learning (ML) model. The method further includes obtaining one or more outputs of the first ML model, the one or more outputs indicating a first feature vector corresponding to the second image frame. The method further includes using the third image frame as input to a second ML model and obtaining one or more outputs of the second ML model indicating a second feature vector corresponding to the third image frame. The method further includes updating one or more parameters of at least one of the first ML model or the second ML model based on a comparison between the one or more outputs of the first ML model and the one or more outputs of the second ML model.

[0023] In an example embodiment, a non-transitory machine-readable storage medium includes instructions that, when executed by a processing device, cause the processing device to perform operations. The operations may include storing a plurality of feature vectors in a data storage device, the feature vectors representing previously processed image frames corresponding to various substrate processing defects. The operations may further include receiving first image data, the first image data including one or more image frames indicating a first substrate processing defect. The operations may further include determining a first feature vector corresponding to the first image data. The operations may further include determining a selection of the plurality of feature vectors based on proximity between the first feature vector and each of a selection of the plurality of feature vectors. The operations may further include determining second image data, the second image data including one or more image frames corresponding to a selection of the plurality of embedding vectors. The operations may further include performing an action based on the determination of the second image data.

[0024] FIG1 is a block diagram illustrating an example system architecture 100 in which embodiments of the present disclosure may operate. As shown in FIG1 , system architecture 100 includes a manufacturing system 102, a metrology system 110, a client device 150, a data store 140, a server 120, and a machine learning system 170. Machine learning system 170 may be part of server 120. In some embodiments, one or more components of machine learning system 170 may be fully or partially integrated into client device 150. Manufacturing system 102, metrology system 110, client device 150, data store 140, server 120, and machine learning system 170 may each be hosted by one or more computing devices, including a server computer, a desktop computer, a laptop computer, a tablet computer, a notebook computer, a personal digital assistant (PDA), a mobile communication device, a cellular phone, a handheld computer, a cloud server, a cloud-based system (e.g., a cloud service device, a cloud network device, or the like).

[0025] Manufacturing system 102, metrology system 110, client device 150, data storage 140, server 120, and machine learning system 170 can be coupled to one another (e.g., for performing the methods described herein) via network 160. In some embodiments, network 160 is a private network that provides each component of system architecture 100 with access to each other and other privately available computing devices. Network 160 can include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, and / or any combination thereof. In some embodiments, network 160 is a cloud-based network capable of executing cloud-based functionality (e.g., providing cloud service functionality to one or more devices in the system). Alternatively or additionally, any of the elements of system architecture 100 may be integrated together or otherwise coupled without using network 160 .

[0026] Client device 150 can be or include any personal computer (PC), laptop, mobile phone, tablet, notebook, internet-connected television ("smart TV"), internet-connected media player (e.g., Blu-ray player), set-top box, over-the-top (OTT) streaming device, operator box, etc. Client device can perform cloud-based operations (e.g., utilizing server 120, data storage 140, manufacturing system 102, machine learning system 170, metrology system 110, etc.). Client device 150 can include browser 152, application 154, and / or other tools described and executed by other systems of system architecture 100. In some embodiments, the client device 150 is capable of accessing the manufacturing system 102, the metrology system 110, the data storage 140, the server 120, and / or the machine learning system 170 and communicating (e.g., sending and / or receiving) indications of metrology data, processed data (e.g., enhanced image data, embedding vectors, and the like), processing result data (e.g., critical dimension data, thickness data), and / or inputs and outputs of various processing tools (e.g., the imaging tool 114, the data preparation tool 116, the image enhancement tool 124, the embedding tool 126, the search tool 128, the defect tool 130, and / or the retrieval component 194) at various stages of the processing system architecture 100, as described herein.

[0027] As shown in FIG. 1 , a manufacturing system 102 includes a processing tool 104, a processing program 106, and a process controller 108. The process controller 108 can coordinate the operation of the processing tool 104 to execute one or more processing programs 106. For example, each processing tool can include specialized chambers such as etch chambers, deposition chambers (including chambers for atomic layer deposition, chemical vapor deposition, sputtering chambers, physical vapor deposition, or plasma-enhanced versions thereof), annealing chambers, implant chambers, plating chambers, treatment chambers, and / or the like. In another example, a machine can incorporate a sample transport system (e.g., a selective compliance assembly robot arm (SCARA) robot, a transfer chamber, a front opening pod (FOUP), a side storage pod (SSP), and / or the like) to transport samples between the machine and the processing steps.

[0028] The processing program 106, sometimes referred to as a processing recipe or processing steps, may include various specifications for performing operations by the processing tool 104. For example, the processing program 106 may include processing specifications such as the start-up duration of the processing operation of the machine (e.g., chamber), the processing tool used for the operation, the temperature, flow, pressure, etc., the deposition sequence, and the like. In another example, the processing program may include transfer instructions for transporting the sample to further processing steps or for measurement by the metrology system 110.

[0029] The process controller 108 may include a device designed to manage and coordinate the actions of the processing tool 104. In some embodiments, the process controller 108 is associated with a process recipe or a series of process instructions 106 that, when applied in a designed manner, result in a desired process outcome for substrate processing. For example, a process recipe may be associated with processing a substrate to produce a target process outcome (e.g., critical dimension, thickness, uniformity criteria, etc.).

[0030] As shown in FIG. 1 , metrology system 110 includes imaging tools 114 and data preparation tools 116 . Imaging tools 114 may include various sensors to measure process results (e.g., critical dimensions, thickness, uniformity, etc.) within manufacturing system 102 . For example, imaging tools 114 may include a scanning tunneling microscope (STM) or a scanning electron microscope (SEM). In another example, wafers processed within one or more processing chambers may be used to measure critical dimensions. Imaging tools 114 may also include devices for measuring process results of substrates processed using the manufacturing system. For example, process results such as critical dimensions and thickness measurements (e.g., from etching, deposition, etc.) may be assessed for substrates processed according to process recipes and / or actions executed by process controller 108 . These measurements may also be used to measure chamber conditions throughout the substrate processing sequence.

[0031] The data preparation tools 116 may include processing methods for extracting features and / or generating synthetic / engineered data associated with the data measured by the imaging tools 114. In some embodiments, the data preparation tools 116 may identify correlations, patterns, and / or anomalies in metrology or process performance data. For example, the data preparation tools 116 may perform feature extraction, where the data preparation tools 116 use a combination of metrology data to determine whether a criterion is met. For example, the data preparation tools 116 may analyze multiple data points of associated parameters to determine whether rapid changes occur across multiple processing chambers during a substrate processing sequence. In some embodiments, the data preparation tools 116 perform normalization on various sensor data associated with various processing chamber conditions. Normalization may include processing the input sensor data so that it appears similar across the various chambers and sensors used to acquire the data.

[0032] In some embodiments, the data preparation tool 116 can perform one or more of process control analysis, univariate limit violation analysis, or multivariate limit violation analysis on metrology data (e.g., metrology data obtained by the imaging tool 114). For example, the data preparation tool 116 can perform statistical process control (SPC) by employing statistically based methods to monitor and control the process controller 108. For example, SPC can improve the efficiency and accuracy of substrate processing procedures (e.g., by identifying data points that fall within and / or outside control limits).

[0033] In some embodiments, the extracted features, generated synthetic / engineered design data, and statistical analysis can be used in conjunction with the machine learning system 170 (e.g., for training, validating, and / or testing the machine learning model 190). Additionally and / or alternatively, the data preparation tool 116 can output data to the server 120 for use by any of the image enhancement tool 124, the embedding tool 126, the search tool 128, and the defect tool 130.

[0034] Data storage 140 can be a memory (e.g., random access memory), a drive (e.g., a hard drive, a flash drive), a database system, a cloud-based system, or another type of component or device capable of storing data. Data storage 140 can store one or more historical data 142, including an image repository 144 containing previously processed images (e.g., of substrate defects) and corresponding vectorized image features and metadata 146. The vectorized image data can include feature vectors or embedded data representing imaging data (e.g., image view frames, images acquired using imaging tool 114).

[0035] Server 120 may include one or more computing devices, such as a rack-mount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc. Server 120 may include an image enhancement tool 124, an embedding tool 126, a search tool 128, and / or a defect tool 130. Server 120 may include a cloud server or a server capable of performing one or more cloud-based functions. For example, one or more operations of image enhancement tool 124, embedding tool 126, search tool 128, and / or defect tool 130 may be provided to a remote device (e.g., client device 150) using a cloud environment.

[0036] The image enhancement tool 124 receives image data (e.g., image frames) indicative of substrate processing defects and performs image enhancement on the received image data. In some embodiments, the image enhancement tool 124 executes a filtering process, wherein processing logic processes a selection of received image frames and removes image frames that do not meet specific criteria by identifying features in the images. For example, images of substrate defects may be included in an image collection that includes charts, data sheets, and / or metrology-related images. The filtering process identifies which images depict substrate defects and removes images that do not directly indicate substrate processing defects.

[0037] In some embodiments, image enhancement tool 124 performs a cropping process using one or more received image frames. Cropping involves removing unwanted external areas from a photograph or displayed image. This process typically involves removing some peripheral areas of the image to remove extraneous garbage from the image, improve its framing, change the aspect ratio, or highlight or isolate a subject from its background. This can be performed using image editing software and / or algorithms that mimic image editing programs. For example, image enhancement tool 124 may receive a selection of one or more image frames (e.g., from metering system 110 and / or client device 150) and crop the one or more image frames based on the selection (e.g., generating a second image frame based on cropping the first image frame based on the selection).

[0038] In some embodiments, the image enhancement tool 124 performs a masking process on one or more received image frames. Image masking is a technique used to isolate different portions of an image. For example, image masking can include photo-compositing multiple images, hiding all or part of an applied selective adjustment, performing cropping (e.g., removing a background), adjusting transparency, and / or the like. For example, the image enhancement tool 124 can receive a selection of one or more image frames (e.g., from the metering system 110 and / or the client device 150) and mask the one or more image frames based on the selection (e.g., generating a second image frame by cropping the first image frame based on the selection).

[0039] Embedding tool 126 receives image data comprising one or more image frames (e.g., from metrology system 110) and determines embedding data representing the image data. Embedding tool 126 includes process methods for extracting features in the form of feature data (e.g., feature vectors) and / or generating synthetic / engineered data associated with the data measured by imaging tool 114. In some embodiments, embedding tool 116 can identify correlations, patterns, and / or anomalies in metrology or process performance data. An embedding is a relatively low-dimensional space into which a high-dimensional representation (e.g., an image) can be transformed. The embedding data (e.g., feature vectors) captures the semantics of the received image frames. Embedding tool 126 outputs the embedding data (e.g., for use by search tool 128). The output of the embedding layer can be further passed to other machine learning techniques, such as clustering and k-nearest neighbor analysis.

[0040] The search tool 128 receives the embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., image repository). The search tool 128 performs a proximity search between the received feature embeddings and one or more feature embeddings of the vectorized image and the features and metadata 146 of the historical data 142.

[0041] In some embodiments, the search tool 128 uses vector searching and / or nearest neighbor solution methods to determine a set of close images for the image repository 144. The search tool can identify a vector that is closest to (e.g., most similar to) a received and / or provided feature vector.

[0042] In some embodiments, the search tool 128 scans historical data 142 and retrieves an image repository 144 and vectorized image features and metadata 146. The search tool 128 can utilize and index the vectorized image features to quickly parse feature vectors. In some embodiments, the search tool utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0043] Defect tool 130 receives one or more similar image frames indicating substrate processing defects. Defect tool 130 identifies substrate processing defects based on the selection of similar image frames. Defect tool 130 identifies anomalies in the manufacturing process based on a comparison between the current image and each of the selected similar image frames. In some embodiments, defect tool 138 receives similar images from pattern mapping tool 137 and identifies anomalies based on a sample pattern.

[0044] Defect tool 130 can retrieve failure mode and effect analysis (FMEA) data. FMEA data can include a list of known issues and root causes for a given piece of equipment, each with associated known symptoms. Identified defects and / or similar images received by defect tool 138 are applied to the list of known issues, and a report is generated identifying common causes of the defects. For example, defect tool 130 can identify the defect and identify the operation of the tool, machine, or manufacturing process that corresponds to the identified defect.

[0045] In some embodiments, the defect tool 138 can be used in conjunction with process dependency data to identify tools, machines, or processes operating upstream from the current machine operation being performed on the current sample (e.g., an operation step that occurred before the current manufacturing step of the same manufacturing process). For example, the current sample may have recently undergone a first operation on a first machine. In some embodiments, the defect tool 130 can utilize a combination of process dependency data and failure mode and effects analysis data to identify past operations of the sample, such as a second operation on a second machine or tool.

[0046] Once an anomaly instance is identified, the defect tool 130 may proceed by performing one of the following: modifying at least one of the operation of the machine or the implementation of the process associated with the anomaly instance and / or providing a graphical user interface (GUI) presenting a visual indicator of the machine or process associated with the anomaly instance. The GUI may be sent via the network 160 and presented on the client device 150. In some embodiments, modifying the operation of the machine or the implementation of the process may include sending instructions to the manufacturing execution system 102 to modify the process tool 104, the process program 106, and / or the process controller 108.

[0047] As previously described, some embodiments of the image enhancement tool 124, embedding tool 126, search tool 128, and / or defect tool 130 may use machine learning models to perform the methods described herein. The associated machine learning models may be generated (e.g., trained, validated, and / or tested) using the machine learning system 170. The following example description of the machine learning system 170 will be described in the context of using the machine learning system 170 to generate the machine learning model 190 associated with the embedding tool 126. However, it should be noted that this description is merely an example. Similar processing layers and methods may be used to generate and execute machine learning models associated with the image enhancement tool 124, embedding tool 126, search tool 128, and / or defect tool 130, and the machine learning models may be used to perform the methods described herein.

[0048] The machine learning system 170 may include one or more computing devices, such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a cloud computer, a cloud server, a system stored on one or more clouds, and the like. The machine learning system 170 may include an embedding component 194 and a retrieval component 196. In some embodiments, the embedding component 194 may receive as input one or more image frames indicating substrate processing defects and use the historical data 142 and the trained machine learning model 190 to determine feature embeddings corresponding to the image frames. In some embodiments, the retrieval component 196 may use the trained machine learning model 190 to search the image repository 144 using the vectorized image features and the metadata 146.

[0049] In some embodiments, the machine learning system 170 further includes a server machine 172 and a server machine 180. The server machines 172 and 180 can be one or more computing devices (such as rack-mounted servers, router computers, server computers, personal computers, mainframe computers, laptops, tablet computers, desktop computers, cloud computers, cloud servers, systems stored on one or more clouds, etc.), data storage (e.g., hard drives, memory databases), networks, software components, or hardware components.

[0050] Server machine 172 may include a dataset generator 174 capable of generating datasets (e.g., a set of data inputs and a set of target outputs) for training, validating, or testing machine learning models. Dataset generator 174 may partition historical data 142 into a training set (e.g., 60% of the historical data, or any other portion of the historical data), a validation set (e.g., 20% of the historical data, or some other portion of the historical data), and a test set (e.g., 20% of the historical data). In some embodiments, dataset generator 174 generates multiple sets of training data. For example, one or more sets of training data may include each of the datasets (e.g., the training set, the validation set, and the test set).

[0051] Server machine 180 includes a training engine 182, a validation engine 184, and a testing engine 186. Training engine 182 can train a machine learning model 190 using one or more images from image repository 144, vectorized image features and metadata 146, and / or historical process result data 148 from historical data 142 (from data store 140). In some embodiments, machine learning model 190 can be trained using one or more outputs from data preparation tool 116, image enhancement tool 124, embedding tool 126, search tool 128, and / or defect tool 130. For example, machine learning model 190 can be a hybrid machine learning model that uses image data and / or embedded features (such as feature extraction, mechanical modeling, and / or statistical modeling). Training engine 182 can generate multiple trained machine learning models 190, each corresponding to a different set of features for each training set.

[0052] The validation engine 184 can determine the accuracy of each of the trained machine learning models 190 based on the corresponding set of features of each training set. The validation engine 184 can discard trained machine learning models 190 with an accuracy that does not meet a threshold accuracy. The testing engine 186 can determine the trained machine learning model 190 with the highest accuracy of all the trained machine learning models based on the test (and, optionally, validation) set.

[0053] In some embodiments, training data is provided to train the machine learning model 190, such that the trained machine learning model can receive new inputs including new image data indicative of new substrate processing defects. New outputs can indicate new feature embeddings (e.g., feature vectors). In some embodiments, the training data can be further utilized such that the new outputs further include a selection of similar feature vectors corresponding to images of similar substrate processing defects.

[0054] The machine learning model 190 may refer to a model created by the training engine 182 using a training set comprising data inputs and corresponding target outputs (image frames and corresponding vectorized image features and metadata). Patterns in the dataset that map the data inputs to target outputs (e.g., identifying connections between portions of sensor data and resulting chamber states) may be discovered and provided to the machine learning model 190. The machine learning model 190 may utilize one or more of logistic regression, parsing, decision trees, or support vector machines (SVMs). Machine learning may consist of a single-stage linear or nonlinear operation (e.g., SVM) and / or may be a neural network.

[0055] Embedding component 194 can provide current data (e.g., image frames indicative of substrate processing defects) as input to a trained machine learning model 190 and can run the trained machine learning model 190 on the input to obtain one or more outputs including a set of vectorized image features and metadata. Embedding component 194 can identify confidence data from the outputs, indicating a confidence level for the predicted vectorized image features and metadata. In one non-limiting example, the confidence level is a real number between 0 and 1, inclusive, where 0 indicates no confidence in one or more chamber states, and 1 indicates absolute confidence in the chamber state.

[0056] For purposes of illustration and not limitation, aspects of the present disclosure describe training a machine learning model and using the trained learning model using information related to historical data 142. In other embodiments, a heuristic model or a rule-based model is used to determine chamber status.

[0057] In some embodiments, the functionality of client device 150, server 120, data storage 140, and machine learning system 170 can be provided by a smaller number of machines than shown in FIG1 . For example, in some embodiments, server machines 172 and 180 can be integrated into a single machine, while in other embodiments, server machines 172, 180, and 192 can be integrated into a single machine. In some embodiments, machine learning system 170 can be provided in whole or in part by server 120.

[0058] In general, functions described in one embodiment as being performed by client device 150, data storage 140, metrology system 110, manufacturing system 102, and machine learning system 170 may also be performed on server 120 in other embodiments, as appropriate. Furthermore, functionality attributed to a particular component may be performed by different or multiple components operating together.

[0059] In some embodiments, a "user" may be a single individual. However, other embodiments of the present disclosure encompass entities controlled by multiple users and / or automated sources. For example, a collection of independent users united by a group of administrators may be considered a "user."

[0060] 2 is a block diagram illustrating a substrate defect image indexing and retrieval system 200 in which embodiments of the present disclosure may operate. The substrate defect image indexing and retrieval system 200 may include aspects and / or features of the system architecture 100.

[0061] As shown in FIG. 2 , a substrate defect indexing and retrieval system 200 receives an input image 202. The input image may include one or more image frames indicating substrate processing defects. The input image 202 may include an image of a substrate processing result having substrate processing defects. For example, the input image 202 may include a scanning tunneling microscope (STM) or scanning electron microscope (SEM) image.

[0062] As shown in FIG. 2 , substrate defect indexing and retrieval system 200 includes image enhancement logic, including cropping logic 204 , masking logic 206 , and size detection logic 208 . Cropping logic 204 performs a cropping process with one or more received image frames. Cropping involves removing unwanted external areas from a photograph or displayed image. This process typically involves removing some peripheral areas of the image to remove extraneous garbage from the image, improve its framing, change the aspect ratio, or highlight or isolate a subject from its background. This can be performed using image editing software and / or algorithms that emulate image editing programs. For example, cropping logic 204 may receive a selection of one or more image frames and crop the one or more image frames based on the selection (e.g., generating a second image frame based on cropping a first image frame based on the selection).

[0063] Masking logic 206 performs a masking process with one or more received image frames. Image masking is a technique used to isolate different portions of an image. For example, image masking may include photocompositing multiple images, hiding all or part of an image, applying selective adjustments, performing cropping (e.g., removing a background), adjusting transparency, and / or the like. For example, masking logic 206 may receive a selection of one or more image frames and mask the one or more image frames based on the selection (e.g., generating a second image frame by cropping a first image frame based on the selection).

[0064] Size detection logic 208 processes input image 202 and extracts content from input image 202 indicating an image scale factor associated with input image 202. The image scale factor indicates the relative size of a depiction within the image relative to the size of the image. For example, the image scale factor may include image zoom, a magnification factor, an indicated size of a depicted object, and the like. Size detection logic 208 extracts size identification information (e.g., text) and determines the size of the depicted defect. The defect size is further passed to one or more downstream processes (e.g., embedding logic 210, search logic 212, defect detection logic 214, and the like). Further details regarding size detection logic 208 are discussed in association with FIG. 3 .

[0065] As shown in FIG. 2 , substrate defect indexing and retrieval system 200 includes embedding logic 210. Embedding logic 210 receives image data, including input image 202. Embedding logic 210 includes process methods for extracting features in the form of feature data (e.g., feature vectors) and / or generating synthetic / engineered data associated with data measured by imaging tools. In some embodiments, embedding logic 210 can identify correlations, patterns, and / or anomalies in metrology or process performance data. Embedding is a relatively low-dimensional space into which a high-dimensional representation (e.g., an image) can be transformed. Embedding data (e.g., feature vectors) captures the semantics of the received image frame. Embedding logic 210 outputs embedded data (e.g., for use by search tool 128). The output of the embedding layer can be further processed using other machine learning techniques, such as clustering and k-nearest neighbor analysis.

[0066] As shown in FIG. 2 , the substrate defect indexing and retrieval system 200 includes search logic 212. The search logic 212 receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., an image repository). The search logic 212 performs proximity searches between the received feature embeddings and one or more feature embeddings of the vectorized image and features and metadata of previously processed images of other substrate processing defects.

[0067] In some embodiments, the search logic 212 uses vector search and / or nearest neighbor solution methods to determine a set of close images of the image repository. The search tool can identify the vector that is closest to (e.g., most similar to) a received and / or provided feature vector.

[0068] In some embodiments, search logic 212 scans the data structure and retrieves the image repository, along with the corresponding vectorized image features and metadata. Search logic 212 may employ an indexing method for vectorized image features to rapidly parse feature vectors. In some embodiments, the search tool utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0069] As shown in FIG. 2 , the substrate defect indexing and retrieval system 200 includes defect detection logic 214. The defect detection logic 214 receives one or more similar image frames indicating substrate processing defects. Based on the selection of similar image frames, the defect detection logic 214 identifies substrate processing defects. The defect detection logic 214 identifies anomalies in the manufacturing process based on a comparison between the current image and each of the selected similar image frames. In some embodiments, the defect detection logic 214 receives similar images from the search logic 212 and identifies anomalies based on the similar images.

[0070] Defect detection logic 214 may retrieve failure mode and effects analysis (FMEA) data. FMEA data may include a list of known issues and root causes for a given piece of equipment, each with associated known symptoms. Identified defects and / or similar images received by defect detection logic 214 are applied to the list of known issues, and a report is generated identifying common causes of the defects. For example, defect detection tool 214 may identify defects and identify the operation of the tool, machine, or manufacturing process that corresponds to the identified defect.

[0071] In some embodiments, the defect detection logic 214 can use process dependency data with the current machine operation being performed on the current sample to identify tools, machines, or processes operating upstream (e.g., operations that occurred before the current manufacturing step of the same manufacturing process). For example, the current sample may have recently undergone a first operation on a first machine. In some embodiments, the defect detection tool 214 can utilize a combination of process dependency data and failure mode and effects analysis data to identify past operations of the sample, such as a second operation on a second machine or tool.

[0072] Once an anomaly instance is identified, the defect tool 130 may proceed by performing one of the following: modifying at least one of the implementation of the operation or process of the machine associated with the anomaly instance and / or providing a graphical user interface (GUI) 216 presenting a visual indicator of the machine or process associated with the anomaly instance. The GUI may be sent via the network 160 and presented on the client device 150. In some embodiments, modifying the implementation of the operation or process of the machine may include sending instructions to the manufacturing execution system 102 to modify a processing entity of the manufacturing system 102 (e.g., the processing tool 104, the processing program 106, and / or the processing controller 108 of FIG. 1 ).

[0073] In some embodiments, the search logic 212 outputs a selection of similar vectorized image features or images having similar vectorized image features as the input image 202. The images may be sent directly to the GUI 216 without identifying specific defects. For example, the GUI 216 may include a graphic representation of the locations of images that have been identified as similar to the input image 202.

[0074] FIG3 illustrates a block diagram of a defect sizing system 300 according to aspects of the present disclosure. One or more features discussed in connection with FIG3 may be performed by the size detection logic 208 of FIG2 . As shown in FIG3 , the defect sizing system 300 may include receiving an input image 302 , line masking logic 304 , line removal logic 306 , text recognition logic 308 , and / or post-processing logic 310 . The input image 302 may include one or more image frames indicating substrate processing defects. The input image 302 may include an image of a substrate processing result having substrate processing defects. For example, the input image 302 may include a scanning tunneling microscope (STM) or scanning electron microscope (SEM) image.

[0075] Line masking logic 304 determines the placement of edges within the image view frame. Line masking logic may determine the boundaries or global edges of the image view frame. For example, some images may include vertical or horizontal lines at the edges of the image view frame. In some embodiments, line masking logic 304 performs gamma correction thresholding to identify one or more edges within the image view frame. For example, vertical and / or horizontal lines may be used within the image view frame to identify data stored within the image, such as, for example, text overlaid on the image. In some embodiments, line masking logic 304 uses segmentation to perform localization and generate a mask for inpainting the image view frame.

[0076] Line removal logic 306 receives line mask data from line masking logic 304. Line removal logic 306 applies image enhancement methods (e.g., cropping, masking) to portions of the image viewport based on the detected lines. For example, an image may include a brand logo or other artificial markings identified using line masking logic 304. Line removal logic 306 may apply a mask to remove the lines detected by line masking logic 304.

[0077] Text recognition logic 308 recognizes text within the image view frame. For example, the image may include a relative image zoom factor, such as a ruler indicating a distance represented on the image. Text recognition logic 308 may recognize the number and units of a given image zoom factor (e.g., magnification, relative depiction distance, etc.). Text recognition logic 308 may isolate the text from the rest of input image 302.

[0078] Post-processing logic 310 takes action based on the recognized text. For example, the post-processing log may perform data cleansing and normalization. This may include providing a clean depiction of the area of ​​input image 302 recognized by text recognition logic 308. Post-processing logic 310 may further include making size data available to other processes in the system. In some embodiments, post-processing logic 310 determines the size of the defect based on scaling and provides the defect size (e.g., as metadata) to further embedding and / or classification processes associated with input image 302.

[0079] FIG4 is a block diagram illustrating a process 400 for training a machine learning model to generate an output, according to certain embodiments. Process 400 includes receiving training data in the form of input images 402. Input images 402 may include one or more image frames indicating substrate processing defects. Input images 402 may include images of substrate processing results with substrate processing defects. For example, input images 402 may include scanning tunneling microscope (STM) or scanning electron microscope (SEM) images.

[0080] As shown in FIG. 4 , process 400 includes cropping augmentation of an input image 402. A copy of the input image 402 may undergo local cropping 404, and a copy of the input image 402 may undergo global cropping 406. The local cropping may be associated with a selection of the input image 402 set within the selection of input images 402. The locally cropped image serves as input to a student model 408. The globally cropped image serves as input to a teacher model 410. The student model outputs a student model prediction 412. The student model prediction 412 and / or the teacher model prediction 414 include a profile (e.g., a distribution) indicating different embedding vectors corresponding to the input image 402 and corresponding confidence levels (e.g., probabilities) associated with each of the outputs.

[0081] In operation, an example training process may include processing logic receiving a first image frame (e.g., input image 402) indicating a substrate processing defect. The processing logic further generates a second image frame by cropping (e.g., global crop 406) a first region of the first image frame. The processing logic further generates a third image frame by cropping (e.g., local crop 404) a second region of the first image frame. The first region (associated with global crop 406) includes the second region (associated with local crop 404). The processing logic uses the second image frame as input to a first machine learning (ML) model (e.g., teacher model 410). The processing logic obtains one or more outputs of the first ML model. One or more outputs (teacher model predictions 414) indicate a first feature vector corresponding to the second image frame. The processing logic uses the third image frame as input to a second ML model (e.g., student model 408). The processing logic obtains one or more outputs of the second ML model (e.g., student model predictions 412). The one or more outputs indicate a second feature vector corresponding to the third image frame. Processing logic updates (eg, model tunes 416 ) one or more parameters of at least one of the first ML model or the second ML model based on a comparison between the one or more outputs of the first ML model and the one or more outputs of the second ML model.

[0082] In some embodiments, the student model 408 may be a machine learning model that is similar to the trained teacher model 410 but contains fewer layers and / or nodes than each of the trained teaching models 410, resulting in a more compact machine learning model. In some embodiments, multiple student models 408 may be trained, where each student model 408 may be trained to predict embedding vectors for a different subset of the input images 402 within the input image cluster for which the teaching model 410 is trained to output error predictions. In some embodiments, a different learning model 408 may be trained for each cluster of input images 402 and / or substrate processing defects. Each teacher model may then be used to train multiple student models.

[0083] In some embodiments, model tuning 416 includes determining an error or result of a loss function, such as, for example, a hierarchical loss, which is back-propagated to the student model 408. The hierarchical loss represents the difference between the probability of an error (from the student model) and the probability of the perceived truth (from the teaching model). For example, if the teacher model 410 determines that a first prediction has a first probability and the student model 408 determines that the first prediction has a second probability, the error is related to the difference between the two probabilities. In some embodiments, the hierarchical loss function can be a categorical cross entropy function, a Kullback-Leibler divergence function, or any other suitable loss function.

[0084] In some embodiments, training can be performed by feeding input images 402 into the machine learning model one at a time. In some embodiments, after one or more rounds of training, processing logic can determine whether a stopping criterion has been met. The stopping criterion can be a target level of accuracy, a target number of processed images from the training dataset, a target change in a parameter from one or more previous data points, a combination thereof, and / or other criteria. In one embodiment, the stopping criterion is met when at least a minimum number of data points have been processed and the loss value has stabilized and / or stopped decreasing. The loss value can represent the sum of errors in the machine learning model. For example, the loss value can represent the sum of deltas between the modeled value and the actual value. In one embodiment, the stopping criterion is met if the accuracy of the machine learning model has ceased to improve. If the stopping criterion has not been met, further training is performed. If the stopping criterion has been met, training can be completed. Once the machine learning model has been trained, a retained portion of the training dataset can be used to test the model.

[0085] FIG5 illustrates a model training workflow 505 and a model application workflow 517 for substrate processing result prediction according to aspects of the present disclosure. In some embodiments, the model training workflow 505 may be executed on a server, which may or may not include a process defect image indexing and retrieval application, and provide the trained model to the process defect image indexing and retrieval application, which may execute the model application workflow 517. The model training workflow 505 and the model application workflow 517 may be performed by processing logic executed by a processor of a computing device (e.g., server 120 in FIG1 ). One or more of these workflows 505 and 517 may be implemented, for example, by a processing device implementing one or more machine learning modules and / or other software and / or firmware executed on the processing device.

[0086] The model training workflow 505 is used to train one or more machine learning models (e.g., regression models, boosted regression models, principal component analysis models, deep learning models, visual transformers) to perform one or more tasks associated with a process outcome predictor (e.g., feature extraction, image retrieval), such as determining, predicting, or modifying. The model application workflow 517 is used to apply one or more trained machine learning models to perform tasks such as determining and / or tuning image data (e.g., image frames indicative of substrate processing defects). One or more machine learning models can receive image data (e.g., image frames indicative of substrate processing defects).

[0087] Various machine learning outputs are described herein. A specific number and arrangement of machine learning models are described and illustrated. However, it should be understood that the number and type of machine learning models used, as well as the arrangement of such machine learning models, can be modified to achieve the same or similar end results. Thus, the described and illustrated arrangements of machine learning models are merely examples and should not be construed as limiting.

[0088] In some embodiments, one or more machine learning models are trained to perform one or more of the following tasks. Each task can be performed by a separate machine learning model. Alternatively, a single machine learning model can perform each of the tasks or a subset of the tasks. Additionally or alternatively, different machine learning models can be trained to perform different combinations of tasks. In one example, one or more machine learning models can be trained, where the trained machine learning (ML) model is a single shared neural network with multiple shared layers and multiple higher-order different output layers, where each of the output layers outputs a different prediction, classification, recognition, etc. The tasks that the one or more trained machine learning models can be trained to perform are as follows: a. Feature Extractor 567 - As previously discussed, the feature extractor receives image data (e.g., raw and / or enhanced image frames indicative of substrate processing defects). The feature extractor includes process methods for extracting features in the form of feature data (e.g., feature vectors) and / or generating synthetic / engineered data associated with the data measured by the imaging tool. In some embodiments, the feature extractor can identify correlations, patterns, and / or anomalies in metrology or process performance data. An embedding is a relatively low-dimensional space into which a high-dimensional representation (e.g., an image) can be transformed. The embedding data (e.g., feature vectors) captures the semantics of the received image frame. b. Image Retriever 564 - The image retriever receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., from an image repository). The image retriever performs a proximity search between the received feature embeddings and one or more feature embeddings and features and metadata of the vectorized image. In some embodiments, the image retriever uses vector search and / or nearest neighbor solution methods to determine a set of close images (e.g., within a threshold proximity). In some embodiments, the image retriever utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0089] To complete the training, processing logic inputs the training dataset 536 into one or more untrained machine learning models. (See FIG. 4 for further details on training.) Before inputting the first input into the machine learning model, the machine learning model may be initialized. Processing logic trains the untrained machine learning model based on the training dataset to generate one or more trained machine learning models that perform the various operations described above.

[0090] Once one or more trained machine learning models 538 are generated, they can be stored in a model store 545 and added to a process defect image indexing and retrieval application. The process defect image indexing and retrieval application can then use the one or more trained ML models 538 and additional processing logic to implement an automated model, minimizing or even eliminating, in some instances, manual user input of information.

[0091] In the model application workflow 517, according to one embodiment, input data 562 (e.g., an image frame indicating a substrate processing defect) may be used as input to a feature extractor 567, which may include a trained machine learning model. Based on the input data 562, the feature extractor 567 outputs feature data 569 and metadata representing the input data 562. The feature data 569 is input to an image retriever 564, which may include a trained machine learning model. Based on the feature data 569, the image retriever 564 identifies one or more other images (e.g., similar image data 566) that are similar to the input data 562.

[0092] Figures 6A-6B illustrate a model architecture 600 for substrate defect image indexing and retrieval according to aspects of the present disclosure. One or more ML models may be implemented using the model architecture 600. Typically, the model architecture 600 consists of an embedding layer 608, an encoder 614, and a final header classifier 616. Initially, the image is subdivided into non-overlapping patches. Each patch is treated as an independent token by the architecture. For an image of size c × h × w (where h is height, w is width, and c represents the number of channels), a patch of each dimension × p × p is extracted. This forms a sequence of patches (x 1 , x 2 ,…, xn ) of length n, where n= In some embodiments, the patch size p is selected to be 16×16 or 32×32.

[0093] As shown in FIG6A , at 604, an input image 602 is divided (e.g., flattened) into independent image patches. For example, the input image 602 is divided into a fixed number of equally sized patches or embedding tokens. The input image can be converted (e.g., flattened) into a sequence of token embeddings 606 indicating the content of the image patches. In some embodiments, the model architecture uses a constant latent vector size n across all layers, and the patches are flattened and mapped to n dimensions using a linear embedding layer 608 using a trainable linear projection.

[0094] Before feeding the patch sequence into the encoder 614, it is linearly projected into a vector of model dimension d using the learned embedding matrix. The embedded representation is then concatenated with a learnable classification token for performing the classification task. The embedded image patches are treated by the converter as a collection of patches without any notion of order. To preserve the spatial arrangement of the patches as in the original image, position information 610 is encoded and appended to the patch representation 612 (e.g., a linear embedding representing the content of the corresponding image patch). The resulting embedding sequence for a patch with token 0 is given by:

[0095] The resulting sequence of embedded patches z 0 is passed to the transformer encoder 614. As shown in Figure 6B, the encoder 614 consists of L equal layers. Each has two main sub-components: (1) a multihead self-attention block (MSA) 656, and (2) a fully connected feed-forward dense block (MLP) 660. Each of the two sub-components of the encoder uses residual skip connections and is preceded by a normalization layer (LN) (e.g., normalization layer 654 and normalization layer 658). At the last layer of the encoder 614, we take the first element in the sequence and pass it to the outer head classifier for predicting the value of the vector generated by The category tag represented by .

[0096] The MSA block 656 in the encoder 614 is the central component of the converter. The MSA block 656 determines the relative importance of a single patch embedding relative to other embeddings in the sequence. This block has four layers: a linear layer, a self-attention layer, a concatenation layer (which concatenates the outputs of multiple attention heads), and a final linear layer. At a high level, attention can be represented by attention weights, which are calculated by taking the weighted sum of all values ​​in the sequence z. The MSA block employs an attention function that maps query and key-value pairs to a set of outputs, where query, key, value, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is a function of the compatibility of the query with the corresponding key. The results of all attention heads are concatenated and then projected to the desired dimension via a feed-forward layer with learnable weights. The MLP 616 makes a class prediction 618 based on the data received from the encoder 614.

[0097] FIG7 depicts a flow chart of an example method 700 for substrate defect image indexing and retrieval, according to some embodiments of the present disclosure. Method 700 is performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, etc.), software (such as running on a general-purpose computer system or a dedicated machine), or any combination thereof. In one embodiment, the method is performed using server 120 and trained machine learning model 190 of FIG1 . In some other embodiments, one or more blocks of FIG7 may be performed by one or more other machines not depicted in the figure.

[0098] Method 700 may include receiving image data (e.g., associated with substrate processing defects) and processing the received image data using the trained machine learning model 190. The trained model may be configured to generate feature embeddings for the images based on the image data. Method 700 further includes identifying similar images that may show similar defects.

[0099] At block 702, processing logic stores a plurality of feature vectors in a data storage device, the feature vectors representing previously processed image frames corresponding to various substrate processing defects. At block 704, processing logic receives first image data, the first image data comprising one or more image frames indicative of a first substrate processing defect. The first image data may include one or more image frames indicative of a substrate processing defect. The input image may include an image of a substrate processing result having the substrate processing defect. For example, the input image may include a scanning tunneling microscope (STM) or scanning electron microscope (SEM) image.

[0100] In some embodiments, processing logic receives a first selection of a first image frame of first image data and generates a second image frame by cropping an area of ​​the first image frame based on the first selection. A first feature vector is determined using the second image frame. In some embodiments, processing logic receives a first selection of a first image frame of first image data. Processing logic generates the second image frame by masking an area of ​​the first image frame based on the first selection. The first feature vector is determined using the second image frame.

[0101] In some embodiments, processing logic extracts a selection of text from a first image frame of first image data. The processing logic further determines an image scaling factor associated with the first image frame. The processing logic further determines a size associated with the first substrate processing defect based on the image scaling factor. The selection of the plurality of feature vectors is determined using the size.

[0102] At block 706, processing logic determines a first feature vector corresponding to the first image data. In some embodiments, processing logic further divides a first image frame of the first image data into a set of image patches corresponding to the first image frame. Processing logic further determines a set of linear embeddings corresponding to the content of each image patch in the set of image patches. Processing logic further determines a set of position embeddings corresponding to the relative position of each image patch in the set of image patches. The first feature vector is determined based on the set of linear embeddings and the set of position embeddings.

[0103] At block 708, processing logic determines a selection of the plurality of feature vectors based on proximity between the first feature vector and each of the selection of the plurality of feature vectors. At block 710, processing logic determines second image data, the second image data comprising one or more image frames corresponding to the selection of the plurality of embedding vectors. The processing logic receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., an image repository). The processing logic performs a proximity search between the received feature embeddings and one or more feature embeddings and features and metadata of the vectorized image. In some embodiments, the processing logic uses vector search and / or nearest neighbor solution methods to determine a set of close images (e.g., within a threshold proximity). In some embodiments, the processing logic utilizes Euclidean distance and / or cosine similarity to determine the distance between feature vectors.

[0104] At block 712, processing logic optionally performs an action based on determining the second image data. In some embodiments, processing logic optionally prepares the second image data for presentation on a graphical user interface (GUI). For example, the second image data may be a collection of image frames similar to the first image frames and indicating similar substrate processing defects. In another example, the second image data may be displayed on the GUI by displaying an indication of the first substrate processing defect.

[0105] In some embodiments, processing logic optionally modifies the operation of a processing chamber and / or processing tool based on the second image data. For example, the processing logic may determine a substrate processing defect based on the second image data. The processing logic may further determine an abnormal instance of the manufacturing process associated with the first substrate processing defect. The processing logic may further send instructions (e.g., to execute corrective actions associated with the manufacturing process equipment) to one or more process controllers to modify one or more operations of the processing equipment associated with the abnormal instance (e.g., to modify a process recipe and / or process parameters, final substrate processing of one or more process tools and / or processing chambers, initiate preventative maintenance associated with the one or more process chambers and / or processing tools, etc.).

[0106] FIG8 depicts a block diagram of an example computing device 800 operating in accordance with one or more aspects of the present disclosure. In various illustrative examples, various components of computing device 800 may represent various components of client device 150, metering system 110, server 120, data storage 140, and machine learning system 170 shown in FIG1 .

[0107] The example computing device 800 can be connected to other processing devices in a LAN, intranet, extranet, and / or the Internet. The computing device 800 can operate in a server capacity in a client-server network environment. The computing device 800 can be a personal computer (PC), a set-top box (STB), a server, a network router, a switch or bridge, or any device capable of executing a set of instructions (serial or otherwise) that specify actions to be taken by that device. Furthermore, while a single example computing device is shown, the term "computer" should be construed to include any collection of computers that, individually or in combination, execute an instruction set (or multiple instruction sets) to perform any one or more of the methodologies discussed herein.

[0108] The example computing device 800 may include a processing device 802 (also referred to as a processor or CPU), a main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), etc.), a static memory 806 (e.g., flash memory, static random access memory (SRAM), etc.), and an auxiliary memory (e.g., a data storage device 818), which may communicate with each other via a bus 830.

[0109] Processing device 802 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, processing device 802 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing another instruction set, or a processor implementing a combination of instruction sets. Processing device 802 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. According to one or more aspects of the present disclosure, processing device 802 may be configured to execute instructions that implement method 700 shown in FIG. 7 .

[0110] The example computing device 800 may further include a network interface device 808 communicatively coupled to a network 820. The example computing device 800 may further include a video display 810 (e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device 812 (e.g., a keyboard), a cursor control device 814 (e.g., a mouse), and an audio signal generating device 816 (e.g., a speaker).

[0111] The data storage device 818 may include machine-readable storage media (or more specifically, non-temporary machine-readable storage media) 828 on which one or more sets of executable instructions 822 are stored. According to one or more of the states disclosed herein, the executable instruction 822 may comprise an executable instruction associated with performing the method 700 shown in FIG.

[0112] The executable instruction 822 may also be fully or at least partially stationed within the main memory 804 and / or within the processing device 802 during its execution by resorting to exemplary computing devices 800 , main memory 804 , and processing devices 802 that also constitute computer-readable storage media. The executable command 822 may further be sent or received on the network via the network interface device 808 .

[0113] Notwithstanding the diagramming of computer-readable storage media 828 as a single media in Figure 8, the term "computer-readable storage media" shall be considered to include a single media or multiple media of one or more sets of storage operation instructions (e.g., centralized or distributed databases, and / or associated cache memory and servers). The term "computer-readable storage media" shall also be considered to include any media capable of storing or encoding a set of instructions for use by a machine and causing the machine to perform any one or more of the methods described herein. The term “computer-readable storage media” shall thereby be considered to include, but not limited to, solid-state memory, and optical and magnetic media.

[0114] Some parts of the detailed description above have been provided with respect to algorithmic and symbolic representations of operations on data bits within computer memory. Such algorithmic descriptions and representations are means used by those skilled in the field of data processing to most effectively communicate the substance of their work to other skilled persons in the art. Algorithms are in this paper and are generally considered to be self-consistent sequences of steps leading to the desired outcome. The steps are required physical amount of physical manipulation of those. This usually, although not mandatory, these quantities take the form of electrical or magnetic signals capable of being stored, transmitted, combined, compared, and otherwise manipulated. Mainly for general reasons, it is convenient to refer to such signals as bits, values, elements, symbols, characters, terms, numbers, or the like sometimes confirmed.

[0115] However, it should be remembered that all such and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to such quantities. Unless specifically stated otherwise, as will be apparent from the following discussion, it is to be understood that throughout the description, the use of terms such as "identifying", "determining", "storing", "adjusting", "causing", "returning", "comparing", "creating", "stopping", "loading", "copying", "throwing", "replacing", "performing", or the like refers to actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in registers and memories of the computer system into other data similarly represented as physical quantities in memories or registers of the computer system or other such information storage, transmission, or display devices.

[0116] Examples of the present disclosure are also directed to an apparatus for performing the methods described herein. This apparatus may be specially constructed for the required purposes, or it may be a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including optical disks, compact disk read-only memory (CD-ROM), and magnetic optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disk storage media, optical storage media, flash memory devices, other types of machine-accessible storage media, or any type of medium suitable for storing electronic instructions, each coupled to the computer system bus.

[0117] The methods and displays provided herein are not inherently related to any particular computer or other device. A variety of general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized devices to perform the desired method steps. The structure required for various such systems will be apparent as elucidated in the following description. Additionally, the scope of the present disclosure is not limited to any particular programming language. It will be understood that a variety of programming languages may be used to implement the teachings of the present disclosure.

[0118] It will be understood that the above description is intended to be illustrative and not restrictive. Numerous other implementation examples will become apparent to those skilled in the art upon reading and understanding the above description. Although this disclosure describes specific embodiments, it will be appreciated that the systems and methods of this disclosure are not limited to the embodiments described herein, but may be practiced with modifications within the scope of the appended claims. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense. The scope of this disclosure should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0119] 100: System Architecture 102: Manufacturing System 104: Processing Tools 106:Handler 108: Processing Controller 110: Metering system 114: Imaging Tools 116: Data Preparation Tools 120: Server 124: Image Enhancement Tools 126: Embedded Tools 128: Search Tools 130: Defect Tools 140: Data Storage 142: Historical Data 144: Image Repository 146: Vectorized Image Features and Metadata 150: Client device 152: Browser 154: Application 160: Internet 170: Machine Learning Systems 172: Server Machine 174:Dataset Generator 180: Server Machine 182: Training Engine 184:Verification Engine 186:Test Engine 190: Machine Learning Model 192: Server Machine 194:Embedded components 196: Retrieve Components 200: Substrate defect image indexing and retrieval system 202: Input image 204: Clipping Logic 206: Masking Logic 208: Size detection logic 210: Embedded Logic 212: Search Logic 214: Defect Detection Logic 216: Graphical User Interface (GUI) 300: Defect size determination system 302: Input image 304: Line Shielding Logic 306: Line removal logic 308: Text Recognition Logic 310: Post-processing logic 400: Process 402: Input image 404: Partial cropping 406: Global Clipping 408: Student Model 410: Teacher Model 412: Student Model Prediction 414: Teacher Model Prediction 416: Model Tuning 505: Model training workflow 517: Model Application Workflow 536: Training dataset 538:Trained Machine Learning Model 545:Model Storage 562: Input data 564: Image Retrieval 566: Similar video material 567: Feature Extractor 569: Feature Information 600: Model Architecture 602: Input image 606:Token Embedding Sequence 608: Linear Embedding Layer 610: Location Information 612: Patch indication 614: Encoder 616: Final Header Classifier 618: Category Prediction 654: Standardization layer 656: Multi-head Self-Attention Block (MSA) 658: Standardization layer 700: Method 702: Block 704: Block 706: Block 708: Block 710: Block 712: Block 800: computing device 802: Processing device 804: Main memory 806: Static Memory 808: Network interface device 810: Video Display 812: Alphanumeric input device 814: Cursor control device 816: Sound signal generating device 818: Data storage device 820: Internet 822: executable instructions 828: Machine-readable storage media 830: Bus

[0120] Domestic storage information (please note the order of storage institution, date, and number) none Overseas deposit information (please note the order of deposit country, institution, date, and number) none

Claims

1. A method for modeling semiconductor defect image indexing and retrieval, comprising the steps of: storing a plurality of feature vectors in a data storage device, the plurality of feature vectors representing previously processed image frames corresponding to various substrate processing defects; receiving first image data by a processing device, the first image data including one or more image frames indicating a first substrate processing defect; determining a first feature vector corresponding to the first image data by the processing device; determining a selection of the plurality of feature vectors by the processing device based on a proximity between the first feature vector and each of the selections of the plurality of feature vectors; determining second image data different from the first image data by the processing device, the second image data including one or more previously processed image frames from the data storage device and corresponding to embedded data associated with the selection of the plurality of feature vectors, wherein the second image data indicates one or more additional substrate processing defects corresponding to the first substrate processing defect; and performing an action by the processing device based on determining the second image data, wherein the action includes identifying the first substrate processing defect based on comparing the first image data with the second image data.

2. The method as described in claim 1 further includes the following steps: identifying a processing defect of the first substrate by the processing device based on the second image data, wherein the action is further based on an identifier of the processing defect of the first substrate.

3. The method as described in claim 2 further includes the following steps: The processing device identifies an instance of anomaly in a manufacturing process associated with a processing defect in the first substrate. And the processing device causes a corrective action associated with the manufacturing processing equipment to be performed based on the anomaly instance.

4. The method as described in claim 1 further includes the following steps: preparing one or more images of the second image data by the processing device for presentation on a graphical user interface (GUI).

5. The method as described in claim 1, further comprising the steps of: receiving a first selection of a first image frame of the first image data by the processing device; and generating a second image frame by the processing device by cropping a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.

6. The method as described in claim 1, further comprising the steps of: receiving a first selection of a first image frame of the first image data by the processing device; and generating a second image frame by the processing device by occluding a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.

7. The method as described in claim 1, further comprising the steps of: extracting a selection of text from a first image frame of the first image data by the processing device; determining an image scaling factor associated with the first image frame by the processing device; and determining a size associated with the first substrate processing defect by the processing device based on the image scaling factor; wherein the selection of the plurality of feature vectors is further determined using the size.

8. The method as described in claim 1, further comprising the steps of: dividing a first image frame of the first image data into a set of image patches corresponding to the first image frame by the processing apparatus; determining a linear embedding set of content corresponding to each image patch in the set of image patches by the processing apparatus; and determining a positional embedding set of a relative position corresponding to each of the image patches in the set of image patches by the processing apparatus, wherein the first feature vector is determined based on the linear embedding set and the positional embedding set.

9. A method for modeling semiconductor defect image indexing and retrieval, comprising the steps of: receiving, by a processing device, a first image frame indicating a substrate processing defect; generating, by the processing device, a second image frame by cropping a first region of the first image frame; generating, by the processing device, a third image frame by cropping a second region of the first image frame, wherein the first region includes the second region; using the second image frame as input to a first machine learning (ML) model; obtaining one or more outputs of the first ML model, the one or more outputs indicating a first feature vector corresponding to the second image frame; using the third image frame as input to a second ML model; obtaining one or more outputs of the second ML model, the one or more outputs indicating a second feature vector corresponding to the third image frame; and updating one or more parameters of at least one of the first ML model or the second ML model based on a comparison between the one or more outputs of the first ML model and the one or more outputs of the second ML model.

10. The method as described in claim 9, wherein: The one or more outputs of the first ML model further indicate a confidence level associated with the first feature vector; and the one or more outputs of the second ML model further indicate a confidence level associated with the second feature vector.

11. The method of claim 9 further includes the following steps: dividing the second image frame into a set of image patches corresponding to the second image frame using the first ML model; determining a set of linear embeddings of content corresponding to each image patch in the set of image patches using the first ML model; and determining a set of positional embeddings of a position corresponding to a position of a corresponding image patch in the set of image patches using the first ML model, wherein the first feature vector is determined based on the set of linear embeddings and the set of positional embeddings.

12. The method as described in claim 9, wherein: The outputs of the first ML model include (i) a first prediction set and (ii) a first probability set, each corresponding to a prediction of the first prediction set; and the outputs of the second ML model include (i) a second prediction set and (ii) a second probability set, each corresponding to a prediction of the second prediction set.

13. The method as described in claim 9, wherein at least one of the first ML model or the second ML model includes a visual transducer (ViT).

14. A non-transitory machine-readable storage medium for modeling semiconductor defect image indexing and retrieval, the non-transitory machine-readable storage medium containing instructions that, when executed by a processing device, cause the processing device to perform operations including the following steps: storing a plurality of feature vectors in a data storage device, the feature vectors representing previously processed image frames corresponding to various substrate processing defects; receiving first image data, the first image data including one or more image frames indicating a first substrate processing defect; determining a first feature vector corresponding to the first image data; determining the selection of the plurality of feature vectors based on a proximity between the first feature vector and each of the selections of the plurality of feature vectors; Determine a second image data that is different from the first image data, the second image data including one or more previously processed image frames from the data storage device and corresponding to embedded data associated with the selection of the plurality of feature vectors, wherein the second image data indicates one or more additional substrate processing defects that correspond to the first substrate processing defect; and perform an action based on determining the second image data, wherein the action includes identifying the first substrate processing defect by comparing the first image data with the second image data.

15. The non-transitory machine-readable storage medium as described in claim 14, further comprising the steps of: identifying a first substrate processing defect based on the second image data, wherein the action is further based on an identifier of the first substrate processing defect.

16. The non-transitory machine-readable storage medium as described in claim 15, further comprising the steps of: identifying an anomaly instance in a manufacturing process associated with the first substrate processing defect; and causing a corrective action associated with the manufacturing processing equipment to be performed based on the anomaly instance.

17. The non-transitory machine-readable storage medium as described in claim 14, further comprising the steps of: preparing one or more images of the second image data for presentation on a graphical user interface (GUI).

18. The non-transitory machine-readable storage medium as described in claim 14, the operations further comprising: receiving a first selection of a first image frame of the first image data; and generating a second image frame by cropping or occluding at least one region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.

19. The non-transitory machine-readable storage medium as described in claim 14, further comprising the steps of: extracting a selection of text from a first image frame of the first image data; determining an image scaling factor associated with the first image frame; and determining a size associated with the first substrate processing defect based on the image scaling factor, wherein the selection of the plurality of feature vectors is further determined using the size.

20. The non-transitory machine-readable storage medium as described in claim 14, further comprising the steps of: dividing a first image frame of the first image data into a set of image patches corresponding to the first image frame; determining a linear embedding set of content corresponding to each image patch in the set of image patches; and determining a positional embedding set of a position corresponding to a position of each of the image patches in the set of image patches, wherein the first feature vector is determined based on the linear embedding set and the positional embedding set.

Images