Substrate defect detection and comparison

Abstract

Various examples described herein include image processing tasks for image data for defect detection and comparison of features on a substrate. A backbone network (common backbone) based on at least one of depth convnet and transformer is arranged to convert various types of raw image data into features, which in turn can be used by a smaller neural network to perform a final computation of a particular task. The smaller neural network performs, for example, final defect detection, die-to-die image comparison, anomaly detection, and customer-specific tasks in manufacturing facilities. The public backbone network may initially be trained using self-supervised learning based on the raw image data, and then the final application of the task-specific network may be trained using transfer learning. Other systems and methods are also disclosed.

Description

[0001] Claiming priority

[0002] This patent application claims priority to U.S. Provisional Application Serial No. 63 / 371,806, filed on August 18, 2022, entitled “SUBSTRATE DEFECT-DETECTION AND COMPARISON”; the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed subject matter generally relates to the field of substrate metrology and / or inspection equipment used in manufacturing facilities. More specifically, in various embodiments, the disclosed subject matter relates to the task of image processing of image data for defect detection and comparison of features on a substrate. Background Art

[0004] State-of-the-art algorithms for image processing tasks are used in tasks such as facial recognition, human pose estimation, semantic segmentation (e.g., assigning a label to each pixel in an image or dividing a digital image into multiple image segments), and template matching are currently used in various social media and other platforms. These algorithms typically rely on some type of deep convolutional neural network (convnet or CNN). These convnets can outperform state-of-the-art algorithms from 20 years ago.

[0005] However, there are significant engineering challenges that prevent the adoption of these high-performance algorithms in environments such as images captured by metrology and inspection of substrates used in, for example, semiconductor manufacturing facilities. Challenges include throughput of substrates and acquiring sufficient amounts of training data from substrates.

[0006] The throughput problem (e.g., substrates processed per hour) is due to the time required for computation of image processing tasks such as anomaly detection, classification, and feature localization of various types of features formed on substrates (e.g., 3D bump patterns, watermarks, etc.) and devices (e.g., patterns on various layers of memory devices or integrated circuit devices). The acquisition of training data is based on current classifications of limited data types obtained by, for example, a single device manufacturer.

[0007] This paper addresses the throughput and training data issues. Summary of the invention

[0008] This document describes, inter alia, a backbone network (common backbone) arranged to convert various types of raw image data into features that in turn can be used by smaller neural networks to perform final computations for specific tasks in a manufacturing facility. The common backbone can be based on, for example, a machine learning based network, such as a deep convnet based network or a transformer based network. The smaller neural networks perform, for example, final defect detection, die-to-die image comparison, anomaly detection, and customer specific tasks in a manufacturing facility. The common backbone network can be initially trained using self-supervised learning based on raw image data, and then transfer learning can be used to train task specific networks for final application.

[0009] In various embodiments, the disclosed subject matter is a system for providing comparative data in a manufacturing facility. The system includes a common backbone network based on, for example, a machine learning-based network (such as a deep convnet-based network or a transformer-based network). The common backbone network includes: an input component for receiving raw image data; a self-supervised training component for generating a machine learning-based (e.g., convnet-based or transformer-based) comparative database from the raw image data; and an analysis engine for performing anomaly detection and classification of images received from multiple task-specific networks electrically coupled to the common backbone network. Each of the multiple task-specific networks is coupled to at least one type of equipment selected from equipment types including metrology tools and inspection tools, and the analysis engine converts the raw image data into features to be transmitted to the multiple task-specific networks to perform final calculations for specific tasks within the manufacturing facility.

[0010] In various embodiments, the disclosed subject matter is an image processing system for classifying defects in a manufacturing facility. The image processing system includes: a common backbone network based on, for example, a machine learning-based network (such as a deep convnet-based network or a transformer-based network); and a plurality of task-specific networks electrically coupled to the common backbone network. Each of the plurality of task-specific networks is based on, for example, a machine learning-based network (such as a deep convnet-based network or a transformer-based network), wherein each of the plurality of task-specific networks is configured to be trained for a specific task within a manufacturing facility.

[0011] In various embodiments, the disclosed subject matter is a method for comparing data in a manufacturing facility. The method includes: receiving raw image data as input to a common backbone network, wherein the common backbone network is based on, for example, a machine learning-based network (such as a deep convnet-based network or a transformer-based network); using self-supervised training to generate a machine learning-based (e.g., convnet-based or transformer-based) comparison database from the raw image data; performing anomaly detection and classification of images received from multiple task-specific networks electrically coupled to the common backbone network; and converting the raw image data into features and transmitting the features to the multiple task-specific networks.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The figures in the drawings depict only example implementations of the present disclosure and should not be considered limiting of its scope.

[0014] Figure 1 An example of a multi-head convolutional neural network (convnet) having a backbone network coupled to multiple smaller networks according to various embodiments of the disclosed subject matter is shown;

[0015] Figure 2 Shows that there is Figure 1 An example of additional details of each part of a multi-head convolutional neural network;

[0016] Figure 3 Various embodiments of the disclosed subject matter are shown for use Figure 1 A generalized example of the multi-head convolutional neural network approach;

[0017] Figure 4 An example of a machine learning framework for performing anomaly detection and / or classification of an image of features or characteristics from a captured image on a substrate according to various embodiments of the disclosed subject matter is shown;

[0018] Figure 5 A block diagram is shown that includes an example of a machine on which any one or more of the techniques (e.g., methods) discussed herein may be performed; and

[0019] Figure 6 An exemplary embodiment of a transformer encoder that may be used in a transformer based network is shown. DETAILED DESCRIPTION

[0020] The disclosed subject matter relates to image analysis and classification within a manufacturing facility (e.g., a semiconductor manufacturing facility). In various embodiments, a common backbone is arranged to convert various types of raw image data into features, which in turn can be used by smaller neural networks to perform final calculations for specific tasks in the manufacturing facility. The common backbone can be, for example, a deep convnet-based network or a transformer-based network. The smaller neural networks perform, for example, final defect detection, die-to-die image comparison, anomaly detection, and customer-specific tasks. The common backbone network can be initially trained using self-supervised learning based on the raw image data, and then transfer learning can be used to train the final application of the task-specific network.

[0021] As disclosed herein, image data can be processed using a machine learning framework such as a convolutional neural network (CNN or convnet). Processing image data includes, for example, finding spatial relationships within a captured image to determine features or characteristics of features or devices on a substrate as described herein. A generalized example of a machine learning framework that can be used for the disclosed subject matter is described in more detail below.

[0022] Convolutional neural networks (convnets or CNNs) can perform image processing tasks such as anomaly detection, classification, and feature localization of various types of features (e.g., 3D bump patterns, watermarks, etc.) and devices (e.g., patterns on memory devices or integrated circuit devices) formed on substrates. Such substrates are used in semiconductor and related industries. However, each of the described image processing tasks is highly computationally intensive, typically requiring a significant amount of time to process each image. In addition, these tasks require large amounts of data as input to various convnets in order to compare and classify images captured by metrology and inspection tools.

[0023] Although various embodiments are described in terms of convolutional neural networks, other types of networks may also be used. For example, instead of or in addition to using layers of a convolutional network, a transformer-based network may use N transformer encoder blocks that are stacked similarly to the layers in a convnet. Figure 6 An exemplary embodiment of a transformer encoder 600 that may be used in a transformer-based network is shown in FIG.

[0024] Instead of passing multi-channel images between layers, the images are compressed into N-dimensional vectors. The transformer encoder 600 takes vectors and emits vectors, allowing them to be stacked. The lowest level input vector is a linear transformation of all image slices (e.g., 16×16 pixels) to basis vectors. Within the transformer encoder block, multiple “attention heads” look for blocks of interest (e.g., via dot products), which are added to skip connections and then added to a multi-layer perception (MLP) block that is a stack of multiple linear activation layers.

[0025] Attention heads allow neural networks to learn relationships across images that do not need to be next to each other or exist when reducing resolution like convnets. A possible tradeoff with convnets is that training the backbone network requires additional images, plus additional training computation time and / or intensity.

[0026] Regarding the amount of data used as input, millions of manually labeled images are traditionally used to train machine learning components of convnet-based or transformer-based networks. The images available for input to the network are typically "out-of-distribution" (e.g., not within the metrology level of such feature being imaged). Such out-of-distribution images are generalized and therefore not particularly suitable for a particular combination of metrology or inspection tools for comparison with, for example, another similar feature located on the same substrate. Furthermore, these out-of-distribution images cannot be used to compare or correlate with architecturally similar features of a given feature or device (e.g., a particular layout of transistors within a particular manufacturing facility of a device manufacturer).

[0027] However, a metrology or inspection tool manufacturer may, for example, collect a large amount of data in collaboration with multiple manufacturing facilities of one or several device manufacturers, to supplement the data that each may have already collected, stored, and characterized. Thus, data from a metrology or inspection tool manufacturer can be used to develop a self-supervised training algorithm. Such data can be used to train a large backbone network without the need to label every component of the data.

[0028] In various embodiments of the disclosed subject matter, multi-head deep convolutional neural networks (convnets) or transformer-based methods can be used to perform operations such as defect detection on substrates and die-to-die comparisons. These multi-head networks allow specific machine learning algorithms to run in essentially real time on a metrology or inspection tool.

[0029] The generalized architecture described in more detail below consists of a deep convnet-based or transformer-based backbone network (common backbone) that converts raw image data into features that can in turn be used by smaller neural networks to perform final computations for specific tasks. The smaller networks perform, for example, final defect detection, die-to-die image comparison, anomaly detection, and customer-specific tasks. The common backbone network can be initially trained using self-supervised learning, and then transfer learning can be used to train task-specific networks for final application.

[0030] Thus, in the disclosed subject matter, a convnet-based or transformer-based common backbone can analyze most or all types of defects present on a substrate. The common backbone network can be used for all metrology and inspection tools of a tool manufacturer. Any number of task-specific networks can be point-coupled to the common backbone network (e.g., hardwired, wirelessly linked, or in electronic communication). The combination of a convnet-based or transformer-based common backbone network with smaller task-specific networks is referred to herein as a multi-head network. Multi-head networks allow for the reuse of computationally expensive convolutional networks. Although the remaining embodiments are described in terms of convnets, transformer-based nets as described above can be used as an alternative or supplement to convnets.

[0031] Figure 1 An example of a multi-headed convnet 100 having a convolutional neural network (convnet) based backbone network coupled to multiple smaller networks is shown, according to various embodiments of the disclosed subject matter. Figure 1 The convnet-based backbone network 101 is shown as including an input data component 150 coupled to the convnet-based backbone network 101. The convnet-based backbone network 101 is also coupled to multiple smaller networks 110 in electrical communication, for example, wirelessly or otherwise. The smaller networks 110 include, for example, task-specific neural network 1111A and task-specific neural network 2111B to task-specific neural network N 111C. At least one of the task-specific neural networks 111A to 111C is electrically coupled to at least one of the metrology tools and inspection tools 120, for example, within a semiconductor manufacturing facility. In an embodiment, the backbone network 101 can be viewed as an encoder using a transformer-based network, as described below with reference to Figure 6 In an embodiment, backbone network 101 may use a hybrid arrangement of convnet-based networks and transformer-based networks.

[0032] The metrology tool and inspection tool 120 are used to determine metrology-related information from a substrate 130 or to observe features on the substrate. The substrate 130 may include a wafer formed of an elemental semiconductor (e.g., silicon or germanium), a wafer formed of a compound semiconductor (e.g., gallium arsenide (GaAs) or gallium nitride (GaN)), or various other substrate types known in the art (including conductive substrates, semi-conductive substrates, and non-conductive substrates). Thus, the substrate 130 may include, for example, any one or more of 400 mm, 300 mm, 200 mm, 150 mm, 125 mm, and 100 mm circular substrates (e.g., wafers), including opaque (e.g., at visible wavelengths), transparent, and bonded substrates. The substrate 130 may also include various types of non-circular (e.g., square, rectangular, polygonal, etc.) substrates, including quartz photomasks (e.g., with or without pellicles), flat panel displays, or thin film heads for the data recording industry. Substrate 130 may also be formed of a non-rigid material, such as polyethylene terephthalate (PET) having a film deposited thereon or otherwise formed with a semiconductor layer, as used in the manufacture of mobile devices (eg, cellular telephones).

[0033] The metrology tools are used to measure various features or characteristics on the substrate 130. The various features and characteristics may include, for example, film thickness measurements, critical dimension (CD) measurements (in x-dimension, y-dimension, and / or z-dimension) of features formed on the substrate, pitch of line-space features formed on the substrate, overlay shift from one layer to another on the substrate, or many other measurements known to those of ordinary skill in the art. Inspection tools may be used to inspect the consistency of features formed on or otherwise present on the substrate. For example, the inspection object may be the substrate on which the integrated circuit die is to be fabricated, the location of the die, the location of unwanted particulate matter or other unwanted or unplanned features, etc. However, the metrology tools and inspection tools 120 typically use some basis to compare each of the features measured or identified on the substrate 130. In the disclosed subject matter provided herein, the data, objects, or components used in such comparisons may be obtained from image data provided to the input data component 150 and transmitted to the convnet-based backbone network 101.

[0034] The input data component 150 may include a variety of different image-based data types. The different image-based data types are provided so that the metrology tool and / or inspection tool 120 can compare images captured from the substrate 130 with images having the same or similar characteristics. Figure 2 The characteristics and types of image-based data types are described in more detail. The input data component 150 provides different image-based data types to the input component 103. The input component 103 may include, for example, a memory or storage unit within the convnet-based backbone network 101.

[0035] In addition, despite Figure 1 An input data component 150 is shown as being separate from the convnet-based backbone network 101, but this figure is provided only to facilitate understanding of the disclosed subject matter. In some embodiments, the input data component 150 may be located within or near the convnet-based backbone network 101 (e.g., hardwired or directly electrically coupled to the convnet-based backbone network). In other embodiments, the input data component 150 may be located at a location remote from the convnet-based backbone network 101. In the latter embodiment, the input data component 150 may be located in another part of the manufacturing facility, in another facility separate from the manufacturing facility, or in another part of the world. That is, the input data component 150 may be geographically and physically separated from the convnet-based backbone network 101. However, the input data component 150 either electronically communicates with the convnet-based backbone network 101 (e.g., wirelessly), or is periodically electrically coupled to the convnet-based backbone network 101 to provide raw image information input. Although the input data component 150 can be electrically coupled to the convnet-based backbone network 101 at all times, continuous electrical coupling need not occur. Image data from the input data component 150 may be updated only occasionally. In addition, the image data within or transmitted to the input data component 150 may include, for example, 2D and 3D data from multiple image sources, such as fluorescence data, color filter data, etc., as well as other various data types received from various types of sensors (including optical sensors). Moreover, the fusion of different types of images (including multi-channel images and data) may be used as a feed to the convnet-based backbone network 101, Figure 6 The input of the transformer encoder 600 is based on a mixture of two networks.

[0036] The self-supervised training component 105 is coupled to the input component 103 to prepare a convnet-based comparison database. The convnet-based comparison database uses different image-based types to compare with images captured from the task-specific neural networks 111A-111C within the analysis engine 107. The analysis engine 107 may include multiple hardware-based processors and may perform, for example, anomaly detection and classification of images received from one or more of the task-specific neural networks 111A-111C electrically coupled to the convnet-based backbone network 101. The task-specific neural networks 111A-111C may therefore be relatively small (e.g., limited in computational power) because all major processing has been or may be performed by the analysis engine 107 within the convnet-based backbone network 101.

[0037] The task-specific neural networks 111A-111C are coupled to the convnet-based backbone network 101 and can be trained for a specific end user of the network. Each of the task-specific neural networks 111A-111C can also include a smaller convnet-based network to perform a comparison task that may be less computationally intensive than the convnet contained within the convnet-based backbone network 101. Thus, various tasks can be divided and processed by various parts of the multi-headed convolutional neural network 100. Thus, at least a portion of the input data for the task-specific neural networks 111A-111C is provided from the output of the convnet-based backbone network 101. Thus, the multi-headed convolutional neural network 100 allows multiple different task-specific neural networks located at different facilities in multiple different geographic regions and multiple different device manufacturers to reuse computationally expensive data.

[0038] In addition, each of the task-specific neural networks 111A-111C can be customized via task-specific processing. For example, a comparison task described in more detail below can classify what is considered normal variation for a particular process (e.g., normal process variation or tolerance value). These normal variations can then be ignored by the task-specific neural networks 111A-111C. Therefore, in this example, only detected features or anomalies that are of interest to process engineers can be noted and emphasized. These detected features or anomalies can then be transmitted to the convnet-based backbone network 101 for further analysis in the analysis engine 107. In an embodiment, the detected features or anomalies can be analyzed at least in part by a hardware-based processor within one or more of the task-specific neural networks 111A-111C, which relies on information provided by the convnet-based backbone network 101. In other embodiments, the task of analyzing the detected features or anomalies can be split between the analysis engine 107 and one or more of the task-specific neural networks 111A-111C. In various embodiments, the task-specific neural networks 111A-111C can be considered to be using the following reference Figure 6 The decoder of the transformer-based network, such as the multi-head attention block 605, can be used to classify visual information. In an embodiment, one or more of the task-specific neural networks 111A-111C can use a hybrid arrangement of convnet-based networks and transformer-based networks.

[0039] The tasks performed by the task-specific neural networks 111A-111C may include, for example, comparing a CAD image to an actual image captured by a metrology tool, comparing an actual image captured by a metrology tool to a plurality of known defect types, comparing one planned die to another planned die of the same type on a substrate, and performing other tasks to verify whether there are any problems with the features / devices formed on the substrate 130. Figure 2 Describe these tasks in more detail.

[0040] Reference now Figure 2 , showing that Figure 1 200. The input data component 150 is shown as including several examples of input image data types, including, for example, a metrology acquisition image type 201, a CAD reference image type 203, and a reference die image type 205. The comparison module 250 provides information that can be used in Figure 1 107, where the results from the comparison module 250 are provided to the task-specific neural networks 111A-111C. In various embodiments, the comparison module 250 may be located entirely within the convnet-based backbone network 101. In other embodiments, the comparison module 250 may be located entirely within one or more of the task-specific neural networks 111A-111C. In other embodiments, portions of the comparison operations of the comparison module 250 may occur within the convnet-based backbone network 101, while other portions of the operations are performed within one or more of the task-specific neural networks 111A-111C. In other embodiments, the comparison operations of the comparison module 250 may occur within the convnet-based backbone network 101 and be replicated on one or more of the task-specific neural networks 111A-111C.

[0041] The metrology acquisition image type 201 may include a large number of metrology images previously collected by, for example, a manufacturer of a particular type of metrology tool. The metrology acquisition image type 201 may also include images collected from tens or hundreds of versions of the same or similar metrology tools used by device manufacturers. Thus, the plurality of metrology images available for input to the convnet-based backbone network 101 may include millions of images collected over a period of many years.

[0042] CAD reference image type 203 may include, for example, an image of a two-dimensional layout of how each layer of a particular device is expected to appear to a metrology tool or a vision inspection tool. CAD reference image type 203 may also include CD measurements of various objects on a given layer.

[0043] Reference die image type 205 may include, for example, images of completed dies for a given device type. Reference die image type 205 may include actual images of known good dies previously collected from a metrology tool or a visual inspection tool.

[0044] Each of the examples of input image data types 201, 203, 205 may be trained in a self-supervised training component 105 (see Figure 1 ) are organized and categorized. Typically, in order to train a larger backbone network from scratch, thousands or more human-labeled images will be required for training. However, the self-supervised training component 105 eliminates the need to label image data types. Various image types, including those described above, are used to train the convnet-based backbone network 101. For example, various semiconductor metrology and inspection tool manufacturers and manufacturing facilities that use these metrology and inspection tools have access to a very large database of collected images acquired over many years. Semiconductor device manufacturers have provided raw images of the entire substrate and the output of the metrology and inspection system. Such images can be obtained from, for example, the front-end process (FEOL) and the back-end process (BEOL) processes, and have been taken at different magnifications and using different light sources (including from the front side and edge of the substrate). Images have been acquired from, for example, CMOS logic device manufacturers, power electronics device manufacturers, LED manufacturers, camera sensor manufacturers, and various other manufacturing facilities. Libraries providing various image types can be maintained and used within the convnet-based backbone network 101 for future machine learning input.

[0045] As described above, the comparison module 250 provides a method for training the self-supervised training component 105 and Figure 1 107. In various embodiments, the results from the comparison module 250 may be provided to one or more of the task-specific neural networks 111A-111C. In various embodiments, one or more of the task-specific neural networks 111A-111C may provide image data from at least one of the metrology tool and the inspection tool 120 to the convnet-based backbone network 101 for comparison with various comparison types within the comparison module 250. In an embodiment, a portion of the results from the comparison module 250 may be provided to one or more of the task-specific neural networks 111A-111C while some image data from at least one of the metrology tool and the inspection tool 120 is transmitted to the convnet-based backbone network 101 for comparison.

[0046] The comparison types include, for example, comparing image data collected from the substrate 130 by at least one of the metrology tool and inspection tool 120 with the CAD comparison data component 251. The CAD comparison data component 251 can be output from the analysis engine 107 based on, for example, the CAD reference image type 203 used to train the self-supervised training component 105. The comparison of the image data received from at least one of the metrology tool and inspection tool 120 with the CAD comparison data component 251 can occur in one or both of the convnet-based backbone network 101 and one or more of the task-specific neural networks 111A-111C. The comparison can be used to determine whether the feature on the substrate 130 being measured or observed is "correct". That is, whether the feature meets the expected characteristics within the predetermined metrology value (e.g., CD value, overall shape, orientation relative to the rest of other features on the substrate 130, and other physical factors). The predetermined tolerance value engine 270 can supply a predetermined tolerance value to the comparison module 250 for each of the examples of the comparison type. The predetermined tolerance value engine 270 may include, for example, a software-based, firmware-based, and / or hardware-based database stored as a lookup table.

[0047] The predetermined tolerance value engine 270 can provide predetermined tolerance values ​​for many types of devices. For example, the predetermined tolerance value engine 270 can be trained to ignore variations in a measured or inspected substrate that are within manufacturing tolerances. Variations within tolerances can include process variations, system-to-system variations (e.g., differences between multiple process tools or differences between multiple metrology tools), XY alignment differences from one die to another, and the like.

[0048] In a manner similar to CAD reference image type 203, image data received from at least one of metrology and inspection tools 120 may be compared to pre-classified defect features 253, comparison to one or more die features 255, comparison to known or determined watermark features 257, comparison to three-dimensional bump type features 259, or comparison to various other comparison type features 261. Each of these other comparison types may also receive an associated tolerance value from a predetermined tolerance value engine 270.

[0049] In various embodiments, the comparison types within the pre-classified defect feature 253 may include various types of processing defects. Processing defects include types such as bridged or shorted lines, open lines, unopened vias, scratches, particle defects, or a multitude of other defect types that manufacturing facilities encounter when producing devices.

[0050] The types of comparisons within the comparison of one or more die components 255 may include, for example, comparisons of memory array or controller layouts to measured or observed objects on substrate 130. The comparison of one or more die components 255 may also include physical, optical, and electrical properties, such as the refractive index of a film or the thickness of a film stack or film layer.

[0051] The comparison with known or determined watermark component 257 can be used to identify various types of watermarks on at least a portion of substrate 130. The comparison can also include comparison with known or derived chemical markers and stains on at least a portion of substrate 130.

[0052] The comparison with the three-dimensional bump type feature 259 may be used to compare known or expected bump types, including physical size, location, and electrical characteristics (eg, resistance), to measured or observed values ​​of bumps on the substrate 130 .

[0053] Each of these comparison types is provided merely as an example so that one of ordinary skill in the art will better understand aspects of comparison module 250. Thus, the comparison types may be expanded, contracted, or redefined as needed for a particular manufacturing facility or application.

[0054] Based on the above comparisons, Figure 1 The analysis engine 107 can be used to update the input component 103. By updating the input component 103, the convnet-based backbone network 101 is periodically refreshed with new data comparison.

[0055] Reference now Figure 3 , showing a method for using the disclosed subject matter according to various embodiments Figure 1 A generalized example of the method 300 based on the convnet backbone network 101. Figure 3 The operations of are shown in a particular order and as a series of linear steps, but one of ordinary skill in the art will recognize that certain steps may be repeated, skipped, or performed in a different order.

[0056] Also refer to Figure 1 and Figure 2 , the method 300 includes receiving raw image data at operation 301. The raw image data may be received from the input data component 150. At operation 303, the self-supervised training component 105 may be trained using the raw image data.

[0057] At operation 305, an image measured or observed by at least one of the metrology tool and the inspection tool 120 from, for example, the substrate 130 is received by the convnet-based backbone network 101, as provided from at least one of the task-specific neural networks 111A-111C. Anomaly detection and classification of the image occurs at operation 307. In an embodiment, anomaly detection and classification may occur entirely within the analysis engine 107 of the convnet-based backbone network 101. In other embodiments, anomaly detection and classification may occur as a task split between the analysis engines 107 or within the processing components of at least one of the task-specific neural networks 111A-111C.

[0058] The raw image data may be converted at operation 309. Conversion of the raw image data may include, for example, deriving metrology data (e.g., CD measurements, film, thickness data, sheet resistance values, refractive index values, etc.) from one or both of the raw image data received at operation 301 or measurements and observations of the image received at operation 305.

[0059] Operation 311 includes an optional step of extracting features of the image. The extracted features may include extracting data from the image that may indicate open features in traces on the substrate 130 or vias that are not fully opened.

[0060] At operation 313, an optional step includes aligning and / or scaling the image or image features to match the scaling or alignment of the raw image data received at operation 301. Such alignment or scaling operations may provide for easier comparison of the measured or observed image data with the raw image data input from input data component 150.

[0061] Figure 4 An example of a machine learning framework for performing anomaly detection and / or classification of features or characteristics from captured images on a substrate according to various embodiments of the disclosed subject matter is shown. As described above, the machine learning framework 400 can be used in a training mode to train, for example, the self-supervised training component 105 (see Figure 1 ).

[0062] like Figure 4 As shown, the machine learning framework 400 is shown to include a preprocessor 403 and a machine learning network 430. Figure 1 The input data component 150 receives the raw image 401 and provides it to the preprocessor 403. In this example, the preprocessor 403 filters or otherwise processes the raw image 401 to, for example, crop, scale, extract data from it, or otherwise change or enhance the raw image 401, and generates a preprocessed image 405.

[0063] The pre-processed image 405 can then be input into a machine learning network 430. The machine learning network 430 can be provided as a multi-layer machine learning model. For example, the machine learning network 430 can include the following four layers, including an input layer 407, a feature extraction layer 409, a feature relationship layer 411, and a decision layer 413. In this exemplary model, the decision layer 413 can have multiple outputs. The outputs can include, for example, key points; bounding boxes; multiple physical, optical, and electrical properties of a die or bump; and other substrate-level, die-level, or layer-level properties about possible features present on a substrate for a particular manufacturing process.

[0064] Continue to refer Figure 4 , pixel information from the preprocessed image 405 can be sent to the input layer 407. Each node in the input layer 407 can correspond to a pixel in the preprocessed image 405. The machine learning network 430 can be trained in one or more layers in layers 407 to 413 in an iterative manner. As described above, the decision layer 413 can output decisions about various substrate characteristics of a given substrate. The substrate characteristics are then generated in the output box 415. Therefore, the output box 415 can store substrate-level, tube core-level, or layer-level characteristics extracted from the original image 401. In various embodiments, the output box 415 can provide text indications showing various characteristics (for example, sheet resistance values, refractive indices, physical properties, and sizes of features on the tube core). In various embodiments, the values ​​and / or characteristics within the output box 415 are provided as input to Figure 1 Analysis engine 107.

[0065] The methods and techniques described and described herein can be used as follows: Figure 5 The discussed machine 500 may be executed in part or in whole. Figure 5 An exemplary block diagram is shown including a machine 500 on which any one or more of the techniques (e.g., methodologies) discussed herein may be performed. In various examples, machine 500 may operate as a standalone device or may be connected (e.g., networked) to other machines.

[0066] In a networked deployment, the machine 500 can operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, the machine 500 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 500 can be a personal computer (PC), a tablet device, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) instructions for actions to be taken by the specified machine. In addition, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines that execute one (or more) sets of instructions to perform any of the methods described herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations, either individually or jointly.

[0067] Examples as described herein may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit system is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). Circuit system membership may change over time and potential hardware changes. A circuit system includes members that can perform specified operations individually or in combination when operating. In one example, the hardware of the circuit system can be immutably designed to perform specific operations (e.g., hardwired). In one example, the hardware including the circuit system may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.), which include computer-readable media that are physically modified (e.g., magnetically, electronically, such as via a change in physical state or a transformation of another physical property, etc.) to encode instructions for specific operations. When connecting physical components, the basic electrical properties of the hardware composition may, for example, change from insulating properties to conductive properties or vice versa. Instructions enable embedded hardware (e.g., execution units or loading mechanisms) to form members of the circuit system in hardware via variable connections to perform parts of specific operations when operating. Thus, when the device is operating, the computer-readable medium is communicatively coupled to other components of the circuit system. In one example, any physical component can be used in more than one member of more than one circuit system. For example, in operation, an execution unit can be used in a first circuit of a first circuit system at one point in time and reused by a second circuit in the first circuit system or by a third circuit in the second circuit system at a different time.

[0068] The machine 500 (e.g., a computer system) may include a hardware-based processor 501 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 503, and a static memory 505, some or all of which may communicate with each other via an interconnect 530 (e.g., a bus). The machine 500 may also include a display device 509, an input device 511 (e.g., an alphanumeric keyboard), and a user interface (UI) navigation device 513 (e.g., a mouse). In an example, the display device 509, the input device 511, and the UI navigation device 513 may include at least a portion of a touch screen display. In addition, the machine 500 may additionally include a storage device 520 (e.g., a drive unit), a signal generating device 517 (e.g., a speaker), a network interface device 550, and one or more sensors 515 (such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors). The machine 500 may include an output controller 519, such as a serial controller or interface (e.g., a universal serial bus (USB)), a parallel controller or interface, or other wired or wireless (e.g., an infrared (IR) controller or interface, near field communication (NFC), etc.), which is coupled to communicate or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0069] The storage device 520 may include a machine-readable medium on which is stored one or more sets of data structures or instructions 524 (e.g., software or firmware) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 524 may also reside, completely or at least partially, within the main memory 503, within the static memory 505, within the mass storage device 507, or within the hardware-based processor 501 during execution thereof by the machine 500. In an example, one or any combination of the hardware-based processor 501, the main memory 503, the static memory 505, or the storage device 520 may constitute a machine-readable medium.

[0070] Although the machine-readable medium is considered a single medium, the term "machine-readable medium" may include a single medium or multiple media (eg, a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 524.

[0071] The term "machine-readable medium" may include any medium that can store, encode, or carry instructions for execution by the machine 500 and cause the machine 500 to perform any one or more of the techniques of the present disclosure, or that can store, encode, or carry data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory and optical and magnetic media. Thus, machine-readable media are not transient propagating signals. Specific examples of large-scale machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic or other phase change or state change memory circuits; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0072] The instructions 524 may also be transmitted or received over the communication network 521 using a transmission medium via the network interface device 550 using any of a variety of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include, among other things, a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network known as a cellular network). The Institute of Electrical and Electronics Engineers (IEEE) 802.22 family of standards, known as The IEEE 802.26 family of standards), the IEEE 802.25.4 family of standards, a peer-to-peer (P2P) network. In an example, the network interface device 550 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communications network 521. In an example, the network interface device 550 may include multiple antennas for wireless communications using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term "transmission medium" should be understood to include any intangible medium capable of storing, encoding, or carrying instructions for execution by the machine 500, and includes digital or analog communication signals or other intangible media that facilitate the communication of such software.

[0073] Figure 6An exemplary embodiment of a transformer encoder 600 that can be used in a transformer-based network is shown, as briefly described above. A transformer-based network is a type of machine learning (ML) model based on self-attention. Transformer-based networks are often used in natural language processing (NLP). However, transformer models can also be directly applied to images. In one example, a transformer-based network can be trained on image classification in a supervised manner. In various embodiments, the input to the transformer can be pre-trained on a large number of images (e.g., 2D or 3D images). The results of the large number of pre-trained images can then be fine-tuned on a smaller task-specific dataset (as described in the reference above). Figure 1 In an embodiment, the transformer encoder 600 can also be trained in an unsupervised manner as a generative model. The resulting representation can then be fine-tuned for classification performance.

[0074] An image (e.g., a 2D image, but such restriction is not necessary) is divided into "slices" of fixed size. Each slice is embedded linearly, possibly with an additional positional embedding. The slices are then provided to input 601 (slices can be processed in the same way as tokens (words) in NLP applications). The resulting sequence of vectors is provided as input to the transformer encoder 300.

[0075] Local multi-headed dot products with self-attention blocks can be used instead of convolutions. Attention mechanisms were originally used to process word sequences in machine translation, which have an implicit temporal aspect to them. However, attention can be generalized to process information that can be static and not necessarily related in a sequential manner, such as in the context of image processing as described in this paper.

[0076] At the first normalization block 603, the input patch may be normalized as needed. A multi-head attention block 605 may be used to classify the visual information received from the normalization block 603. As briefly discussed above, "attention" in the ML sense is the ability to dynamically emphasize and use significant portions of received information (e.g., image input).

[0077] Self-attention can occur in parallel in the multi-attention block 605, where k self-attention operations can run in parallel. The summation of the values ​​from the multi-attention block 605 can be applied before the second normalization block 607.

[0078] The output from the sum of the multi-head attention block 605 and the normalization block 607 is then passed to a multi-layer perception (MLP) block 609, which may act as a class prediction layer. Additional summation may be applied before generating the output 611 from the transformer encoder 600.

[0079] As used herein, the term "or" may be interpreted as inclusive or exclusive. In addition, based on reading and understanding the disclosure provided, one of ordinary skill in the art will understand other embodiments. In addition, one of ordinary skill in the art will readily appreciate that various combinations of the techniques and examples provided herein may all be applied in various combinations.

[0080] Throughout the specification, multiple instances can be implemented as parts, operations or structures described as single instances. Although each operation is shown and described as a separate operation, one or more of each operation can be performed simultaneously, and unless otherwise specified, it is not required that these operations must be performed in the order of illustration. The structure and function presented as a separate component in the example configuration can be implemented as a combined structure or component. Similarly, the structure and function presented as a single component can be implemented as a separate component. These and other variations, modifications, additions and improvements all fall within the scope of the subject matter described herein.

[0081] Furthermore, although not explicitly shown, a skilled person will appreciate that each of the various arrangements, quantities, and numbers of components may vary (e.g., the number of task-specific neural networks, the number of metrology tools, the number of inspection tools, etc.). Furthermore, each example shown and described herein represents only one possible configuration and should not be considered to limit the scope of the invention.

[0082] Although various embodiments are discussed separately, these separate embodiments are not intended to be considered independent techniques or designs. As indicated above, each of the various parts can be interrelated, and each can be used alone or in combination with other embodiments discussed herein. For example, although various embodiments of operations, systems, and processes have been described, these methods, operations, systems, and processes can be used alone or in various combinations.

[0083] Therefore, as will be apparent to those of ordinary skill in the art when reading and understanding the disclosure provided herein, many modifications and variations can be made. According to the foregoing description, in addition to those methods and devices listed herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art. Portions and features of some embodiments may be included in portions and features of other embodiments, or replace portions and features of other embodiments. Such modifications and variations are intended to fall within the scope of the appended claims. Therefore, the present disclosure is limited only by the terms of the appended claims and the full scope of equivalents to which these claims are entitled. It should also be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting.

[0084] The abstract of the specification is provided to allow the reader to quickly determine the essence of the present technical disclosure. The abstract is submitted with the understanding that it will not be used to interpret or limit the claims. In addition, in the foregoing detailed description, it can be seen that various features can be combined together in a single embodiment for the purpose of simplifying the present disclosure. This method of disclosure should not be interpreted as limiting the claims. Therefore, the following claims are hereby incorporated into the detailed description, with each claim acting as a separate embodiment by itself.

[0085] The description provided herein includes illustrative examples, devices, and apparatuses that embody various aspects of the subject matter described in the present invention document. In this description, for the purpose of explanation, numerous specific details are set forth in order to provide an understanding of the various embodiments of the subject matter discussed. However, it is apparent to one of ordinary skill in the art that the various embodiments of the disclosed subject matter can be practiced without these specific details. In addition, well-known structures, materials, and techniques are not shown in detail in order to avoid obscuring the various exemplified embodiments. As used herein, the terms "about," "approximately," and "substantially" may refer to, for example, within a given value or range of values. + Value within 10%.

[0086] The following numbered examples are specific embodiments of the disclosed subject matter.

[0087] Embodiment 1: A system for providing comparative data in a manufacturing facility. The system includes a common backbone network based on a machine learning-based network (e.g., such as at least one of a convolutional neural network (convnet) and a transformer-based network). The common backbone network includes: an input component for receiving raw image data; a self-supervised training component for generating a machine learning-based comparative database (e.g., such as at least one of a convolutional neural network (convnet) and a transformer-based comparative database) from the raw image data; and an analysis engine for performing anomaly detection and classification of images received from multiple task-specific networks electrically coupled to the common backbone network. Each of the multiple task-specific networks is coupled to at least one type of equipment selected from equipment types including metrology tools and inspection tools, and the analysis engine is used to convert the raw image data into features to be transmitted to the multiple task-specific networks to perform final calculations for specific tasks within the manufacturing facility.

[0088] Embodiment 2: The system according to embodiment 1, wherein the original image data includes an acquired image and a reference image.

[0089] Embodiment 3: The system according to any one of Embodiment 1 or Embodiment 2, wherein the reference image comprises a CAD-based reference image and a reference die image.

[0090] Embodiment 4: The system according to any one of the preceding embodiments, wherein the common backbone network is further configured to extract image features, and align and scale the image features to the image data provided to the input data component.

[0091] Embodiment 5: The system of embodiment 4, wherein the image features are transmitted to the plurality of task-specific networks to determine defects within the manufacturing facility.

[0092] Embodiment 6: A system according to any one of the preceding embodiments, wherein the raw image data can be used to train a backbone network without labeling each component of the raw image data.

[0093] Embodiment 7: The system according to any of the preceding embodiments, wherein the analysis engine is further configured to compare images received from the plurality of task-specific networks with relevant images in the comparison database generated from the raw image data.

[0094] Embodiment 8: An image processing system for classifying defects in a manufacturing facility. The image processing system includes a common backbone network based on a machine learning-based network (e.g., such as at least one of a convolutional neural network (convnet) and a transformer-based network); and a plurality of task-specific networks electrically coupled to the common backbone network. Each of the plurality of task-specific networks is based on a machine learning-based network (e.g., such as at least one of a convolutional neural network (convnet) and a transformer-based network) separate from the common backbone network, and each of the plurality of task-specific networks is configured to be trained for a specific task within the manufacturing facility.

[0095] Embodiment 9: An image processing system according to embodiment 8, wherein the common backbone network comprises: an input component for receiving raw image data; a self-supervised training component for generating a comparison database from the raw image data; and an analysis engine for performing anomaly detection and classification of images received from the multiple task-specific networks, the analysis engine for converting the raw image data into features to be transmitted to the multiple task-specific networks to perform final calculations for specific tasks within the manufacturing facility.

[0096] Embodiment 10: The image processing system according to embodiment 9, wherein the analysis engine is further configured to compare the images received from the plurality of task-specific networks with relevant images in the images in the comparison database generated from the original image data.

[0097] Embodiment 11: The image processing system according to embodiment 9, wherein each of the plurality of task-specific networks is configured to perform defect detection and die-to-die comparison operations based on the features received from the analysis engine of the common backbone network.

[0098] Embodiment 12: An image processing system according to any one of Embodiments 8 to 11, wherein each of the plurality of task-specific networks will be coupled to at least one type of tool including a metrology tool and an inspection tool and collect data from the at least one type of tool.

[0099] Embodiment 13: The image processing system according to any one of Embodiments 8 to 12, wherein each of the plurality of task-specific networks is trained to ignore normal process variations of a specific process based on a predetermined tolerance value.

[0100] Embodiment 14: The image processing system according to any one of Embodiments 8 to 12, wherein each of the plurality of task-specific networks is customized with a task-specific treatment for a specific process.

[0101] Embodiment 15: A method for comparing data in a manufacturing facility. The method includes: receiving raw image data as input to a common backbone network, wherein the common backbone network is based on a machine learning-based network (e.g., such as at least one of a convolutional neural network (convnet) and a transformer-based network); using self-supervised training to generate at least one of a machine learning-based comparison database (e.g., such as at least one of a convolutional neural network (convnet) comparison database and a transformer-based network comparison database) from the raw image data; performing anomaly detection and classification of images received from multiple task-specific networks electrically coupled to the common backbone network; and converting the raw image data into features and transmitting the features to the multiple task-specific networks.

[0102] Embodiment 16: The method according to embodiment 15 also includes performing final calculations for specific tasks within the manufacturing facility based on the characteristics.

[0103] Embodiment 17: The method according to any one of Embodiment 15 or Embodiment 16 further includes using transfer learning to train the final application of each task-specific network in the multiple task-specific networks.

[0104] Embodiment 18: The method according to any one of Embodiments 15 to 17 further includes: extracting image features; and aligning and scaling the image features from the original image data to a common coordinate system within the common backbone network.

[0105] Embodiment 19: The method according to any one of Embodiments 15 to 18 further includes comparing the images received from the multiple task-specific networks with relevant images in the images in the comparison database generated from the original image data.

[0106] Embodiment 20: The method according to any one of Embodiments 15 to 19 further includes receiving an image from at least one task-specific network of the plurality of task-specific networks from at least one type of tool selected from tool types including metrology tools and inspection tools.

Claims

1. A system for providing comparative data in a manufacturing facility, the system comprising: A public backbone network, the public backbone network being based on a machine learning based network, the public backbone network comprising: An input component, the input component is used to receive original image data; a self-supervised training component for generating a machine learning-based comparison database from the raw image data; and an analysis engine for performing anomaly detection and classification of images received from a plurality of task-specific networks electrically coupled to the common backbone network, Each of the plurality of task-specific networks is coupled to at least one type of equipment, and the analysis engine is used to convert the raw image data into features to be transmitted to the plurality of task-specific networks to perform final calculations for a specific task within the manufacturing facility.

2. The system of claim 1, wherein the raw image data comprises an acquired image and a reference image. 3 . The system of claim 2 , wherein the reference image comprises a CAD-based reference image and a reference die image.

4. The system of claim 1, wherein the common backbone network is further configured to extract image features, and align and scale the image features to image data provided to the input data component.

5. The system of claim 4, wherein the image features are to be transmitted to the plurality of task-specific networks to determine defects within the manufacturing facility.

6. The system of claim 1, wherein the raw image data can be used to train a backbone network without labeling each component of the raw image data.

7. The system of claim 1, wherein the analysis engine is further configured to compare images received from the plurality of task-specific networks with relevant images in the machine learning-based comparison database generated from the raw image data.

8. The system of claim 1, wherein the network based on machine learning is based on at least one of a convolutional neural network (convnet) and a transformer-based network.

9. An image processing system for classifying defects in a manufacturing facility, the image processing system comprising: a public backbone network, the public backbone network being based on a machine learning based network; and and a plurality of task-specific networks electrically coupled to the common backbone network, each of the plurality of task-specific networks being based on a machine learning-based network separate from the common backbone network, each of the plurality of task-specific networks being configured to be trained for a specific task within the manufacturing facility.

10. The image processing system according to claim 9, wherein the public backbone network comprises: An input component, the input component is used to receive original image data; A self-supervised training component, the self-supervised training component is used to generate a machine learning-based comparison database from the raw image data; and an analysis engine for performing anomaly detection and classification of images received from the plurality of task-specific networks, the analysis engine for converting the raw image data into features to be transmitted to the plurality of task-specific networks to perform final calculations for specific tasks within the manufacturing facility.

11. The image processing system of claim 10, wherein the comparison database based on machine learning is based on at least one of a convolutional neural network (convnet) and a transformer-based network.

12. The image processing system of claim 10, wherein the analysis engine is further configured to compare images received from the plurality of task-specific networks with relevant images in the machine learning-based comparison database generated from the raw image data.

13. The image processing system of claim 10, wherein each of the plurality of task-specific networks is configured to perform defect detection and die-to-die comparison operations based on the features received from the analysis engine of the common backbone network.

14. The image processing system of claim 9, wherein each of the plurality of task specific networks is to be coupled to and collect data from at least one type of equipment including a metrology tool and an inspection tool.

15. The image processing system of claim 9, wherein each of the plurality of task-specific networks is trained to ignore normal process variations of a specific process based on a predetermined tolerance value.

16. The image processing system of claim 9, wherein each of the plurality of task-specific networks is customized with task-specific processing for a specific process.

17. The image processing system of claim 9, wherein each of the networks based on machine learning is based on at least one of a convolutional neural network (convnet) and a transformer-based network.

18. A method for comparing data in a manufacturing facility, the method comprising: receiving raw image data as input to a common backbone network, the common backbone network being based on a machine learning based network; using self-supervised training for generating a machine learning-based comparison database from the raw image data; performing anomaly detection and classification on images received from a plurality of task-specific networks electrically coupled to the common backbone network; as well as The raw image data is converted into features, and the features are transmitted to the plurality of task-specific networks.

19. The method of claim 18, further comprising performing final calculations for specific tasks within the manufacturing facility based on the characteristics.

20. The method of claim 18, further comprising using transfer learning to train a final application of each of the plurality of task-specific networks.

21. The method of claim 18, further comprising: Extract image features; as well as The image features are aligned and scaled from the raw image data to a common coordinate system within the common backbone network.

22. The method of claim 18, further comprising comparing images received from the plurality of task-specific networks with relevant images in the machine learning-based comparison database generated from the raw image data.

23. The method of claim 18, further comprising receiving images from at least one of the plurality of task specific networks from at least one type of equipment selected from equipment types including metrology tools and inspection tools.

24. The method of claim 18, wherein the network based on machine learning is based on at least one of a convolutional neural network (convnet) and a transformer-based network.

Images