Deep Learning-Based Root Cause Analysis of Process Cycle Images

Abstract

The disclosed technology relates to training a convolutional neural network (CNN) to identify and classify images of segments of an imaging chip that cause process cycle failures. The disclosed technology includes creating a training dataset of images of dimension M×N using labeled images of segments of an imaging chip of dimension J×K. The disclosed technology may fill the M×N frame using horizontal and vertical reflections along edges of the J×K labeled images positioned within the M×N frame. The pre-trained CNN is further trained using the training dataset. The trained CNN may classify the segment images as normal or as depicting a failure. The disclosed technology may train a root cause CNN to classify process cycle images of segments that cause process cycle failures. The trained CNN may classify the segment images by root cause of the process failure among multiple failure categories.

Description

Technical Field

[0001] 《Priority Application》 This application claims the benefit of U.S. Provisional Patent Application No. 63 / 143,673, entitled "DEEP LEARNING-BASED ROOT CAUSE ANALYSIS OF PROCESS CYCLES", filed on January 29, 2021 (Attorney Docket No. ILLM 1044-1 / IP-2089-PRV). The priority application is incorporated by reference for all purposes.

[0002] 《Incorporation》 The following, U.S. Patent Application No. 171 / 161,595, entitled "MACHINE LEARNING-BASED ROOT CAUSE ANALYSIS OF PROCESS CYCLE IMAGES", filed on January 28, 2021 (Attorney Docket No.: ILLM 1026-2 / IP-1911-US), U.S. Patent Application No. 17 / 332,904, entitled "MACHINE LERNING-BASED ANALYSIS OF PROCESS INDICATORS TO PREDICT SAMPLE REEVALUATION SUCCESS", filed on May 27, 2021 (Attorney Docket No.: ILLM 1027-2 / IP-1973-US), U.S. Patent Application No. 17 / 548,424, entitled "MACHINE LEARNING-BASED GENOTYPING PROCESS OUTCOME PREDICTION USING AGGREGATE METRICS", filed on December 10, 2021 (Attorney Docket No.: ILLM 1028-2 / IP-1978-US) is incorporated by reference as if fully set forth herein.

[0003] The disclosed technology relates to the classification of images for the evaluation of production processes and root cause failure analysis.

Background Art

[0004] The subject matter discussed in this section should not be assumed to be prior art merely as a result of mention in this section. Similarly, the problems mentioned in this section, or problems associated with the subject matter provided as background, should not be assumed to have been previously recognized in the prior art. The subject matter of this section merely represents a different approach and, in itself, may also correspond to embodiments of the claimed technology.

[0005] Genotyping is a process that can take several days to complete. The process is vulnerable to both mechanical and chemical processing errors. The samples collected for genotyping are extracted and distributed to sections and areas of an image generation chip. The samples are then chemically processed through multiple steps to generate a fluorescence image. The process generates a quality score for each section analyzed. This quality score cannot provide insight into the root cause of failure of a low-quality process. In some cases, the failed section images still produce an acceptable quality score.

[0006] Accordingly, there is an opportunity to introduce new methods and systems for evaluating section images and determining root cause analysis of failures during production genotyping. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0007] To solve the above problems, one aspect of the present invention provides a method for training a convolutional neural network to identify and classify images of segments of an image generation chip that result in process failures. The method includes using a pre-trained convolutional neural network to extract image features, wherein the pre-trained convolutional neural network accepts an image of dimension M×N; creating a training dataset using labeled images of dimension J×K smaller than M×N that depict process success and failure, wherein the labeled images are from segments of the image generation chip, positioning the J×K labeled images at multiple positions within the M×N frame, filling around the edges of a particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N frame; further training the pre-trained convolutional neural network to generate a segment classifier using the training dataset; and storing the coefficients of the trained classifier to identify and classify images of segments of the image generation chip from a production process cycle, whereby the trained classifier can accept an image of a segment of the image generation chip and classify the image as depicting process success and failure.

[0008] In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale; instead, emphasis has been placed on illustrating the principles of the disclosed technology. In the following description, various embodiments of the disclosed technology are described with reference to the following drawings.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 5H

Figure 5I

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 9

Figure 10A

Figure 10B

Figure 11A

Figure 11B

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 12E

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

DETAILED DESCRIPTION OF THE INVENTION

[0010] The following considerations are presented to enable one of ordinary skill in the art to make and use the disclosed technology and are provided in relation to a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Accordingly, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. Introduction

[0011] The disclosed technology applies vision systems and image classification for the evaluation of production genotyping and root cause failure analysis. Three separate approaches are described, the first with eigen-images, the second based on thresholding by area, and the third using deep learning models such as convolutional neural networks (or CNNs). Principal component analysis (PCA) and non-negative matrix factorization (NMF) are among the disclosed technology. Other dimensionality reduction techniques that can be applied to images include independent component analysis, dictionary learning, sparse principal component analysis, factor analysis, mini-batch K-means. Variations of image decomposition and dimensionality reduction techniques can be used. For example, PCA can be implemented using singular value decomposition (SVD) or as kernel PCA. The output from these techniques is given as input to a classifier. Classifiers that can be applied include random forest, K-nearest neighbors (KNN), multinomial logistic regression, support vector machine (SVM), gradient boosting decision tree, naive Bayes, etc. Since a large body of labeled images is available, convolutional neural networks such as ResNet, VGG, ImageNet can also be used as presented below in the description of the third image processing technique.

[0012] The genotyping production process is vulnerable to both mechanical and chemical processing errors. The samples collected are extracted, partitioned and assigned to BeadChip sections and areas, and then chemically processed through multiple steps to generate fluorescence images. The final fluorescence image, or even intermediate fluorescence images, can be analyzed to monitor production and perform failure analysis.

[0013] Most of the production analysis is successful. Current failure analysis is understood to fit into five categories and the remaining failure categories. The five failure categories are hybridization or hyb failure, spacer shift failure, offset failure, surface wear failure, and reagent flow failure. The remaining categories are unsound patterns due to mixed effects, unidentified causes, and weak signals. At that time, more different causes can be identified, especially since root cause analysis leads to improved production.

[0014] The first image processing technology applied to quality control and failure analysis has evolved from face recognition by eigenface analysis. From tens of thousands of labeled images, linear criteria for 40 - 100 image components were identified. One approach for forming eigen criteria was to rank the components according to the scale of the variance explained after principal component analysis (PCA). It was observed that 40 components explained most of the variance. Beyond 100 components, additional components appeared to reflect noise or natural variance patterns in sample processing. The number of relevant components is expected to be affected by image resolution. Here, resolution reduction was applied such that the section of the image generation chip was analyzed at a resolution of 180×80 pixels. This was sufficient to distinguish success from unsuccessful production and then classify the root causes of failure among the six failure categories. Formal sensitivity analysis was not applied, but images with slightly lower resolution also functioned, and images 4 - 22 times this resolution could be processed in the same way, but with increased computational cost. Each image analyzed by eigenimage analysis is represented as a weighted linear combination of reference images. Each weight for the ordered set of reference components is used as a feature for training a classifier. For example, in one embodiment, 96 weights of the components of the labeled images were used to train a random forest classifier. A random forest classifier with 200 trees and a depth of 20 functioned well. Two tasks were performed by the random forest classifier: separation of successful and unsuccessful production images and then root cause analysis of the unsuccessful production images. This two - stage classification was selected due to the dominance of successful production runs, but a one - stage classification could also be used.

[0015] The second image processing technique applied involved thresholding of the image area. The production image of the section of the image generation chip captures several physically separated areas. Structures that border the section and separate distinct physical areas of the section are visible in the production image. The thresholding strategy involves separating the active area from the boundary structure and then differentiating the separated areas. Optionally, the structures separating the physical areas can also be filter-removed from the image. At least the active area undergoes thresholding of luminescence. The thresholding determines what amount of the active area produces the desired signal intensity. Each active area is evaluated after successful or failed thresholding. The pattern of failures between the areas and sections of the image generation chip can be further evaluated for root cause classification.

[0016] The processing of the production image to detect failed production runs and determine root causes can be performed immediately during production, reading from the image generation chip more quickly than the results, and the quality is judged. On the side, reducing the pixel-based image size to 1 / 20 of the original size significantly reduces the computational requirements, and the direct processing of the reduced-resolution image does not require the correlation of individual luminescent pixels within the area with individual probes, so this image processing can be performed quickly. The short-turnaround root cause analysis can be used to correct upstream processes before chemicals and processing time are wasted.

[0017] The third image processing technique involves applying deep learning models such as convolutional neural networks (CNNs). ResNet (He et al., CVPR 2016, available at <<arxiv.org / abs / 1512.03385>>) and VGG (Simonyan et al., 2015, available at <<arxiv.org / abs / 1409.1556>>) are examples of convolutional neural networks (CNNs) used for identification and classification. The inventors applied the ResNet-18 and VGG-16 architectures of each model to detect failed images and classify the failed images into their respective failure categories. The CNN model parameters are pre-trained on the ImageNet dataset (Deng et al., 2009, "ImageNet: A large-scale hierarchical image database" published in the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255) containing approximately 14 million images. The pre-trained model is fine-tuned using the labeled images of the segments of the image generation chip. Approximately 75,000 labeled images of the segments are used to fine-tune the pre-trained CNN. The training data consists of normal images from successful process cycles and abnormal (or defective, or failed) images of the segments from failed process cycles. The images from the failed process cycles belong to the five failure categories presented above.

[0018] The ResNet-18 and VGG-16 CNN models can use square input images of size 224×224 pixels. In one embodiment, the segments of the image generation chip are rectangular, such as 180×80 pixels as described above. Larger image size segments can be used. The disclosed technique applies feature engineering to create training data using labeled rectangular images that may be smaller than the 224×224 pixel size image required as input to the CNN model. The technique can apply three feature engineering techniques, including cropping, zero-padding, and reflection-padding, to create the input data set.

[0019] In cropping, the central portion of the rectangular shaped segment image of the image generation chip is cropped to a size of 224×224 pixels. In this case, the input image of the segment is larger than 224×224 pixels. In one embodiment, the input segment image is of size 504×224 pixels. Other sized labeled images larger than 224×224 can be used to crop out the square portion. The input image is cropped to match the input image size (224×224 pixels) required by the CNN model.

[0020] In zero-padding, the size of the labeled input image is smaller than 224×224 pixels. For example, the input image can be 180×80 pixels. The input labeled image of a smaller size (such as 180×80 pixels) is placed within a 224×224 pixel analysis frame. The image can be placed at any position within the analysis frame. The pixels in the peripheral area within the analysis frame can be zero-padded, in other words, zero image intensity values are assigned to the peripheral pixels. Then, the larger sized analysis frame can be provided as input to the CNN model.

[0021] In reflection padding, a labeled input image of a smaller size can be placed at the center of an analysis frame of a larger size. Next, the labeled image is reflected horizontally and vertically along the edges to fill the surrounding pixels within the analysis frame of a larger size (224×224 pixels). The reflected labeled image is provided as an input to the CNN model. Since the features within the input labeled image are copied to multiple positions within the analysis frame of a larger size, reflection padding can yield favorable results.

[0022] The disclosed technique can perform data augmentation to increase the size of the training data. Horizontal and vertical translations can be performed by placing a rectangular labeled input image of a smaller size (J×K pixels) at multiple positions within an analysis frame of a larger size (M×N pixels). In one embodiment, the rectangular labeled input image is 180×80 pixels in size, and the analysis frame of a larger size is 224×224 pixels. Other sizes of the labeled input image and the analysis frame can be used. In one embodiment, the J×K image may be systematically translated horizontally, vertically, or diagonally within the M×N analysis frame to generate additional training data. In one embodiment, the J×K image may be randomly positioned at different locations within the M×N analysis frame to generate additional training data.

[0023] A two-step detection and classification process can be applied using two separately trained convolutional neural networks (CNNs). The first CNN is trained for the detection task, and the trained classifier can classify an image as normal or as depicting a process failure. Images of failed process cycles can be fed to a second CNN trained to classify the images by the root cause of the process failure. In one embodiment, the system can classify images into five different types of failure types enumerated above. The two-step process can be combined into a one-step process using a CNN to classify production segment images as normal or as belonging to one of the failure categories. environment

[0024] The inventors describe a system for early prediction of failures in a genotyping system. Genotyping is the process of determining differences in an individual's genetic structure (genotype) by using a biological assay to examine the individual's DNA sequence and comparing it to a reference sequence. Genotyping enables researchers to search for genetic variants such as single nucleotide polymorphisms (SNPs) and structural changes in DNA. The system is described with reference to FIG. 1, which shows a schematic diagram of the architecture level of the system according to an embodiment. Since FIG. 1 is an architecture diagram, certain details are intentionally omitted to improve the clarity of the description. The discussion of FIG. 1 is organized as follows. First, the elements of the figure are described, followed by their interconnections. Then, the use of the elements within the system is described in more detail.

[0025] FIG. 1 includes a system 100. This paragraph names the labeled parts of system 100. The figure illustrates a genotyping device 111, a process cycle image database 115, a failure category label database 117, a labeled process cycle image database 138, a trained good-versus-bad classifier 151, a reference database of eigen-images 168, a trained root cause classifier 171, a feature generator 185, and a network 155.

[0026] The disclosed technology is applicable to various genotyping devices 111, also referred to as genotyping scanners and genotyping platforms. The network 155 couples the genotyping device 111, the process cycle image database 115, the failure category label database 117, the labeled process cycle image database 138, the trained good-versus-bad classifier 151, the reference database of eigen-images 168, the trained root cause classifier 171, and the feature generator 185 to communicate with each other.

[0027] The genotyping device may include an Illumina BeadChip imaging system such as the ISCAN (trademark) system. The device may detect the fluorescence intensity of hundreds to millions of beads arranged in segments at mapped locations on the image generation chip. The genotyping device may include a device control computer that controls various aspects of the device, such as laser control, precision machine control, detection of excitation signals, image registration, image extraction, and data output. The genotyping device is used in a wide variety of physical environments and can be operated by technicians with various skill levels. Sample preparation can take two to three days and may include manual and automated handling of the sample.

[0028] The inventors illustrate the process steps of an exemplary genotyping process 300 of FIG. 3. This exemplary genotyping process is referred to as the Illumina INFINIUM™ assay workflow. The process is designed to survey many SNPs with a wide range of locus multiplexing. Using a single bead type and a two-color (red and green) channel approach, the process scales the genotyping of hundreds to millions of SNPs per sample. The process begins with the receipt and extraction of a DNA sample. The process can operate with a relatively low input sample, such as 200 ng, that can assay millions of SNP loci. The sample is amplified. The amplification process can take several hours to overnight to complete. The amplified sample undergoes controlled enzymatic fragmentation. This is subsequently subjected to alcohol precipitation and resuspension. An imaging chip is prepared for hybridization in a capillary flow chamber. The sample is then applied to the prepared imaging chip and incubated overnight. During this overnight hybridization, the sample anneals to locus-specific 50mers covalently attached to up to millions of bead types. One bead type corresponds to each allele per SNP locus. Allele specificity is conferred by enzymatic base extension, followed by fluorescent staining. A genotyping instrument or scanner (such as an iScan™ system) detects the fluorescence intensity of the beads and performs genotype determination.

[0029] In one example, the results of genotyping are presented using a measurement criterion called the "call rate". This measurement criterion represents the percentage of genotypes that were correctly scanned on the image generation chip. A separate call rate is reported for each section of the image generation chip. A threshold value can be used to accept or reject the results. For example, a call rate of 98% or higher can be used to accept the genotyping results for a section. Different threshold values such as less than 98% or more than 98% can be used. If the call rate of a section falls below the threshold value, the genotyping process is considered a failure. The genotyping process can take many days and is therefore expensive to repeat. Failures in the genotyping process can occur due to operational errors (such as mechanical or handling errors) or chemical processing errors.

[0030] The genotyping system can provide, upon completion of the genotyping process, the process cycle images of the sections of the image generation chip, along with the respective call rates of the sections. The disclosed technology can process these section images to classify whether the genotyping process was successful (good images of the sections) or not (bad or failed images of the sections). The disclosed technology can further process the bad or failed images to determine the category of failure. Currently, the system can classify a failed image into one of six failure categories: hybridization or hyb failure, spacer shift failure, offset failure, surface wear failure, reagent flow failure, and overall unsound images due to mixed effects, unknown causes, weak signals, etc. At that time, more different causes can be identified, especially since root cause analysis leads to improved production.

[0031] Here, the inventors refer to FIG. 1 to provide an explanation of the remaining components of system 100. Failure category labels for the six failure types can be stored in the failure category label database 117. The training data set of labeled process image cycles is stored in database 138. The labeled training examples can consist of successful (good) and unsuccessful (bad) process cycle images. The unsuccessful process cycle images are labeled as belonging to one of the six failure categories listed above. In one embodiment, the training database 138 consists of at least 20,000 training examples. In another embodiment, the size of the training data set is increased to 75,000 training examples using feature engineering techniques. The size of the training database can increase as more labeled image data is collected from the laboratory using the genotyping instrument.

[0032] The disclosed technology includes three independent image processing techniques for extracting features from process cycle images. The feature generator 185 can be used to apply one of the three techniques to extract features from the process cycle images for input to the machine learning model. The first image processing technique has evolved from face recognition by eigenface analysis. A relatively small number of linear criteria, such as 40 to 100 or more image components, are identified from tens of thousands of labeled images. One approach for forming the eigencriteria is principal component analysis (PCA). The production cycle images are represented as a weighted linear combination of the reference images for input to the classifier. For example, in one embodiment, 96 weights of the components of the labeled images are used to train the classifier. The eigenimage criteria can be stored in database 168.

[0033] The second image processing technique for feature extraction involves thresholding the segmented image. The segmented production image of the image generation chip captures several physically separated areas. Structures that are in contact with the segments and separate distinct physical areas of the segments are visible in the production image. The thresholding technique determines the amount of active area that generates the desired signal strength. The output from the thresholding technique can be provided as input to a classifier to distinguish between defective and good images. The pattern of failures between the areas and segments of the image generation chip can be further evaluated for root cause analysis.

[0034] The third image processing technique includes various feature engineering techniques for preparing the image for input to a deep learning model. The segmented image of the image generation chip is rectangular. The image generation chip can have 12, 24, 48, or 96 segments arranged in two or more columns. The convolutional neural network (CNN) applied by the disclosed technique requires a square input image. Thus, the system includes logic for positioning the rectangular (J×K pixel) segmented image within a square (M×N pixel) analysis frame.

[0035] The system can apply one or more of the following feature engineering techniques. The system can apply zero-padding to fill the pixels surrounding the segmented image within a larger square analysis frame. The system can crop the central portion of the segmented image with square dimensions and fill the analysis frame with the cropped segmented image. In this case, the segmented image is larger in dimension than the analysis frame. When the labeled segmented image is smaller than the analysis frame, the system can position the labeled input image within the analysis frame. The system can use horizontal and vertical reflections along the edges of the input segmented image to fill a larger-sized analysis frame. By using translations where the labeled input image is positioned at multiple locations within the analysis frame, the system can augment the labeled training data.

[0036] The image features of the production images generated by the feature generator 185 are provided as inputs to the trained classifiers 151 and 171. Two types of classifiers are trained. The good-versus-bad classifier can predict successful and unsuccessful production images. The root cause analysis classifier can predict the failure category of the unsuccessful images. In one embodiment, the classifier used by the disclosed technique includes a random forest classifier. Other examples of classifiers that can be applied include the k-nearest neighbor method (KNN), multinomial logistic regression, and support vector machines. In another embodiment of the disclosed technique, a convolutional neural network (CNN) is applied to identify and classify the images of the sections of the image generation chip.

[0037] Once the description of FIG. 1 is complete, all of the components of the system 100 described above are connected to communicate with the network 155. The actual communication path can be point-to-point via public and / or private networks. Communication can occur across various networks, such as private networks, VPNs, MPLS circuits, or the Internet, and can use appropriate application programming interfaces (APIs) and data exchange formats, such as Representational State Transfer (REST), JavaScript Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), Java Message Service (JMS), and / or Java Platform Module System. All communication can be encrypted. Communication generally occurs via networks such as local area networks (LANs), wide area networks (WANs), telephone networks (Public Switched Telephone Network (PSTN), Session Initiation Protocol (SIP)), wireless networks, point-to-point networks, star networks, token ring networks, hub networks, mobile Internet via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX, and the Internet. The engine or system components of FIG. 1 are implemented by software operating on various types of computing devices. Exemplary devices are workstations, servers, computing clusters, blade servers, and server farms. In addition, various authorization and authentication techniques, such as username / password, Open Authorization (OAuth), Kerberos, Secured, digital certificates, etc., can be used to secure communication. Feature Generator - System Component

[0038] FIG. 2 is a high - level block diagram of the components of the feature generator 185. These components are computers implemented using various different computer systems as presented below in the description of FIG. 16. The illustrated components may be combined or further separated when implemented. The feature generator 185 consists of three high - level components implementing three image - processing techniques: a principal - component - analysis or PCA - based feature generator 235, an image - segmentation - based feature generator 255, and a training - data generator 275 for CNN. The PCA - based feature generator consists of an image scaler 237 and a reference creator 239 for eigen - images. The image - segmentation - based feature generator 255 consists of an image converter 257 and an intensity extractor 259. The training - data generator 275 for CNN comprises an image cropper 277, an image translator 279, and an image reflector 281. In the following sections, the inventors present further details of the implementation of these components. PCA - based Feature Generator

[0039] The first image - processing technique has evolved from face recognition by eigen - face analysis. One approach for forming eigen - references is principal - component analysis (PCA). The PCA - based feature generator 235 applies PCA to the resized process image. The image scaler component 237 resizes the process - cycle image. The scaling reduces the size of the process image so that the process image can be processed in a computationally efficient manner by the reference creator component 239 for eigen - images. The inventors present details of these components in the following sections. Image Scaler

[0040] Higher resolution images obtained from a genotyping instrument or scanner may require more computational resources to process. Images obtained from a genotyping scanner are resized by an image scaler 237 such that the images of the sections of the image generation chip are analyzed at a reduced resolution, such as 180×80 pixels. Throughout this specification, the inventors refer to scaling (or rescaling) as resampling the image. Resampling changes the number of pixels in the displayed image. Increasing the number of pixels in the initial image increases the image size and is referred to as upsampling. Decreasing the number of pixels in the initial image decreases the image size and is referred to as downsampling. In one embodiment, the section images obtained from the scanner are at a resolution of 3600×1600 pixels. In another embodiment, the section images obtained from the scanner are at a resolution of 3850×1600 pixels. These original images are downsampled to reduce the size of the image in pixel units to 1 / 20th per side from the original resolution. This is sufficient resolution to distinguish successful production images from unsuccessful production images and then classify the root cause of failure among the six failure categories. The images may be downsampled to 1 / 4 to 1 / 40 of the number of pixels per side at the original resolution and processed in the same way. In another embodiment, the images may be downsampled to 1 / 2 to 1 / 50 of the number of pixels per side at the original resolution and processed in the same way. Exemplary techniques for resampling high resolution images are presented below.

[0041] The disclosed technology can apply various interpolation techniques to reduce the size of production images. In one embodiment, bilinear interpolation is used to reduce the size of the segmented image. Linear interpolation is a method of curve fitting using a linear polynomial to construct new data points within the range of individual sets of known data points. Bilinear interpolation is an extension of linear interpolation for interpolating functions of two variables (e.g., x and y) on a two-dimensional grid. Bilinear interpolation is performed first in one direction and then again in a second direction using linear interpolation. Each step is linear in the sampled values and positions, but the overall interpolation is not linear, but rather quadratic at the sample locations. Other interpolation techniques can also be used to reduce (rescale) the size of the segmented image, such as neighborhood interpolation and resampling using pixel area relationships. Canonicalizer for Eigen Images

[0042] The first image processing technique applied to the segmented image to generate input features for the classifier has evolved from face recognition by eigenface analysis. From tens of thousands of labeled images, linear canonical forms of 40 to 100 image components are identified. One approach for forming the canonical forms of eigen images is principal component analysis (PCA). A set B of elements (vectors) in a vector space V is called a basis if every element of V can be written in a unique way as a linear combination of the elements of B. Equivalently, B is a basis if its elements are linearly independent and every element of V is a linear combination of the elements of B. A vector space can have several bases. However, all bases have the same number of elements called the dimension of the vector space. In the inventors' technology, the basis of the vector space is the eigen image.

[0043] PCA is often used to reduce the dimension of a d-dimensional dataset by projecting it onto a k-dimensional subspace, where k < d. For example, the resized labeled images in our inventors' training database represent vectors in a d = 14,400-dimensional space (180 x 80 pixels). In other words, an image is a point in a 14,400-dimensional space. The eigen-space based approach approximates the image vectors with lower-dimensional feature vectors. The main assumption behind this technique is that the image space given by the feature vectors has a lower dimension than the image space given by the number of pixels in the image, and that image recognition can be performed in this reduced space. The sectional images of the image generation chip, which are similar in overall configuration, will not be randomly distributed in this large space and can therefore be explained by a relatively low-dimensional subspace. The PCA technique finds the vectors that best explain the distribution of the sectional images within the entire image space. These vectors define a subspace of the image, also referred to as the "image space". In our inventors' embodiments, each vector represents an 180 x 80 pixel image and is a linear combination of the images in the training data. In the following text, our inventors present details on how principal component analysis (PCA) can be used to create a criterion for eigen-images.

[0044] The PCA-based analysis of labeled training images can consist of the following five steps. Step 1: Accessing multi-dimensional correlation data

[0045] The first step in the application of PCA is to access high-dimensional data. In one example, a PCA-based feature generator used 20,000 labeled images as training data. Each image was resized to 180 x 80 pixel resolution and represented as a point in a 14,400-dimensional space with one dimension per pixel. This technique can handle images with higher or lower resolutions than those specified above. The size of the training dataset is expected to increase as our inventors collect more labeled images from the laboratory. Step 2: Data Standardization

[0046] Standardization (or Z-score normalization) is a process of rescaling features such that they have the properties of a Gaussian distribution with a mean of 0, i.e., μ = 0, and a standard deviation from the mean of 1, i.e., σ = 1. Standardization is performed to construct features that have similar ranges to each other. The standard score of an image can be calculated by subtracting the mean (image) from the image and dividing the result by the standard deviation. When PCA obtains a feature subspace that maximizes the variance along the axes, it helps to standardize the data so that the data is centered around the axes. Step 3: Calculating the Covariance Matrix

[0047] The covariance matrix is a d×d matrix in a d-dimensional space, where each element represents the covariance between two features. The covariance between two features measures the tendency for them to vary together. Variance is the average of the squared deviations of a feature from its mean. Covariance is the average of the product of the deviations of the feature values from their means. Consider feature k and feature j. Let {x(1,j), x(2,j),..., x(i,j)} be the set of i examples of feature j and {x(1,k), x(2,k),..., x(i,k)} be the set of i examples of feature k. Similarly,

Number

Number

Number

Number

Number

[0048] The average vector is a d-dimensional vector, and each value within this vector represents the sample average of the feature columns in the training dataset. The covariance value σ jk is between the negative linear correlation of “-(σ ij )(σ ik )”, that is, the positive linear correlation of “+(σ ij )(σ ik )” and can vary. When there is no dependency between two features, the value of σ jk is zero. Step 4: Calculating the eigenvectors and eigenvalues

[0049] The eigenvectors and eigenvalues of the covariance matrix represent the core of PCA. The eigenvectors (or principal components) determine the directions of the new feature space, and the eigenvalues determine their magnitudes. In other words, the eigenvalues explain the dispersion of the data along the axes of the new feature space. Eigenvalue decomposition is a method of matrix decomposition by representing a matrix using its eigenvectors and eigenvalues. An eigenvector is defined as a vector that changes only by a scalar when a linear transformation is applied to the vector. If A is a matrix representing a linear transformation, v is an eigenvector, and λ is the corresponding eigenvalue, it can be expressed as Av = λv. A square matrix can have many eigenvectors because it has dimensions. If the present inventors represent all eigenvectors as columns of matrix V and the corresponding eigenvalues as entries of diagonal matrix L, the above equation can be expressed as AV = VL. In the case of the covariance matrix, all eigenvectors are orthogonal to each other and are the principal components of the new feature space. Step 5: Using the dispersion described to select a criterion for the eigenimages

[0050] The above steps can result in 14,400 principal components for the embodiments of the present inventors, equal to the dimension of the feature space. An eigenpair consists of an eigenvector and a scalar eigenvalue. The present inventors can select eigenpairs based on the eigenvalues and create criteria for eigenimages using a measure called "explained variance". The explained variance indicates the amount of information (or variance) that can be attributed to each of the principal components. The present inventors can plot the results of the measured values explained in a two-dimensional graph. The selected principal components are represented along the x-axis. A graph showing the cumulatively explained variance can be plotted. The first m components representing the major part of the variance can be selected.

[0051] In the embodiments of the present inventors, the first 40 components represent a high percentage of the explained variance. Therefore, the present inventors selected the first 40 principal components to form the criteria for their new feature space. In other embodiments, 25 to 100 principal components or more than 100 principal components, up to 256 or 512 principal components can be selected to create criteria for eigenimages. Each production image analyzed by eigenimage analysis is represented as a weighted linear combination of the reference images. Each weight of the ordered set of reference components is used as a feature for training the classifier. For example, in one embodiment, 96 weights for the components of the labeled images were used to train the classifier.

[0052] The disclosed technology may use other image decomposition and dimensionality reduction techniques. For example, non-negative matrix factorization (NMF), which learns a part-based representation of an image, as compared to PCA that learns a complete representation of an image. Unlike PCA, NMF learns to represent an image as having a set of reference images that resemble parts of the image. NMF factorizes a matrix X into two matrices W and H, where all three matrices have the property of not having negative elements. Since the inventors assume that matrix X is set, there are n data points (such as segmented images on an image generation chip), each having p dimensions (e.g., 14,400). Thus, matrix X has p rows and n columns. The inventors wish to reduce the p dimensions to r dimensions, or in other words, create a rank-r approximation. NMF approximates matrix X as the product of two matrices: W (p rows and r columns) and H (r rows and n columns).

[0053] The interpretation of matrix W is that each column is a reference element. By reference elements, the inventors mean averaging some of the components present in the n original data points (or images). These are components that the inventors can reconstruct an approximation for all of the original data points or images. The interpretation of matrix H is that each column gives the coordinates of the data points with respect to the reference matrix W. In other words, it shows the inventors how to reconstruct an approximation from a linear combination of the components of matrix W to the original data point. In the case of face images, the reference elements (or reference images) of matrix W may include features such as eyes, nose, lips, etc. The columns of matrix H indicate which features are present in which image. Image segmentation-based feature generator

[0054] The second image processing technique for extracting features from the process cycle image is based on the thresholding of the image area. The image segmentation-based feature generator 255 first segments the image of the section of the image generation chip using the image segmenter 257, and then applies thresholding by extracting the intensity of the active area or the region of interest of the segmented image. Thresholding determines how much amount of the active area produces the desired signal intensity.

[0055] The image generation chip consists of a plurality of sections such as 24, 48, 96 or more, and can be organized into rows and columns. This design enables the processing of multiple samples (one per section) in parallel, allowing the processing of multiple samples in one process cycle. The sections are physically separated from other sections so that the samples are not mixed with each other. In addition, the sections can be organized into a plurality of parallel regions called "slots". Therefore, the structure of the boundaries of the sections and slots is visible in the process cycle image from the genotyping scanner. The inventors present the details of two components of the image segmentation-based feature generator 255 that can implement the technique for converting the segmented image for extracting image features below. Image converter

[0056] The image converter 257 applies a series of image conversion techniques to prepare a segmented image for extracting the intensity from the region of interest. In one embodiment, this process of image conversion and intensity extraction is performed by some or all of the following five steps. Image conversion converts the grayscale image of the section into a binary image consisting of black and bright pixels. The average intensity values of the active areas of the grayscale image and the binary image are given as input features to a classifier for classifying the image as a healthy (good) or unhealthy (bad) image. In the following text, the inventors present the details of the image conversion step including applying thresholding to convert the grayscale image into a binary image. The process step includes applying a filter to remove noise.

[0057] The first step of the image conversion process is to apply a bilateral filter to the segmented process cycle image. The bilateral filter is a technique for smoothing an image while preserving edges. It replaces the intensity of each pixel with a weighted average of the intensity values from its neighboring pixels. Each neighborhood is weighted by a spatial component that penalizes distant pixels and a range component that penalizes pixels with different intensities. The combination of both components ensures that only nearby similar pixels contribute to the final result. Thus, the bilateral filter is an efficient way to smooth an image while preserving its discontinuities or edges. Other filters such as median filters and anisotropic diffusion may be used.

[0058] The second step of the image conversion includes applying thresholding to output the image from step 1. In one embodiment, the inventors use the intensity histogram and apply Otsu's method (Otsu, N., 1979, "A threshold selection method from gray-level histograms", IEEE Transactions on Systems, Man, and Cybernetics, Volume 9, Issue 1) to search for a threshold that maximizes the weighted sum of the gray-scale variances between the pixels assigned to the dark and bright intensity classes. Otsu's method attempts to maximize the variance between classes. The basic idea is that well-thresholded classes should be different with respect to the intensity values of their pixels, and conversely, the threshold that gives the best separation between classes with respect to their intensity values is the best threshold. Additionally, Otsu's method has the property of being completely based on calculations performed on the histogram of the image, which is a one-dimensional array that is easily obtainable. For further details on Otsu's method, see Section 10.3.3 of Gonzalez and Woods, "Digital Image Processing", 3 rd Edition.

[0059] The third step of the image conversion is the application of a noise reduction Gaussian blur filter to remove the speckle-like noise. The noise can contaminate the process cycle image with small speckles. Gaussian filtering is the weighted average of the intensities of adjacent positions with weights that decrease with the spatial distance to the central position.

[0060] The fourth step of the image conversion involves image morphological operations. The binary output image from the third step is processed by morphological transformation to fill the holes in the image. The holes can be defined as background regions (represented by 0s) surrounded by connected boundary lines of foreground pixels (represented by 1s). Two basic image morphological operations are "erosion" and "dilation". In the erosion operation, the kernel slides (or moves) over the binary image. If all the pixels under the kernel are 1s, the pixel (1 or 0) in the binary image is considered 1. Otherwise, it is eroded (changed to 0). The erosion operation is useful for removing isolated 1s in the binary image. However, erosion also shrinks the clusters of 1s by shrinking the edges. The dilation operation is the opposite of erosion. In this operation, when the kernel slides over the binary image, if the value of at least one pixel under the kernel is 1, the values of all the pixels in the overlapping binary image area by the kernel are changed to 1. When the dilation operation is applied to the binary image and then the erosion operation is applied, the effect is to close the small holes (represented by 0s in the image) within the clusters of 1s. The output from this step is provided as the input to the intensity extractor component 259 that performs the fifth step of this image conversion technique. Intensity extractor

[0061] The intensity extractor 259 divides the segmented image into active regions or segments by filtering out the structure at the boundaries of the segments and slots. The intensity extractor can divide the segmented image into active areas from 8 to 17 by applying different segmentations. Examples of areas within the segmented image include four slots, four corners, four edges between the corners, and various vertical and horizontal lines at the boundaries of the segments and slots. Next, the areas corresponding to known structures that separate the active areas are removed from the image. The image portions of the remaining active areas are processed by the intensity extractor 259. Intensity values are extracted and averaged for each active area of the transformed image and the corresponding untransformed image. For example, if intensity values are extracted from 17 active areas of the transformed image, the intensity extractor also extracts intensity values from the same 17 active areas of the untransformed image. Thus, a total of 34 features are extracted for each segmented image.

[0062] In the case of a binary image, the average intensity of the active area can be from 1 to 0. For example, consider that the intensity of a black pixel is 0 and the intensity of a bright (or blank) pixel is 1. If all pixels within the active area are black, the average intensity of the active area is 0. Similarly, if all pixels within the active area are bright, the intensity of that area is 1. The active areas of a healthy image appear blank or bright in a binary image, while black pixels represent an unhealthy image. The average intensity of the corresponding active areas in a grayscale image is also extracted. The average intensities of the active areas from both the grayscale image and the transformed binary image are provided as inputs to a good-versus-bad classifier. In one embodiment, the classification confidence score from the classifier is compared to a threshold to classify the image as a healthy (or good, or successful) image or an unhealthy (or bad, or failed) image. An example of the threshold is 80%. The higher the value of the threshold, the more images will be classified as unhealthy as a result. Training data generator for convolutional neural network (CNN)

[0063] The third image processing technique may use cropping, translation, and reflection to prepare input images for training a convolutional neural network (CNN). The segmented images are rectangular, and CNNs such as ResNet (He et al., CVPR 2016, available at <<arxiv.org / abs / 1512.03385>>) and VGG (Simonyan et al., 2015, available at <<arxiv.org / abs / 1409.1556>>) use square input images of size 224 x 224 pixels. A training data generator 275 for the CNN includes an image cropper 277, an image translator 279, and an image reflector 281 to prepare input images for the CNN. The image translator 279 may also augment the labeled input images to increase the size of the training data. Image cropper

[0064] The image cropper 277 includes logic for cropping the segmented image of the image generation chip to match the input image size required by a convolutional neural network (CNN). In one embodiment, the high-resolution image obtained from the genotyping device has dimensions of 3600 x 1600 pixels. As described above, high-resolution images may require a lot of computational resources for processing, and thus the disclosed techniques downsample the images to reduce their size. The size of the image is reduced from the pixel size of the original image to 1 / 4 to 1 / 40 per side, and thus an image with a size in the range of 964 x 400 pixels to 90 x 40 pixels is obtained. In one example, the segmented image is downsampled by a factor of 1 / 20 per side from the size of the original image of 3600 x 1600 pixels, resulting in an image of size 180 x 80 pixels (J x K image). The downsampled image may be supplied to the image cropper to prepare an input as required by a convolutional neural network (CNN).

[0065] The image cropper can cut out various parts of the rectangular partitioned image to prepare a square input image for the CNN. For example, the central part of the rectangular shaped partitioned image of the image generation chip can be cut out and placed within a 224×224 pixel analysis frame. If the input image of the partition is larger than 224×224 pixels, such as 504×224 pixels, the image cropper can cut out 224×224 pixels from the image and fill the M×N analysis frame. The image cropper can cut out a part smaller than 224×224 pixels for placement within the M×N analysis frame. Image Translator

[0066] The image translator 279 includes logic for positioning the labeled input image of the partition of the image generation chip at multiple positions within the analysis frame. "Translation" can refer to moving the shape or partitioned image in this case without rotating or flipping. Translation can also refer to sliding the partitioned image within the analysis frame. After translation, the partitioned image looks the same and has the same dimensions but is positioned at a different location within the analysis frame. The image translator includes logic for moving or sliding the partitioned image horizontally, vertically, or diagonally at different positions within the analysis frame.

[0067] The analysis frame can be made larger than the size of the labeled input image. In some cases, the analysis frame is a square with a size of 224×224 pixels. Analysis frames of other sizes and shapes can be used. The labeled input image of the partition is positioned at multiple positions within the analysis frame. When the image size is 180×80 pixels and the analysis frame is 224×224 pixels, the image can be translated horizontally and vertically. In another example, when the image size is 224×100 pixels, the image can be translated only horizontally within the 224×224 pixel analysis frame. In another example, when the image size is 200×90 pixels, the image can be translated horizontally and vertically within the 224×224 pixel analysis frame.

[0068] Pixels within the analysis frame surrounding the labeled input image can be zero-padded. In this case, a "0" value is assigned to each intensity for the pixels surrounding the labeled input image. The translation of one input labeled image can result in multiple analysis frames containing the input image at different positions within the frame. This process can expand the training data and thus increase the number of training examples. Image reflector

[0069] The image reflector 281 includes logic for reflecting a smaller-sized input labeled image positioned within a larger-sized analysis frame both horizontally and vertically. Reflection refers to the image of an object as seen in a mirror, or in this case, the segmented image. The smaller-sized labeled input image can be placed at the center of the larger-sized analysis frame. The image reflector 281 includes logic for reflecting the segmented image along the edges both horizontally and vertically to fill the surrounding pixels within a larger-sized analysis frame (such as 224×224 pixels). Since the image or a part of the image is copied to multiple positions within the analysis frame, the reflective padding can increase the probability of detecting a failed process image. Process cycle image

[0070] Here, the inventors present examples of successful and unsuccessful production images on the image generation chip. FIG. 4 is an illustration 400 of production images of 24 sections on the image generation chip. The sections are arranged in 12 rows and 2 columns. Each section has 4 slots. Illustration 400 shows the section images of a successful production cycle. Image generation chips having other configurations of sections, such as including 48, 96, or more sections, may also be used. In the following figures, the inventors present examples of section images of unsuccessful production cycles. The production process is vulnerable to both operational and chemical processing errors. Operational defects can be caused by mechanical or sample handling problems. Chemical processing errors can be caused by problems with the sample or the chemical processing of the sample. The disclosed technology attempts to classify defective process image cycles that occur due to both operational and chemical processing errors.

[0071] FIG. 5A shows an example 510 of a section image from an unsuccessful production cycle. The image of section 512 in the second column and the seventh row of the image generation chip in FIG. 5A is dark in the bottom half and slightly lighter in the upper part. The cause of this failure is related to the hybridization process. Therefore, the failed image of the section is labeled as a "Hyb" failure. Hybridization failures can also occur due to the failure of the robot that handles the sample during the sample preparation process on the image generation chip. The call rate for this section is "97.545", which is less than the 98 percent pass threshold. In some cases, the call rate for a section from the genotyping instrument can exceed the pass threshold, and then, furthermore, the section image can fail due to hybridization errors.

[0072] In illustration 510, it may be noted that the image of section 514 at row 11 and column 2 has a dark region on the right wall. This may also indicate a processing problem, but the overall call rate of this image exceeds the passing threshold and is not labeled as a failed image. There is sufficient redundancy of samples on the section due to which small areas of the section with apparent failures can be ignored and may not cause an error in the result. For example, in one instance, the scanner reads fluorescence from about 700K probes on a section with 10 redundancy. Therefore, the call rate is based on the reading of about 7 million probes. The inventors present a further example of hybridization failure in illustration 515 of FIG. 5B. The four sections on the image generation chip with the dashed boundary show defective production images of the sections due to hybridization failure. It should be noted that the call rate values for these four sections exceed the passing threshold, but the images of these sections are labeled as failed due to hybridization errors.

[0073] Figure 5C presents an illustration 520 of nine sectional images showing unsuccessful processes resulting from spacer shift failures. When a sample is prepared on a section on the image generation chip, a dark marker is placed around the section. The spacer separates the sample in each section from other samples in adjacent sections. If the marker is not placed correctly, it may block a part of the image signal. As shown in Figure 5C, an offset error can occur across multiple adjacent sections. The upper part of the nine sections in this figure appears dark. The dark part at the top of the section increases as it moves from left to right. The space shift problem is an operational error because it is caused by the incorrect placement of the marker by a laboratory technician during the preparation of the sample on the image generation chip. Figure 5D presents three or more examples of failed images of sections resulting from spacer shift failures. Box 525 shows five sectional images with spacer shift failures when the upper part of the sectional image is dark with an increasing width from the upper right to the upper left. Box 527 shows two sectional images indicating a failed process due to a spacer shift problem at the bottom part of the section. Similarly, box 529 shows images of two sections that failed due to a space shift problem.

[0074] Figure 5E shows an example of a failed image of a section due to an unsuccessful process caused by an offset failure. In an offset failure, the image of the section on the image generation chip is shifted to one side. For example, in illustration 530, all the sectional images on the image generation chip are shifted towards the left side, and thus, the dark outer boundary line of the image generation chip on the left edge is cut off from the image. The offset failure can be caused by a scanning error such as misalignment or incorrect placement of the scanner of the image generation chip on the chip carrier.

[0075] Figure 5F shows an example of a failure classification image due to surface wear failure. Surface wear is caused by scratches on the surface of the classification of the image generation chip during the manufacturing process or during the preparation of the sample on the classification. The scratch is visible as a line on the image of the classification, as shown in Exemplification 535. Note that it is labeled as a failure due to surface wear failure even though the call rate value exceeds the passing threshold for the three classifications within the dashed box on the left side.

[0076] Figure 5G is Exemplification 540 of a failure classification image due to reagent flow failure. The 10 classification images within Box 542 are labeled as failure images due to reagent flow failure. The classification images failed due to an unsuccessful process caused by inappropriate reagent flow. During the genotyping process, the reagent is introduced into the image generation chip from one side. The reagent flows from one end of the image generation chip towards the opposite end, covering all the classifications completely. Sometimes, there are problems with the flow of the reagent and it may not propagate uniformly to all the classifications. In this case, when a sufficient amount of the reagent does not cover the classification, the reagent can dry out. Inappropriate reagent flow can cause the fluorescent dye not to be distributed uniformly across all the classifications and thus can reduce the intensity of the emission signal from some of the classifications, affecting the image quality. The failure images due to reagent flow failure may appear darker in color compared to the classification images representing the successful process cycle. Figure 5H shows a further example of a failure classification image due to reagent flow failure in Exemplification 545. Reagent flow failure can affect multiple adjacent classifications within the area of the image generation chip, as shown in Figures 5G and 5H.

[0077] Figure 5I presents an example of a failure image due to unknown reasons. The failure classification image is labeled as "unsound". Failure images in the unsound class of failures can be due to mixed or unidentified causes and weak signals. Exemplification 550 of the classified image also shows an example of a spacer failure for the classification in the upper left of the image generation chip. The image classification in the upper left position (row 1 and column 2) is labeled as a spacer failure. It can be seen that the upper part of the failure classification image is dark. The part of the upper dark area increases from the right corner to the left corner of the classified image. Principal Component Analysis-Based Feature Generation

[0078] Here, the inventors present an example of a unique image, called a unique face, in the field of face recognition. From tens of thousands of labeled images, linear criteria for 40 - 100 image components are identified. Figure 6A presents an example of 96 unique images obtained by applying principal component analysis (PCA). The 96 unique images are selected based on the ranking of the components according to the scale of variation described as presented above. Figure 6B shows the top 40 unique images ranked from the 96 unique images of Figure 6A. In one embodiment, it was observed that 40 components explained most of the variation. The additional components selected seemed to reflect noise or natural variation patterns in the sample processing.

[0079] Here, the inventors explain the dimensionality reduction and the creation of criteria for unique images using PCA. The first step is to reduce the resolution of the classified image and prepare a reduced image for input to PCA. Figure 7A presents an exemplification 710 showing dimensionality reduction. The classified image with a resolution of 3600×1600 pixels is downsampled by a factor of 1 / 20 per side, resulting in a classified image with a reduced size of 180×80 pixels. The downsampled classified image is flattened. The resulting flattened classified images are each one-dimensional arrays, i.e., 14,400×1 pixels.

[0080] As an alternative to the 1 / 20 size at each edge, other dimensionality reduction may be used. The principle is that in order to evaluate the overall soundness of the flow cell rather than calling individual clusters or balls within the flow cell, very little information and very low pixel density are required. Thus, 1 / 2 倍 ~1 / 50 倍 range or 1 / 4 倍 ~1 / 40 倍 reduction can be used, and as the initial resolution of the segmented image increases, more extreme resolution reduction is expected. In particular, when transfer learning can be applied to utilize the pre-training of the deep learning framework, it is desirable to select a resolution reduction that adapts the captured segments to the input aperture of the deep learning framework. A 1 / 20 downsampling to 180×80 pixels with reflections in both the horizontal and vertical directions 倍 has proven to be a good choice with an input aperture of 224×224 pixels. Other reductions are obvious for different segmented images and different deep learning input apertures.

[0081] In an alternative embodiment, PCA is applied to the downsampled image as described in the previous application. The flattened segmented image is normalized as described above, and thus results in a normalized, flattened, and rescaled segmented image shown in the example 740 of FIG. 7B, which is provided as input to PCA. Thus, PCA generates 14,400 principal components or eigenimages. Each input image is a vector in a 14,400-dimensional space. The inventors then rank the principal components or eigenimages using the variances described and create a criterion, for example, a criterion of 40 to 100 components. The components form a basis for the linear space. Image Segmentation-based Feature Generation

[0082] A second image processing technique for generating features from a segmented image involves thresholding of image areas or segments. FIG. 8A shows an example 810 of segmentation-based features applied to a segmented image. An illustration 812 of example 810 is a production image of a segment of an image generation chip. The inventors apply image conversion to convert this grayscale generated image 812 to generate a corresponding binary image 814. In one embodiment, some or all of the five steps presented above with reference to image converter 257 may be implemented to convert a grayscale image to a binary image. The black pixels of binary image 814 indicate defective or bad image pixels, while bright pixels indicate healthy or good image pixels.

[0083] In FIG. 8A, the right illustration 816 is an exemplary schematic diagram of a segment showing various areas of the segment and the boundary lines or lines around these areas. The area where the intensity of the fluorescence signal is recorded is also referred to as the active area or region of interest. For example, the schematic diagram 816 of the segment shows the active area of four slots extending parallel to each other from top to bottom. Areas of the segmented image that are not active areas are filtered out of the image. For example, the boundary areas of the slots separated from each other by vertical lines indicating the boundaries or boundary lines of the slots. Similarly, the boundary lines on the four sides of the segmented image can be filtered out. The segmentation technique can divide the segmented image into 4 to 20 or more segments or active areas. Thresholding determines what amount of the active area produces the desired signal intensity.

[0084] The number of active areas determines the number of features generated for each image. For example, if a segmented image is segmented into eight active areas, the image intensities from the eight active areas of the transformed image and the image intensity values from the same eight active areas of the original segmented image before transformation are provided as inputs to the classifier. Thus, in this example, a total of 16 features are provided to the classifier for each segmented image. The average intensity of the signal intensity from the active areas can be used as an input to the classifier. For example, if a segmented image is segmented into eight active areas, the average intensity of these eight active areas is calculated for both grayscale and binary images. These 16 intensity values are provided as inputs to the classifier to classify the segmented image as good or bad. Other segmentation schemes that divide into fewer or more segments, such as 4, 12, 17 or more segments per image, may be used. When provided as an input to a random forest classifier, a subset of the features is randomly selected for each decision tree. The decision trees vote on the image as sound (or successful), or unsound (or failed). The majority of the votes in the random forest are used to classify the image. In one embodiment, the value of the number of trees in the random forest classifier is in the range of 200 to 500, and the value of the depth of the model is in the range of 5 to 40. The area of the image generation chip and the pattern of failures between segments can be further evaluated for root cause classification.

[0085] FIG. 8B presents three pairs of examples 820 of segmented images 822, 824, and 826. The left image of each pair is a pre-transformed grayscale segmented image, and the right image of each pair is a processed binary image after applying image transformation as described with reference to FIG. 2. The first image pair 822 is a production image of a successful genotyping process. The second image pair 824 is a generated image of a failed production image due to hybridization (or hyb) failure. The third image pair 826 is a failed image due to a surface wear problem. One-versus-Rest (OvR) Classification

[0086] Figure 9 presents an illustrative example 900 of executing a one-vs.-rest classifier. The graph shows an example of executing a one-vs.-rest (OvR) classifier on a dataset consisting of samples belonging to three classes (squares, circles, and triangles), as shown in the left graph 931. As shown in the upper-right graph 915, the first hyperplane 916 indicates the hyperplane determination for the square class as ground truth. The hyperplane 916 separates the data points of the square class from the rest of the data points (circles and triangles). Similarly, graphs 936 and 955 separate the data points of the circle and triangle classes, respectively, from the other classes of data via hyperplanes 937 and 956. The position of the hyperplane is determined by the weight vector. The training algorithm attempts to maximize the margin of the hyperplane from the ground truth class for generalization, but that can result in the misclassification of one or more data points. The inventors apply OvR classification to distinguish segmented images from process cycles belonging to the good class from images belonging to multiple bad (or failed) classes. Random forest classifier

[0087] The disclosed technique can apply various classifiers to distinguish images from good or sound images from bad or unsound images belonging to multiple failed classes. Examples of classifiers that can be applied include random forest, k-nearest neighbor, multinomial logistic regression, and support vector machine. The inventors present an embodiment of the disclosed technique using, as an example, a random forest classifier.

[0088] A random forest classifier (also known as a random decision forest) is an ensemble machine learning technique. Ensemble techniques or algorithms combine more than one technique of the same or different types to classify an object. A random forest classifier consists of multiple decision trees that operate as an ensemble. Each individual decision tree in the random forest acts as a base classifier and outputs a class prediction. The class with the most votes becomes the prediction of the random forest model. The basic concept behind random forests is that a number of relatively uncorrelated models (decision trees) operating as a committee will tend to outperform any of the individual constituent models.

[0089] The disclosed technique applies a random forest classifier in a two-stage classification process. A first trained random forest classifier performs the task of separating successful production images from unsuccessful production images. A second trained random forest classifier performs the task of root cause analysis of unsuccessful production images by predicting the failure class of the unsuccessful images. This two-stage classification was selected due to the prevalence of successful production runs, but a one-stage classification may also be used. Another reason for choosing the two-stage approach is to allow the inventors to control the sensitivity threshold for classifying images as healthy or successful production images versus unhealthy or failed production images. The inventors can increase the threshold in the first stage of classification and thus cause the classifier to classify more production images as failed images. These failed images are then processed by a second-stage classifier for root cause analysis by identifying the failure class. Training of the Random Forest Classifier

[0090] Figure 10A illustrates the training of two random forest classifiers as shown in Illustration 1000. The training data consists of input features for the labeled process cycle images stored in the training database 138 as shown in Figure 1. In one exemplary training of the classifier, the inventors used 96 weights of the components of the labeled production images. A random forest classifier having 200 decision trees and a depth of 20 functioned well. It is understood that random forest classifiers having a range of 200 to 500 decision trees and a range of depths of 10 to 40 are expected to provide good results for this embodiment. The inventors adjusted the hyperparameters using randomized search cross-validation. The search range for the depth was 5 to 150, and the search range for the number of trees was 100 to 500. Increasing the number of trees can improve the performance of the model but also increase the time required for training. The training database 1001 containing features for 20,000 production cycle images is used to train a binary classifier labeled as the good-versus-bad classifier 151. The same training database can be used to train the root cause classifier 171 for predicting the failure class. The root cause classifier 171 is trained on a training database 1021 consisting of only the bad or failed production images as shown in Figure 10A.

[0091] In one embodiment, the inventors used 96 weights of the components of the labeled production images to train the random forest classifier. A random forest classifier having 200 decision trees and a depth of 20 functioned well. Random forest classifiers having a range of 200 to 500 decision trees and a range of depths of 10 to 40 are expected to provide good results for this embodiment. The inventors adjusted the hyperparameters using randomized search cross-validation. The search range for the depth was 5 to 150, and the search range for the number of trees was 100 to 500. Increasing the number of trees can improve the performance of the model but also increase the time required for training. The training database 1001 containing features for 20,000 production cycle images is used to train a binary classifier labeled as the good-versus-bad classifier 151. The same training database can be used to train the root cause classifier 171 for predicting the failure class. The root cause classifier 171 is trained on a training database 1021 consisting of only the bad or failed production images as shown in Figure 10A.

[0092] Decision trees are prone to overfitting. To overcome this problem, bagging techniques are used to train decision trees in random forests. Bagging is a combination of the bootstrap and aggregation techniques. In the bootstrap, during training, we obtain row samples from our training database and use them to train each decision tree in the random forest. For example, a subset of the features of the selected rows can be used to train decision tree 1. Therefore, the training data for decision tree 1 can be called row sample 1 with column sample 1, or RS1+CS1. Columns or features can be randomly selected. Decision tree 2 and subsequent decision trees in the random forest are trained in a similar manner by using subsets of the training data. Note that the training data for the decision trees is generated by replacement, i.e., the same row data can be used in the training of multiple decision trees.

[0093] The second part of the bagging technique is the aggregation part applied during production. Each decision tree outputs a classification for each class. In the case of binary classification, it can be 1 or 0. The output of the random forest is the aggregation of the outputs of the decision trees in the random forest with the majority vote selected as the output of the random forest. By using the votes from multiple decision trees, the random forest reduces the high variance in the results of the decision trees and thus results in good prediction results. By using row and column sampling to train individual decision trees, each decision tree becomes an expert regarding the training records with the selected features.

[0094] During training, the output of the random forest is compared to the ground truth label, and the prediction error is calculated. During backpropagation, the weights of the 96 components (or eigen-images) are adjusted so that the prediction error is reduced. The number of components or eigen-images depends on the number of components selected from the output of principal component analysis (PCA) using the described measure of variance. During binary classification, the good-versus-bad classifier uses image description features from the training data and applies one-versus-rest (OvR) classification of the good class (or images with healthy labels) versus multiple bad classes (images labeled with one of six failure classes). The parameters of the trained random forest classifier (such as the weights of the components) are stored for use in the good-versus-bad classification of production cycle images during inference.

[0095] Training of the root cause classifier 171 is performed in a similar manner. The training database 1021 consists of features from labeled process cycle images from bad process cycles belonging to multiple failure classes. The random forest classifier 171 is trained using image description features for one-versus-rest (OvR) classification of each failure class versus the rest of the labeled training examples. Classification Using a Random Forest Classifier

[0096] Here, the inventors describe the classification of production images using the trained classifiers 151 and 171. FIG. 10B presents a two-stage classification 1080 of production images using a good-versus-bad classifier 151 in the first stage and a root-cause classifier 171 in the second stage. The process is presented using a series of process flow steps labeled 1 through 9. The process begins, in step 1, by accessing a trained random forest classifier labeled as the good-versus-bad classifier 151. The input features of the production images stored in the database 1030 are provided as input to the classifier 151. The classifier distinguishes good images belonging to successful process cycles from bad images belonging to failed process cycles. Bad images belong to, for example, multiple failure classes, and each image can belong to one of the six failure classes described above. The trained classifier accesses the criteria of the eigen-images used to analyze the production images. The trained classifier creates image description features for the production images based on a linear combination of the eigen-images. The weights of the eigen-images are learned during the training of the classifier as described above.

[0097] When the inventors apply the one-vs.-the-rest classification, all the decision trees within the random forest classifier predict the output for each class, i.e., whether the image belongs to one of the seven classes (one good class and six failure classes). Therefore, each decision tree within the random forest outputs seven probability values, i.e., one value for each class. The results from the decision trees are aggregated, and the majority vote is used to predict whether the image is good or bad. For example, if more than 50% of the decision trees within the random forest classify the image as good, the image is classified as a good image belonging to the successful production cycle. The sensitivity of the classifier can be adjusted, e.g., by setting a higher threshold, which will result in more images being classified as bad. In process step 2, the output 151 from the classifier is checked. If the image is classified as a good image (step 3), the process ends (step 4). Otherwise, if the image is classified as a bad image indicating a failed process cycle (step 5), the system calls the root cause classifier 171 (step 6).

[0098] The root cause classifier is applied in the second stage of the two-stage process to determine the failure class of the bad image. The process continues in the second stage by accessing the production image input features for the bad image (step 7) and providing the input features to the trained root cause classifier 171 (step 8). Each decision tree in the root cause classifier 171 votes on the input image features by applying the one-vs.-the-rest classification. In this case, the classification determines whether the image belongs to one of the six failure classes versus the remaining five failure classes. Each decision tree provides a classification for each class. The majority vote from the decision trees determines the failure class of the image (step 9).

[0099] The inventors can use other classifiers to classify good-segmented images and bad-segmented images and perform root cause analysis. For example, the disclosed technology can apply the k-nearest neighbor (k-NN or KNN) algorithm to classify the segmented images. The k-NN algorithm assumes that similar examples (or segmented images in the inventors' embodiments) exist in proximity. The k-NN algorithm incorporates the idea of similarity (also referred to as proximity or closeness) by calculating the distance between data points or images. For this purpose, the straight-line distance (or Euclidean distance) is commonly used. In k-NN classification, the output is class membership, e.g., a good-image class or a bad-image class. An image is classified by a plurality of votes from its neighbors, and an object is assigned to the most common class among its k nearest neighbors. The value of k is a positive integer.

[0100] To select the right value of k for the inventors' data, the inventors run the k-NN algorithm several times with different values of k and select the value of k that reduces the number of errors the inventors encounter while maintaining the algorithm's ability to make accurate predictions when given data not previously seen. The inventors assume that the value of k is set to 1. This can result in inaccurate predictions. The inventors consider two clusters of data points: good images and bad images. Suppose the inventors have a query example surrounded by many good-image data points, but it is close to one bad-image data point that is also within the cluster of good-image data points. According to k = 1, k-NN inaccurately predicts that the query example is a bad image. As the inventors increase the value of k, the predictions of the k-NN algorithm become more stable due to majority voting (in classification) and averaging (in regression). Thus, the algorithm is more likely to make more accurate predictions up to a certain value of k. As the value of k increases, the inventors begin to observe an increase in the number of errors. A value of k in the range of 6 to 50 is expected to work.

[0101] Examples of other classifiers that can be trained and applied by the disclosed technology include multinomial logistic regression, support vector machine (SVM), gradient boosting tree, naive Bayes, and the like. The inventors evaluated the performance of the classifiers using three criteria: training time, accuracy, and interpretability of the results. The random forest classifier performed better than other classifiers. The inventors briefly present other classifiers in the following text.

[0102] The support vector machine classifier also performed equally well as the random forest classifier. The SVM classifier locates a hyperplane between the feature vectors for the good class and the feature vectors for multiple bad classes. The disclosed technology may include training multinomial logistic regression. The multinomial regression model can be trained to predict the probabilities of different possible outcomes (multiclass classification). The model is used when the output is categorical. Therefore, the model can be trained to predict whether an image belongs to the good class or one of the multiple bad classes. The performance of the logistic regression classifier was lower than that of the random forest and SVM classifiers. The disclosed technology may include training a gradient boosting model, which is an ensemble of prediction models such as decision trees. The model attempts to optimize the cost function over the function space by repeatedly selecting a function that points in the direction of the negative gradient. For example, the model can be trained to minimize the mean squared error over the training dataset. The gradient boosting model required more training time compared to other classifiers. The disclosed technology may include training a naive Bayes classifier that assumes that the value of a particular feature is independent of the values of any other features. The naive Bayes classifier considers each feature that independently contributes to the probability of an example belonging to a class. The naive Bayes classifier can be trained to classify images with the good class against multiple bad classes. Examples of Feature Engineering for Deep Learning-Based Classification

[0103] The present inventors present examples of feature engineering of images for input to a convolutional neural network (CNN). Using feature engineering techniques shown in FIG. 11A, such as cropping, zero-padding, and reflection operations, analysis frames 1105, 1107, and 1109 having a size of M×N pixels are generated. The analysis frames are sized to match the input image size used by a particular convolutional neural network. In one example, the size of the analysis frame is 224×224 pixels, which matches the size of the input image required by the ResNet-18 and VGG-16 models. The dimensions used in this example are 3600×1600 pixels for the segmented image. This example shows how a 3600×1600 pixel segmented image is adapted to a square frame of size 224×224 pixels for input to a convolutional neural network (CNN). The first step of this process is to downsample the 3600×1600 pixel segmented image by 1 / 20 on each side to obtain a reduced-size image of 180×80 pixels. The second step is to place the 180×80 pixel image into a 224×224 pixel frame. Next, the placed image is reflected horizontally to fill the left and right 224×224 pixel square frames of the placed 180×80 pixel image. Next, the 180×80 pixel image is reflected vertically to fill the top and bottom of the 224×224 square frame.

[0104] Figure 11A includes an illustration 1100 showing cropping, zero-padding, and reflection operations on a segmented image. An exemplary segmented image 1103 is shown. The segmented image 1103 is rectangular. In cropping, the central portion or any other portion of the segmented image is cropped. Figure 11A illustrates a portion cropped from the segmented image 1103, which is placed in the analysis frame 1105. The cropped portion of the segmented image can be of square dimension, i.e., the cropped portion has the same number of pixels along the height and width of the cropped portion. For example, the cropped image placed within the analysis frame 1105 can be an 80×80 pixel cropped portion of the J×K image 1103. The cropped portion can be positioned at the center of the analysis frame. In other embodiments, the cropped portion may be positioned near one side of the analysis frame. In another embodiment, the cropped portion may be resampled to make the size of the cropped segment equal to the size of the analysis frame, e.g., 224×224 pixels. This will result in the cropped segment completely filling the analysis frame.

[0105] Figure 11A shows the zero-padding operation after the segmented image 1103 is placed in the analysis frame 1107. The segmented image can be positioned at the center of the analysis frame, as shown in illustration 1107. The segmented image may be positioned at any other location within the analysis frame. The surrounding pixels within the analysis frame can be given a "0" image intensity value.

[0106] FIG. 11A also shows applying reflection to fill pixels within analysis frame 1109. The segmented image is positioned at the center of analysis frame 1107 and then reflected horizontally to fill the analysis frame along the left and right edges of the analysis frame. It can also be reflected vertically along the top and bottom edges that fill the top and bottom of analysis frame 1109. Evaluation of various strategies detailed below has shown that reflection for filling the frame functions well in both training and production without requiring any changes in the input aperture of the image processing framework being evaluated. Surprisingly, reflection was significantly more performant than zero-padding.

[0107] FIG. 11B illustrates the augmentation of training data by applying translation to the training example segmented image 1103. Applying translation for data augmentation involves placing the segmented image at multiple positions within the analysis frame. For example, as shown in analysis frame 1111, the segmented image is positioned at the center of the analysis frame. Horizontal and vertical translations of the segmented image can generate multiple analysis frames as shown in illustrations 1113, 1115, 1117, etc. As shown in FIG. 11B, the surrounding area can first be zero-padded and depicted by a dark color (or black). However, the surrounding area does not necessarily have to be black. The disclosed technique can apply horizontal and vertical reflection padding to fill the surrounding area with the reflected segmented image, as shown in FIG. 11A. Transfer learning using fine-tuning

[0108] In many real-world applications, the entire convolutional neural network (CNN) is not trained from scratch using random initialization. This is mostly because the training dataset is small. It is common to pre-train a CNN on a large dataset such as ImageNet, which contains approximately 14 million images with 1000 categories (available at <<image-net.org>>), and then use the pre-trained CNN as an initialization or fixed feature extractor for the target task. This process is known as transfer learning for transferring the knowledge learned from the source dataset to the target dataset. The commonly used transfer learning technique is called fine-tuning. Figure 12A reproduced from <<d2l.ai / chapter_computer-vision / fine-tuning.html>> illustrates fine-tuning (1200). There are four steps to fine-tune a deep learning model.

[0109] Step 1: Pre-train a neural network model, i.e., the source model, on the source dataset (e.g., the ImageNet dataset).

[0110] Step 2: The second step is to create a new neural network model, i.e., the target model. This replicates all model designs and their parameters on the source model except for the output layer. The inventors assume that these model parameters contain the knowledge learned from the source dataset and that this knowledge is equally applicable to the target dataset. The inventors also assume that the output layer of the source model is closely related to the labels of the source dataset and is therefore not used in the target model.

[0111] Step 3: The third step is to add an output layer to the target model whose output size is the number of target dataset categories and randomly initialize the model parameters of this layer. Thus, in the case of a detection task, the output layer of the inventors' model can have two categories (normal and failed). In the case of a classification task, the output layer of the inventors' model can have five categories corresponding to five defect categories. The model can have additional categories to classify images with unknown failure types that are not classified into existing known failure categories.

[0112] Step 4: The fourth step is to train the target model on the target dataset. In this case, the training dataset includes approximately 75,000 labeled segmented images. The inventors train the output layer from scratch while the parameters of all the remaining layers are fine-tuned based on the parameters of the source model. Network architecture of the VGG-16 model

[0113] The VGG architecture (Simonyan et al., 2015, available at <<arxiv.org / abs / 1409.1556>>) has been widely used in computer vision in recent years. It includes stacked convolutional layers and max-pooling layers. The inventors used a smaller and thus faster 16-layer architecture known as VGG-16. The architecture (1210) is presented in FIG. 12B. This model consists of five convolutional layers, conv1(1220), conv2(1222), conv3(1224), conv4(1226), and conv5(1228). One way to analyze the output of the convolution is through a fully connected (FC) network. Thus, the output of the convolutional layer is fed to the fully connected (FC) layer. There are three fully connected layers, fc6(1240), fc7(1242), and fc8(1244). This model uses "max-pooling" between the convolutional layers and between the last convolutional layer and the fully connected layer. There are five "max-pooling" layers 1230, 1232, 1234, 1236, and 1238. The inventors extract features from the "conv5" layer (1228) with a stride of 16 pixels.

[0114] The model takes as input an image size of 224×224 pixels. The input image may be an RGB image. The image is passed through a stack of convolutional (conv) layers, and the filters are used with a very small receptive field: 3×3 (to capture the concepts of left / right, up / down, and center). In one configuration, 1×1 convolutional filters are utilized, which can be regarded as a linear transformation of the input channels (followed by non-linearity later). The convolutional stride is fixed at 1 pixel, that is, the spatial padding of the convolutional layer input is such that the spatial resolution is preserved after convolution, that is, the padding is 1 pixel for a 3×3 convolutional layer.

[0115] Spatial pooling is performed by five "max pooling" layers 1230, 1232, 1234, 1236, and 1238 as shown in FIG. 12B, following some of the convolutional layers. The output of the convolution is also referred to as a feature map. This output is provided as the input to the max pooling layers. The goal of the pooling layer is to reduce the dimensions of the feature map. For this reason, it is also called downsampling. The factor by which downsampling is performed is called the "stride" or "downsampling factor". The pooling stride is denoted by "s". In one type of pooling called "max pooling", the maximum value is selected for each stride. For example, consider that max pooling with s = 2 is applied to the 12-dimensional vector x = [1, 10, 8, 2, 3, 6, 7, 0, 5, 4, 9, 2]. Max pooling vector x with stride s = 2 means that the inventors select the maximum value for every two values starting from index 0, resulting in the vector [10, 8, 6, 7, 5, 9]. Thus, max pooling vector x with stride s = 2 results in a 6-dimensional vector. In the example of the VGG-16 architecture, max pooling is performed with a stride of 2 over a 2×2 pixel window.

[0116] Figure 12B illustrates that the image size is reduced after passing through each of the five convolutional layers and their respective max-pooling layers in the VGG model. The input image has dimensions of 224 x 224 pixels. After passing through the first convolutional layer labeled conv1(1220) and the max-pooling layer (1230), the image size is reduced to 112 x 112 pixels. After passing through the second convolutional layer labeled conv2(1222) and the max-pooling layer (1232), the image size is reduced to 56 x 56 pixels. The third convolutional layer labeled conv3(1224) and the max-pooling layer (1234) reduce the image size to 28 x 28 pixels. The fourth convolutional layer labeled conv4(1226) and the max-pooling layer (1236) reduce the image size to 14 x 14 pixels. Finally, the fifth convolutional layer labeled conv5(1228) and the max-pooling layer (1238) reduce the image size to 7 x 7 pixels.

[0117] Three fully-connected (FC) layers 1240, 1242, and 1244 follow the stack of convolutional layers. The first two FC layers 1240 and 1242 each have 4096 channels, and the third FC layer 1244 performs 1000-way classification and thus includes 1000 channels (one for each class). The last layer is a softmax layer. The depth of the convolutional layers can vary in different architectures of the VGG model. The configuration of the fully-connected layers may be the same in different architectures of the VGG model.

[0118] Figure 12C illustrates the VGG-16 architecture 1270 applied by the technique disclosed for the detection task. The model architecture illustrates the parameter values of five convolutional layers conv1 to conv5 and three fully-connected layers fc6 to fc8. The output from the final FC layer 1244 generates two outputs that classify the segmented image as normal or failed. The labels in Figure 12C correspond to the respective elements of the architecture diagram in Figure 12B. NETWORK ARCHITECTURE OF THE RESNET-18 MODEL

[0119] The ResNet architecture (He et al., CVPR 2016, available at <<arxiv.org / abs / 1512.03385>>) was designed to avoid problems related to ultra-deep neural networks. Most notably, the use of residual connections helps overcome the vanishing gradient problem. The inventors used the ResNet-18 architecture with 18 trainable layers. FIGS. 12D and 12E illustrate an exemplary ResNet-18 architecture 1280. The architecture of ResNet-18 is organized into four layers: 1260, 1262 shown in FIG. 12D, and 1264, 1268 shown in FIG. 12E. Each layer comprises two blocks labeled "0" and "1". Each block comprises two convolutional layers labeled "conv1" and "conv2". Prior to layer 1 block (1260), there is one convolutional layer 1259. Thus, there are a total of 17 convolutional layers separated by batch normalization and ReLU. One end fully connected (FC) layer (1270) generates two outputs. This architecture is used for a detection task where the network classifies segmented images into normal (or good) images and failed (or bad) images.

[0120] When weights are updated, the distribution of the input to the deeper layers of the network can change after each mini-batch, making it difficult to train deep learning models with dozens of convolutional layers. This reduces the convergence speed of the model. The batch normalization (Ioffe and Szegedy 2015, available at <<arxiv.org / abs / 1502.03167>>) technique can overcome this problem. Batch normalization normalizes the input to the layer for each mini-batch and reduces the number of epochs required to train a deep learning model. Performance Comparison - Feature Engineering and Deep Learning Models

[0121] The inventors compared the performance of two models, ResNet-18 and VGG-16, using the augmented input data generated by applying the feature engineering techniques presented above. Illustration 1300 in FIG. 13 presents the results of the performance comparison using the macro F1 score. The macro F1 performance score can be considered to reflect a combination of precision and recall values. For example, the F1 score can combine the precision (p) and recall (r) values with equal weights, as shown in the following equation (1). The macro F1 score can be calculated as the average of the F1 scores for all failure categories.

Number

[0122] Referring back to the results in FIG. 13, the top two bars 1305 and 1310 present the scores of the VGG-16 and ResNet-18 models, respectively. Both models generated the same macro F1 (or F1) performance score of 96.2 when trained using a fine-tuning approach for transfer learning. Both the VGG-16 and ResNet-18 models were trained on the full-segment images of the image generation chip. Reflection padding was used to fill the larger-sized analysis frames as described above. The training data was augmented by creating copies of the images. The system can augment the training data by creating multiple copies of the labeled images within one class (such as one failure class) so that when the images belonging to this class are fewer compared to the images belonging to one or more other classes, the balance of the dataset is achieved. Alternatively, the image copies can be used multiple times during training to balance the training data.

[0123] The graph of FIG. 13 also illustrates the performance of the ResNet-18 model using different feature engineering techniques in the 3rd to 5th bars of the performance comparison graph. The 3rd bar 1315 illustrates that the ResNet-18 model achieves a score of 77.2 when trained using only the cropping technique for feature engineering. When using only cropping for training data generation, since there are many defects in the central part of the segmented image, the results are not good. Therefore, such defects may be missed by the model during training. The 4th bar 1320 shows that the performance score is 92.3 when the ResNet-18 model is trained using training data generated using 0-padding used to fill the area within the analysis frame surrounding the segmented image. 0-padding does not perform well due to the unnatural (dark) areas within the analysis frame surrounding the position of the segmented image. The 5th bar labeled 1325 from the top shows that the performance score is 94.9 when the ResNet-18 model is trained using training data augmented by creating copies of the labeled images. In this case, translation is not used to create additional training data deformations.

[0124] The bar 1335 at the bottom of graph 1300 presents the performance of the basic model. The basic model is a ResNet-18 CNN used as a feature extractor without fine-tuning. Feature engineering techniques such as reflection padding are not used. Data augmentation is also not used. The score of the basic model is 74.3.

[0125] FIG. 14 presents the breakdown of the performance scores by the failure categories of the disclosed best-performing model. The best model achieved an overall macro F1 score of 97% and an accuracy of 96%. Table 1401 in FIG. 14 presents the F1 scores for different failure categories, examples of which were presented above in FIGS. 5A - 5I.

[0126] The confusion matrix 1420 illustrates the performance of the model with respect to the predicted labels versus the true labels for five failure categories. The majority of the defect categories are correctly predicted as indicated by the values on the diagonal (labeled 1425). The vertical bars on the right indicate the number of samples with different failure categories. The number of samples with a particular type of failure can be calculated by summing the numbers in the row labeled (1430) for that failure type. For example, the number of samples including offset and spacer type failures are 38 and 26 respectively. The correct prediction of the failure category can be determined by looking at the diagonal values. For example, the model correctly predicted 36 samples with offset failures and 24 samples with space shift failures. Similarly, out of 129 samples with hybridization or "hyb" failures, 123 were correctly predicted. This model correctly predicted 163 out of a total of 170 samples with reagent flow failures. The largest number of samples included surface wear failures. Out of a total of 437 samples with wear failures, a total of 428 samples were correctly predicted. Misclassified samples

[0127] Figure 15 presents an analysis of six images from the 26 misclassifications in the confusion matrix presented in Figure 14. For each segmented image, the manually annotated failure labels are listed in the upper row and the predicted labels from the model are listed in the lower row. There are images misclassified by the deep learning model as well as images misclassified by the human who labeled the training set.

[0128] The four sample images 1503, 1505, 1507, and 1509 from the left have multiple defects belonging to different failure categories. The manual annotation represents only one of the multiple defects. For example, the first image on the left (labeled 1503) is labeled by a human annotator as having surface wear. The surface wear is present in the upper left part of the image as shown within the bounding box. The model predicted that the image has hybridization or hyb failure. It can be seen that the image has hyb failures at two positions, in the upper right part of the image and near the bottom of the image, as indicated by the arrows.

[0129] The last two images 1511 and 1513 were mislabeled by a human annotator, and the deep learning model predicted these images correctly. For example, the fifth image from the left (labeled 1511) is labeled as having spacer shift failure. However, the model predicted that the image has surface wear failure. The human annotator may have misidentified the failure category due to the dark part of the image being located close to the bottom edge of the image. Similarly, the sixth image from the left (labeled 1513) was labeled by a human annotator as having hybridization or hyb failure. The model predicted the failure category as reagent flow failure, which is the correct failure category of the image. Therefore, the performance of machine learning is even better than that indicated by the F1 score. Performance improvement by using deep learning

[0130] The inventors compared the performance of a deep learning-based approach (deepIBEX) with a previous solution using a random forest model (IBEX) for the same split of the dataset. The results show that deepIBEX functions better than IBEX measured by the macro F1 score and accuracy in both tasks of anomaly detection (separating good images from bad images) and classification (identifying the root cause of bad images). Methodology

[0131] The dataset of the discriminative images is divided into a training set, a validation set, and a test set, and the ratios of the samples are 70%, 15%, and 15%. For the two models (deepIBEX and IBEX) being compared here, the inventors used the same split of the dataset for parameter adjustment, training, and evaluation. The inventors adjusted the model hyperparameters for the training set and the validation set using random grid search. The inventors finally evaluated the model performance on the test set. Parameters tested

[0132] Ten sets of hyperparameters were examined for each model through random grid search. In IBEX, the main hyperparameters adjusted by the inventors were the final dimension of PCA and the depth of the trees within the random forest. In deepIBEX, the inventors adjusted the learning rate, batch size, and momentum. Macro F1 score and accuracy metric

[0133] The deepIBEX model functioned better than the IBEX model as shown by the macro F1 score and accuracy. The F1 score for each category can be defined as the harmonic mean of the recall and precision of that category. The F1 score can also be calculated using equal weights for precision and recall as shown in Equation (1) above. The macro F1 score can be calculated as the average of the F1 scores of all failure categories. The macro F1 score and accuracy measuring the performance of the model on the test set are presented in the following table (Table 1). A high macro F1 score, or high accuracy, means good performance of the model on the test data. Accuracy is defined as the proportion of correctly labeled samples among all test samples. For both detection and classification, deepIBEX outperformed IBEX.

Table 1

[0134] Figure 16 presents flowchart 1601 which presents the process steps for training a good classifier versus a bad classifier and applying the trained classifier to classify production images of sections of an image generation chip.

[0135] The processing steps presented in flowchart 1601 can be implemented using a computer program stored in a memory executable by a computer system accessible to a processor, by dedicated logic hardware including a field programmable integrated circuit, and by a combination of dedicated logic hardware and a computer program, using a processor programmed using the computer program. As with all flowcharts herein, it will be understood that many of the steps can be combined, performed in parallel, or performed in a different order without affecting the functions achieved. Further, it will be understood that the flowcharts herein show only the steps relevant to an understanding of the technology, and that numerous additional steps for achieving other functions can be performed before, after, and between the steps shown.

[0136] The process starts at step 1602. The training data creation process step 1610 may include a plurality of operations (1612, 1614, and 1616) that may be performed to create training data including labeled images of segments of an image generation chip. Training data creation may include generating J×K labeled images (step 1612). This step may include cutting portions from a larger image to generate the J×K labeled images. The J×K labeled images may be positioned at a plurality of locations within an M×N analysis frame (step 1614). The M×N analysis frame is sized to match the input image size required by a convolutional neural network (CNN). Since the J×K images may be smaller in size than the M×N analysis frame, the M×N analysis frame may not be completely filled. Further, the M×N analysis frame may be square and the J×K images may be rectangular. In step 1616, a portion of the J×K labeled image positioned within the M×N analysis frame may be used to fill around the edges of the J×K labeled image. Horizontal reflection may be used to fill the M×N analysis frame along the left and right edges of the analysis frame. Vertical reflection may be used to fill the M×N analysis frame along the top and bottom edges of the M×N analysis frame. The steps presented above may be repeated to generate multiple training examples by varying the position of the same J×K labeled image within the M×N analysis frame.

[0137] A convolutional neural network (CNN) can be trained in step 1620 using the training data generated by executing the process steps presented above. The system can train a pre-trained CNN such as the VGG-16 or ResNet-18 model. Next, the trained CNN model can be applied to the production images of the segment to classify the images as good or bad (step 1624). Feature engineering techniques such as reflection padding can be applied to the production images of the segment of the image generation chip to fill the M×N input frame into the CNN model. Images classified as good or normal by the classifier can indicate a successfully completed process. Images classified as bad or failed can indicate a process failure (step 1626). The process can continue to step 1628 to further classify the bad process cycle images to determine the root cause of the failure. If the production image is classified as good, the process can end at step 1630. Specific embodiments

[0138] The disclosed technology applies image classification for the evaluation and root cause analysis of the genotyping process. Two tasks, the separation of successful and unsuccessful production images and then the root cause analysis of the unsuccessful images, are performed by the classifier. Training and inference of good classifier vs. bad classifier

[0139] The inventors first present the classification of successful and failed production images. In one embodiment of the disclosed technology, a method for training a convolutional neural network (CNN) to identify and classify images of segments of an image generation chip from defective or failed or unsuccessful processes that result in process cycle failures is described. The method includes using a pre-trained convolutional neural network (CNN) to extract image features. The pre-trained CNN can accept an image of dimension M×N. Examples of image dimensions for the input to the CNN can include 224×224 pixels, 227×227 pixels, 299×299 pixels. Alternatively, the size of the input image to the CNN can be within a range of 200 to 300 pixels on a side, or within a range of 75 to 550 pixels on a side. The input image can be square or rectangular. The method includes creating a training data set using labeled images of dimension J×K that are smaller than M×N, normal, and depict process failures. Exemplary dimensions of the J×K sized labeled images can be 180×80 pixels, 224×100 pixels, 200×90 pixels, 120×120 pixels, 224×224 pixels, 504×224 pixels, etc. The method includes creating a training data set using labeled images of dimension J×K, where J×K is smaller than M×N, normal, and depicts process failures. The images are from segments of the image generation chip. The method includes positioning the J×K labeled images at multiple positions within an M×N (224×224) frame. The method includes filling around the edges of a particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N frame.

[0140] The method further includes training the pre-trained CNN to generate a segment classifier using the training data set. The method includes storing the coefficients of the trained classifier to identify and classify images of segments of the image generation chip from a production process cycle. The trained classifier can accept an image of a segment of the image generation chip and classify the image as either normal or depicting a process failure.

[0141] In a production implementation, the trained CNN can be applied to identify defective process cycle images of sections of the image generation chip. Here, the inventors present a method for identifying defective process cycle images of sections of the image generation chip that cause process cycle failures. The method includes creating an input to a trained classifier.

[0142] The input can fill the M×N input aperture of the image processing framework or can have a small dimension J×K and be reflected to fill the M×N analysis frame. Taking the latter approach, creating the input includes accessing the image of the section having dimension J×K. The method includes positioning the J×K image within the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned within the M×N analysis frame. Depending on the relative sizes of J×K to M×N, some zero-padding can be applied, for example, to fill narrow strips along the top and bottom of the analysis frame, but it has been found that reflection functions better. The method includes inputting the M×N analysis frame to a trained classifier. The method includes using the trained classifier to classify the image of the section of the image generation chip as normal or as depicting a process failure. The method includes outputting the classification result of the section of the image generation chip.

[0143] Embodiments of the method and other methods disclosed optionally include one or more of the following features. The method can also include features described in relation to the method presented above. For the sake of brevity, alternative combinations of method features are not individually listed. Features applicable to methods, systems, and articles of manufacture are not repeated for each legal classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other legal classifications.

[0144] In one embodiment, the method further includes positioning the J×K image at the center of the M×N analysis frame.

[0145] In one embodiment, the method includes applying a horizontal reflection to at least a portion of a particular J×K labeled image to fill around the edges of the particular J×K labeled image within the M×N analysis frame.

[0146] In one embodiment, the method includes applying a vertical reflection to at least a portion of a particular J×K labeled image to fill around the edges of the particular J×K labeled image within the M×N analysis frame.

[0147] In one embodiment, a J×K labeled image is generated by cropping a portion from a large image, and the cropped J×K portion is placed in the M×N frame. Examples of large image sizes include images with dimensions of 504×224 pixels or even larger images.

[0148] The labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image can be reduced to 1 / 2 to 1 / 50 times per side of the original pixel resolution. When reduced to 1 / 25 times per side, the number of pixels is reduced to 1 / 625 of the original number of pixels. In one embodiment, the segmented high-resolution image obtained from a scanner or genotyping device has a size of 3600×1600 pixels.

[0149] The above computer-implemented method can be executed in a system including computer hardware. A computer-implemented system can execute one or more of the above methods. A computer-implemented system can incorporate any of the features of the methods described immediately before or throughout this application, as applied to the methods implemented by the system. For the sake of brevity, alternative combinations of system features are not individually enumerated. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory grouping of basic features. The reader will understand how the features identified in this section can be readily combined with the basic features in other statutory groupings.

[0150] Rather than a method, program instructions executable by a processor can be stored on a non-transitory computer readable medium (CRM). When the program instructions are executed, one or more of the above computer-implemented methods are implemented. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may form one or more components of a computer-implemented system that implements the disclosed methods.

[0151] Each of the features considered in this particular embodiment section for method embodiments is equally applicable to CRM and system embodiments. As noted above, all method features are not repeated here and should be considered repeated by reference. Training and Inference Root Cause Analysis

[0152] In one embodiment of the disclosed technology, a method for training a convolutional neural network (CNN) to classify images of sections of an image generation chip by the root cause of process failure is described. The method includes using a pre-trained CNN to extract image features. The pre-trained CNN can accept an image of dimension M×N. Examples of image dimensions include 224×224 pixels. The method includes creating a training dataset using labeled images of dimension J×K belonging to at least one of a plurality of failure categories that result in process failure. Exemplary dimensions of the J×K sized labeled images are 180×80 pixels, 200×90 pixels, etc. The images are from sections of the image generation chip. The method includes positioning the J×K labeled images at multiple positions within the M×N (224×224) frame. The method includes using at least a portion of a particular J×K labeled image to fill around the edges of the particular J×K labeled image, thereby filling the M×N frame. The method includes further training the pre-trained CNN to generate a section classifier using the training dataset. The method includes storing the coefficients of the trained classifier to identify and classify images of sections of the image generation chip from the production process cycle. The trained classifier can accept an image of a section of the image generation chip and classify the image by the root cause of process failure from among a plurality of failure categories.

[0153] In a production embodiment, the trained CNN can be applied to classify defective process cycle images of sections. The inventors herein present a method for identifying and classifying defective process cycle images of sections of an image generation chip that cause failure of the process cycle. The method includes creating an input to the trained classifier.

[0154] The input can fill the M×N input aperture of the image processing framework or can be reflected to have a small dimension J×K and fill the M×N analysis frame. Taking the latter approach, creating the input involves accessing a segmented image having dimension J×K. The method includes positioning the J×K image within the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned within the M×N analysis frame. Depending on the relative sizes of J×K to M×N, some zero-padding may be applied, for example, to fill narrow strips along the top and bottom of the analysis frame, but reflection has been found to work better. The method includes inputting the M×N analysis frame to a trained classifier. The method includes using the trained classifier to classify the segmented image of the image generation chip by the root cause of the process failure from among a plurality of failure categories. The method includes outputting the classification result of the segmentation of the image generation chip.

[0155] Embodiments of the method and other methods disclosed may optionally include one or more of the following features. The method may also include features described in connection with the methods presented above. For the sake of brevity, alternative combinations of method features are not individually listed. Features applicable to methods, systems, and articles of manufacture are not repeated for each legal classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other legal classifications.

[0156] In one embodiment, the method further includes positioning the J×K image at the center of the M×N analysis frame.

[0157] In one embodiment, the method includes applying a horizontal reflection to at least a portion of a specific J×K labeled image to fill around the edges of the specific J×K labeled image within the M×N analysis frame.

[0158] In one embodiment, the method includes applying a vertical reflection to at least a portion of a particular J×K labeled image to fill around the edges of the particular J×K labeled image within an M×N analysis frame.

[0159] In one embodiment, a J×K labeled image is generated by cropping a portion from a large image, and the cropped J×K portion is placed in an M×N frame. Examples of large image sizes include an image with dimensions of 504×224 pixels or even larger images.

[0160] The labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image can be reduced to 1 / 2 to 1 / 50 times the original resolution. In one embodiment, the segmented high-resolution image obtained from a scanner or genotyping device has a size of 3600×1600 pixels.

[0161] The plurality of failure categories can include at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0162] The plurality of failure categories can include the remaining failure categories that indicate an unsound pattern on the image due to an unidentified cause of failure.

[0163] The computer-implemented method described above can be executed in a system including computer hardware. The computer-implemented system can execute one or more of the methods described above. The computer-implemented system can incorporate any of the features of the methods described immediately above or throughout this application that are applicable to the methods implemented by the system. For the sake of brevity, alternative combinations of system features are not individually listed. Features applicable to the system, method, and article of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with the basic features in other statutory classifications.

[0164] As a manufactured article rather than a method, program instructions executable by a processor can be stored in a non-transitory computer readable medium (CRM). When the program instructions are executed, one or more of the computer-implemented methods described above are implemented. Alternatively, the program instructions may be stored in a non-transitory CRM and, when combined with appropriate hardware, may form one or more components of a computer-implemented system that implements the disclosed methods.

[0165] Each of the features considered in this specific embodiment section for method embodiments is equally applicable to CRM and system embodiments. As noted above, all method features are not repeated here and should be considered by reference and repeated as appropriate. Combined, single-pass good-versus-bad detection and root cause of process failure

[0166] In one embodiment of the disclosed technology, a method for training a convolutional neural network (CNN) to identify and classify images of segments of an image generation chip from defective process cycles that result in process cycle failures is described. The method includes using a pre-trained CNN to extract image features. The pre-trained CNN can accept an image of dimension M×N. The method includes creating a training dataset using labeled images of dimension J×K that are normal and belong to at least one failure category of a plurality of failure categories that result in process failures. The images are from segments of the image generation chip. The method includes positioning the J×K labeled images at multiple positions within an M×N (224×224) frame. The method includes filling around the edges of a particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling M×N. The method includes further training the pre-trained CNN to generate a segment classifier using the training dataset. The method includes storing the coefficients of the trained classifier to identify and classify images of segments of the image generation chip from a production process cycle. The trained classifier can accept an image of a segment of the image generation chip and classify the image as normal or as belonging to at least one failure category of a plurality of failure categories that result in process failures.

[0167] In a production embodiment, the trained CNN can be applied to classify defective process cycle images of the segment. The inventors present a method for identifying defective process cycle images of segments of an image generation chip that cause failures in the process cycle. The method includes creating an input to the trained classifier.

[0168] The input can fill the M×N input aperture of the image processing framework or can be reflected to have a small dimension J×K and fill the M×N analysis frame. Taking the latter approach, creating the input involves accessing a segmented image having dimension J×K. The method involves positioning the J×K image within the M×N analysis frame. The method involves filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned within the M×N analysis frame. Depending on the relative sizes of J×K to M×N, some zero-padding may be applied, for example, to fill narrow strips along the top and bottom of the analysis frame, although reflection has been found to work better. The method involves inputting the M×N analysis frame to a trained classifier. The method involves using the trained classifier to classify the segmented image of the image generation chip as normal or as belonging to at least one of a plurality of failure categories. The method involves outputting the classification result of the segmented image of the image generation chip.

[0169] The method may also include features described in relation to the methods presented above. For the sake of brevity, alternative combinations of method features are not individually listed. Features applicable to methods, systems, and articles of manufacture are not repeated for each legal classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other legal classifications.

[0170] The computer-implemented method described above can be executed in a system including computer hardware. A computer-implemented system can execute one or more of the methods described above. A computer-implemented system can incorporate any of the features of the methods described immediately above or throughout this application as applied to the methods implemented by the system. For the sake of brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory grouping of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0171] Instead of a method, program instructions executable by a processor can be stored on a non-transitory computer readable medium (CRM). When the program instructions are executed, one or more of the computer-implemented methods described above are implemented. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may be components of one or more computer-implemented systems that implement the disclosed methods.

[0172] Each of the features considered in this particular embodiment section for method embodiments is equally applicable to CRM and system embodiments. As noted above, all method features are not repeated here and should be considered repeated by reference. Item

[0173] The following clauses describe various aspects of the technology described in this specification. Separate good detections from bad detections and then classify the root causes. Training good vs. bad

[0174] Clause 1. A method of training a convolutional neural network to identify and classify images of sections of an image generation chip that result in process failures, Using a convolutional neural network pre-trained to extract image features, the step where the pre-trained convolutional neural network accepts an image of dimension M×N, Creating a training dataset using labeled images of dimension J×K, which is smaller than M×N, to depict process success and failure, where the labeled images are from the classification of the image generation chip, Positioning the J×K labeled images at multiple positions within the M×N frame, Using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image, thereby filling the M×N frame, Further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset, Remembering the coefficients of the trained classifier to identify and classify images of the classification of the image generation chip from the production process cycle including, whereby the trained classifier can accept an image of the classification of the image generation chip and classify the image as depicting process success and failure.

[0175] Clause 2. The step of using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image is the step of applying a horizontal reflection to at least a portion of the specific J×K labeled image The method according to clause 1, further including.

[0176] Clause 3. The step of using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image is the step of applying a vertical reflection to at least a portion of the specific J×K labeled image The method according to clause 1, further including.

[0177] Step 4. The step of creating a training data set using a labeled image of dimension J×K, generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The method according to claim 1, further comprising.

[0178] Step 5. The method according to claim 1, wherein the labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0179] Step 6. The method according to claim 5, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0180] Step 7. A non-transitory computer-readable storage medium having computer program instructions for identifying and classifying images of a section of an image generation chip that result in process failure, the instructions, when executed on a processor, using a convolutional neural network pre-trained to extract image features, the pre-trained convolutional neural network receiving an image of dimension M×N, creating a training data set using labeled images of dimension J×K smaller than M×N that depict process success and failure, wherein the labeled images are from a section of the image generation chip, positioning the J×K labeled images at multiple positions within the M×N frame, filling around the edges of a particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N frame, further training the pre-trained convolutional neural network to generate a classification classifier using the training data set, Storing the coefficients of the trained classifier to identify and classify images of image generation chips from the production process cycle including whereby the trained classifier can accept an image of the image generation chip segment and classify the image as normal or depicting a process failure A non - transitory computer - readable storage medium for implementing the method

[0181] Clause 8. The step of filling around the edge of a specific J×K labeled image using at least a portion of the specific J×K labeled image is the step of applying a horizontal reflection to at least a portion of the specific J×K labeled image The non - transitory computer - readable storage medium according to clause 7, for implementing a method further including this

[0182] Clause 9. The step of filling around the edge of a specific J×K labeled image using at least a portion of the specific J×K labeled image is the step of applying a vertical reflection to at least a portion of the specific J×K labeled image The non - transitory computer - readable storage medium according to clause 7, for implementing a method further including this

[0183] Clause 10. The step of creating a training data set using a labeled image of dimension J×K is generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The non - transitory computer - readable storage medium according to clause 7, for implementing a method further including this

[0184] Clause 11. The non - transitory computer - readable storage medium according to clause 7, wherein the labeled image of dimension J×K is obtained by downsampling a high - resolution image from a scanner, and as a result, the resolution of the high - resolution image is reduced to 1 / 2 to 1 / 50 times per side

[0185] Clause 12. The non - transitory computer - readable storage medium according to Clause 11, wherein the high - resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0186] Clause 13. A system including one or more processors coupled to a memory, wherein the memory is loaded with computer instructions to identify and classify images of segments of an image - generation chip that result in process failures, and when those instructions are executed on the processor, to use a convolutional neural network pre - trained to extract image features, the pre - trained convolutional neural network accepting an image of dimension M×N, to create a training data set using labeled images of dimension J×K, smaller than M×N, that depict process success and failure, wherein the labeled images are from segments of the image - generation chip, to position the J×K labeled images at multiple positions within the M×N frame, to create by using at least a portion of a particular J×K labeled image to fill around the edge of the particular J×K labeled image, thereby filling the M×N frame, to further train the pre - trained convolutional neural network to generate a segment classifier using the training data set, to store the coefficients of the trained classifier to identify and classify images of segments of the image - generation chip from a production - process cycle, and whereby the trained classifier can accept an image of a segment of the image - generation chip and classify the image as normal or depicting a process failure. A system that performs the operations.

[0187] Clause 14. Using at least a portion of a particular J×K labeled image to fill around the edge of the particular J×K labeled image is applying a horizontal reflection to at least a portion of the particular J×K labeled image The system according to clause 13, further performing an operation including

[0188] Clause 15. Using at least a part of a specific J×K labeled image to fill around the edge of the specific J×K labeled image, applying a vertical reflection to at least a part of the specific J×K labeled image The system according to clause 13, further performing an operation including

[0189] Clause 16. Creating a training data set using a labeled image of dimension J×K, generating a J×K labeled image by cutting out a part from a large image and placing the cut J×K part in an M×N frame The system according to clause 13, further performing an operation including

[0190] Clause 17. The system according to clause 13, wherein a labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0191] Clause 18. The system according to clause 17, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels. Training root cause analysis

[0192] Clause 19. A method for training a convolutional neural network to classify an image of a section of an image generation chip by the root cause of a process failure, using a pre-trained convolutional neural network pre-trained to extract image features, the pre-trained convolutional neural network receiving an image of dimension M×N, creating a training data set using a labeled image of dimension J×K belonging to at least one failure category among a plurality of failure categories causing a process failure, The labeled image is from the classification of the image generation chip, Position the J×K labeled image at multiple positions within the M×N frame, Using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image, thereby filling the M×N frame, steps; Further training a pre-trained convolutional neural network to generate a classification classifier using the training dataset; Storing the coefficients of the trained classifier to identify and classify the images of the classification of the image generation chip from the production process cycle; Including, Thereby, a method by which the trained classifier can receive an image of the classification of the image generation chip and classify the image according to the root cause of the process failure from among a plurality of failure categories.

[0193] Clause 20. The step of using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image is Applying a horizontal reflection to at least a portion of the specific J×K labeled image The method according to clause 19, further including.

[0194] Clause 21. The step of using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image is Applying a vertical reflection to at least a portion of the specific J×K labeled image The method according to clause 19, further including.

[0195] Clause 22. The step of creating a training dataset using a labeled image of dimension J×K is Generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in the M×N frame The method according to clause 19, further including.

[0196] Clause 23. The method according to Clause 19, wherein the labeled image of dimension J×K is obtained by downsampling the high-resolution image from the scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0197] Clause 24. The method according to Clause 23, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0198] Clause 25. The method according to Clause 19, wherein the plurality of failure categories include at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0199] Clause 26. The method according to Clause 19, wherein the plurality of failure categories include the remaining failure categories indicating the defective patterns on the image due to the unidentified cause of failure.

[0200] Clause 27. A non-transitory computer-readable storage medium having computer program instructions for training a convolutional neural network to classify the images of the sections of the image generation chip according to the root cause of the process failure, wherein when the instructions are executed on a processor, using a convolutional neural network pre-trained to extract image features, wherein the pre-trained convolutional neural network accepts an image of dimension M×N, creating a training data set using the labeled image of dimension J×K belonging to at least one of the plurality of failure categories that result in process failure, wherein the labeled image is from the section of the image generation chip, positioning the J×K labeled image at a plurality of positions within the M×N frame, filling around the edge of the specific J×K labeled image using at least a part of the specific J×K labeled image, thereby filling the M×N frame. Further training a pre-trained convolutional neural network to generate a classification classifier using a training dataset, Storing the coefficients of the trained classifier to identify and classify images of the image generation chip category from the production process cycle including Thereby, the trained classifier can receive an image of the image generation chip category and classify the image according to the root cause of the process failure from among a plurality of failure categories. A non-transitory computer-readable storage medium that implements the method.

[0201] Clause 28. The step of filling around the edge of a specific J×K labeled image using at least a portion of the specific J×K labeled image is The step of applying a horizontal reflection to at least a portion of the specific J×K labeled image The non-transitory computer-readable storage medium according to clause 27, further including

[0202] Clause 29. The step of filling around the edge of a specific J×K labeled image using at least a portion of the specific J×K labeled image is The step of applying a vertical reflection to at least a portion of the specific J×K labeled image The non-transitory computer-readable storage medium according to clause 27, which implements a method further including

[0203] Clause 30. The step of creating a training dataset using a labeled image of dimension J×K is Generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The non-transitory computer-readable storage medium according to clause 27, which implements a method further including

[0204] Clause 31. The non-transitory computer-readable storage medium according to Clause 27, wherein a labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0205] Clause 32. The non-transitory computer-readable storage medium according to Clause 31, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0206] Clause 33. The non-transitory computer-readable storage medium according to Clause 27, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0207] Clause 34. The non-transitory computer-readable storage medium according to Clause 27, wherein the plurality of failure categories includes the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure.

[0208] Clause 35. A system including one or more processors coupled to a memory, wherein the memory is loaded with computer instructions for training a convolutional neural network to classify an image of a section of an image generation chip according to the root cause of a process failure, and when the instructions are executed on the processor, using a pre-trained convolutional neural network pre-trained to extract image features, the pre-trained convolutional neural network accepting an image of dimension M×N, creating a training dataset using a labeled image of dimension J×K belonging to at least one of the plurality of failure categories that result in a process failure, the labeled image being from a section of the image generation chip, positioning the J×K labeled image at a plurality of positions within the M×N frame, Using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image, thereby filling an M×N frame, creating, Further training a pre-trained convolutional neural network to generate a classification classifier using a training dataset, Storing the coefficients of the trained classifier to identify and classify the segmented images of the image generation chips from the production process cycle Including, Thereby, the trained classifier can receive the segmented image of the image generation chip and classify the image according to the root cause of the process failure from among a plurality of failure categories. A system that performs the operation.

[0209] Clause 36. Using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image is Applying a horizontal reflection to at least a portion of the specific J×K labeled image The system according to clause 35, further performing an operation including.

[0210] Clause 37. Using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image is Applying a vertical reflection to at least a portion of the specific J×K labeled image The system according to clause 35, further performing an operation including.

[0211] Clause 38. Using a labeled image of dimension J×K to create a training dataset is Generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The system according to clause 35, further performing an operation including.

[0212] Clause 39. The system according to Clause 35, wherein a labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0213] Clause 40. The system according to Clause 39, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0214] Clause 41. The system according to Clause 35, wherein the plurality of failure categories include at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0215] Clause 42. The system according to Clause 35, wherein the plurality of failure categories include the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure.

[0216] Inference Good vs. Bad Clause 43. A method for identifying a process cycle image of a section of an image generation chip that causes a failure in a process cycle, comprising: creating an input to a trained classifier, the step of: accessing an image of the section having dimension J×K; positioning the J×K image in an M×N analysis frame; filling around the edge of the specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N analysis frame; inputting the M×N analysis frame to a trained classifier, the step of: the trained classifier classifying the image of the section of the image generation chip as normal or depicting a process failure; outputting a classification result of the section of the image generation chip; and including.

[0217] The method according to clause 43, further comprising the step of positioning the J×K image at the center of the M×N analysis frame.

[0218] Clause 45. The step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image, The step of applying a horizontal reflection to at least a portion of the specific J×K image The method according to clause 43, further comprising.

[0219] Clause 46. The step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image, The step of applying a vertical reflection to at least a portion of the specific J×K image The method according to clause 43, further comprising.

[0220] Clause 47. The step of creating an input to a trained classifier, The step of generating a J×K image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The method according to clause 43, further comprising.

[0221] Clause 48. The method according to clause 43, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0222] Clause 49. The method according to clause 48, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0223] Clause 50. When it is determined that the image of the section of the image generation chip depicts a process failure, the step of accessing a second convolutional neural network, wherein the second convolutional neural network is trained to classify the image according to the root cause of the process failure from among a plurality of failure categories. Applying a second convolutional neural network to the segmented image to classify among a plurality of failure categories and select, from among the plurality of failure categories, a possible root cause of the failure process cycle; The method according to clause 43, further comprising.

[0224] Clause 51. The method according to clause 50, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0225] Clause 52. The method according to clause 50, wherein the plurality of failure categories includes the remaining failure categories that exhibit an unsound pattern on the image due to an unidentified cause of failure.

[0226] Clause 53. A non-transitory computer-readable storage medium having computer program instructions for identifying a segmented process cycle image of an image generation chip that causes a failure in a process cycle, the instructions, when executed on a processor, Creating an input to a trained classifier, comprising: Accessing a segmented image having dimensions J×K; Positioning the J×K image within an M×N analysis frame; Filling around the edges of a specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N analysis frame; Inputting the M×N analysis frame to a trained classifier, comprising: The trained classifier classifying the segmented image of the image generation chip as normal or depicting a process failure; Outputting the classification result of the segmented image generation chip; A non-transitory computer-readable storage medium for implementing the method.

[0227] The non - transitory computer - readable storage medium according to clause 53, implementing a method further including the step of positioning a J×K image at the center of an M×N analysis frame.

[0228] Clause 55. The step of filling around the edge of a specific J×K image using at least a part of the specific J×K image, The step of applying a horizontal reflection to at least a part of the specific J×K image The non - transitory computer - readable storage medium according to clause 53, implementing a method further including the above.

[0229] Clause 56. The step of filling around the edge of a specific J×K image using at least a part of the specific J×K image, The step of applying a vertical reflection to at least a part of the specific J×K image The non - transitory computer - readable storage medium according to clause 53, implementing a method further including the above.

[0230] Clause 57. The step of creating an input to a trained classifier, The step of generating a J×K image by cutting a part from a large image and placing the cut J×K part in an M×N frame The non - transitory computer - readable storage medium according to clause 53, implementing a method further including the above.

[0231] Clause 58. The non - transitory computer - readable storage medium according to clause 53, wherein the J×K image is obtained by downsampling a high - resolution image from a scanner, and as a result, the resolution of the high - resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0232] Clause 59. The non - transitory computer - readable storage medium according to clause 58, wherein the high - resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0233] Clause 60. When it is determined that the segmented image of the image generation chip depicts a process failure, accessing a second convolutional neural network, wherein the second convolutional neural network is trained to classify the image according to the root cause of the process failure from among a plurality of failure categories; applying the second convolutional neural network to the segmented image to select, from among a plurality of failure categories, a possible root cause of the failure process cycle as the root cause; A non-transitory computer-readable storage medium according to clause 53, implementing a method further comprising the steps above.

[0234] Clause 61. The non-transitory computer-readable storage medium according to clause 60, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0235] Clause 62. The non-transitory computer-readable storage medium according to clause 60, wherein the plurality of failure categories includes the remaining failure categories indicating defective patterns on the image due to unidentified failure causes.

[0236] Clause 63. A system including one or more processors coupled to a memory, wherein the memory is loaded with computer instructions to identify a process cycle image of a segment of an image generation chip that causes a failure in a process cycle, and when the instructions are executed on the processor, creating an input to a trained classifier, including: accessing a segmented image having dimensions J×K; positioning the J×K image in an M×N analysis frame; filling around the edges of a specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N analysis frame; inputting the M×N analysis frame to the trained classifier; Inputting and classifying, by a trained classifier, an image of a section of an image generation chip as normal or as depicting a process failure, Outputting a classification result for the section of the image generation chip A non-transitory computer-readable storage medium that implements a method comprising the above.

[0237] The system according to clause 63, further performing an operation including positioning a 64.J×K image at the center of an M×N analysis frame.

[0238] Clause 65. Using at least a portion of a specific J×K image to fill around the edges of the specific J×K image, Applying a horizontal reflection to at least a portion of the specific J×K image The system according to clause 63, further performing an operation including the above.

[0239] Clause 66. Using at least a portion of a specific J×K image to fill around the edges of the specific J×K image, Applying a vertical reflection to at least a portion of the specific J×K image The system according to clause 63, further performing an operation including the above.

[0240] Clause 67. Creating an input to a trained classifier, Generating a J×K image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The system according to clause 63, further performing an operation including the above.

[0241] Clause 68. The J×K image is obtained by downsampling a high-resolution image from a scanner, as a result of which the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side, the system according to clause 63.

[0242] Clause 69. The high-resolution image obtained from the scanner has a size of 3600×1600 pixels, the system according to clause 68.

[0243] Article 70. When it is determined that the sectional image of the image generation chip depicts a process failure, accessing a second convolutional neural network, wherein the second convolutional neural network is trained to classify the image according to the root cause of the process failure from among a plurality of failure categories, and applying the second convolutional neural network to the sectional image to select, from among the plurality of failure categories, a possible root cause of the failure process cycle as performing further operations including, the system according to Article 63.

[0244] Article 71. The system according to Article 70, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0245] Article 72. The system according to Article 70, wherein the plurality of failure categories includes the remaining failure categories that exhibit defective patterns on the image due to unidentified failure causes. Inferred root cause

[0246] Article 73. A method for identifying and classifying a process cycle image of a section of an image generation chip that causes a failure in a process cycle, comprising creating an input to a trained classifier, comprising accessing a sectional image having dimensions J×K, and positioning the J×K image in an M×N analysis frame, and filling around the edges of a specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N analysis, inputting the M×N analysis frame to the trained classifier, A step in which a trained classifier classifies an image of a section of an image generation chip according to the root cause of process failure from among a plurality of failure categories, A step of outputting a classification result of the section of the image generation chip A non-transitory computer-readable storage medium that implements a method including these steps.

[0247] The method according to clause 73, further including a step of positioning the J×K image of clause 74 at the center of the M×N analysis frame.

[0248] The method according to clause 73, wherein the step of filling around the edge of a specific J×K image using at least a part of the specific J×K image includes a step of applying a horizontal reflection to at least a part of the specific J×K image The method according to clause 73, further including these steps.

[0249] The method according to clause 73, wherein the step of filling around the edge of a specific J×K image using at least a part of the specific J×K image includes a step of applying a vertical reflection to at least a part of the specific J×K image The method according to clause 73, further including these steps.

[0250] The method according to clause 73, wherein the step of creating an input to the trained classifier includes a step of generating a J×K image by cutting out a part from a large image and arranging the cut-out J×K part in an M×N frame The method according to clause 73, further including these steps.

[0251] The method according to clause 73, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0252] The method according to clause 78, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0253] Clause 80. The method according to clause 73, wherein the plurality of failure categories include at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0254] Clause 81. The method according to clause 73, wherein the plurality of failure categories include the remaining failure categories indicating defective patterns on the image due to unidentified failure causes.

[0255] Clause 82. A non-transitory computer-readable storage medium having computer program instructions for identifying and classifying a process cycle image of a section of an image generation chip that causes a process cycle failure, the instructions, when executed on a processor, perform a step of creating an input to a trained classifier, including a step of accessing an image of the section having dimensions J×K, a step of positioning the J×K image in an M×N analysis frame, a step of filling around the edge of a specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N analysis, a step of inputting the M×N analysis frame to a trained classifier, a step of the trained classifier classifying the image of the section of the image generation chip according to the root cause of the process failure from among the plurality of failure categories, and a step of outputting a classification result of the section of the image generation chip. A non-transitory computer-readable storage medium for implementing the method including the above steps.

[0256] Clause 83. A step of positioning the J×K image at the center of the M×N analysis frame The non-transitory computer-readable storage medium according to clause 82, implementing a method further including the above step.

[0257] Clause 84. The step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image The step of applying a horizontal reflection to at least a portion of a specific J×K image A non - transitory computer - readable storage medium according to clause 82, which implements a method further including this step

[0258] Clause 85. The step of using at least a portion of a specific J×K image to fill around the edge of the specific J×K image The step of applying a vertical reflection to at least a portion of a specific J×K image A non - transitory computer - readable storage medium according to clause 82, which implements a method further including this step

[0259] Clause 86. The step of creating an input to a trained classifier Generating a J×K image by cropping a portion from a large image and placing the cropped J×K portion in an M×N frame A non - transitory computer - readable storage medium according to clause 82, which implements a method further including this step

[0260] Clause 87. A non - transitory computer - readable storage medium according to clause 82, wherein the J×K image is obtained by downsampling a high - resolution image from a scanner, and as a result, the resolution of the high - resolution image is reduced to 1 / 2 to 1 / 50 times per side

[0261] Clause 88. A non - transitory computer - readable storage medium according to clause 87, wherein the high - resolution image obtained from the scanner has a size of 3600×1600 pixels

[0262] Clause 89. A non - transitory computer - readable storage medium according to clause 82, wherein the plurality of failure categories includes at least hybridization failure, space - shift failure, offset failure, surface - wear failure, and reagent - flow failure

[0263] Clause 90. A non - transitory computer - readable storage medium according to clause 82, wherein the plurality of failure categories includes the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure

[0264] Clause 91. A system including one or more processors coupled to a memory, wherein the memory is loaded with computer instructions to identify and classify process cycle images of segments of an image generation chip that cause process cycle failures, and when the instructions are executed on the processor, creating an input to a trained classifier, comprising accessing an image of a segment having dimensions J×K, positioning the J×K image within an M×N analysis frame, filling around the edges of a particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N analysis, and inputting the M×N analysis frame to a trained classifier, wherein the trained classifier classifies the image of the segment of the image generation chip by the root cause of the process failure from among a plurality of failure categories, and outputting a classification result for the segment of the image generation chip A non - transitory computer - readable storage medium implementing the method.

[0265] Clause 92. Positioning the J×K image at the center of the M×N analysis frame The system according to Clause 91, further performing an operation including this.

[0266] Clause 93. Filling around the edges of a particular J×K image using at least a portion of the particular J×K image further includes applying a horizontal reflection to at least a portion of the particular J×K image The system according to Clause 91, further performing an operation including this.

[0267] Clause 94. Filling around the edges of a particular J×K image using at least a portion of the particular J×K image further includes applying a vertical reflection to at least a portion of the particular J×K image The system according to clause 91, further performing an operation including

[0268] Clause 95. Creating an input to a trained classifier, Generating a J×K image by cutting out a portion from a large image and placing the cut-out J×K portion in an M×N frame The system according to clause 91, further performing an operation including

[0269] Clause 96. The system according to clause 91, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0270] Clause 97. The system according to clause 96, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0271] Clause 98. The system according to clause 91, wherein the plurality of failure categories include at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0272] Clause 99. The system according to clause 91, wherein the plurality of failure categories include the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure. Combined, single-pass good-versus-bad detection and root cause of process failure Combination of training

[0273] Clause 100. A method of training a convolutional neural network to identify and classify an image of a section of an image generation chip that causes a process cycle failure, Using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting an image of dimension M×N, A step of creating a training dataset using labeled images of dimension J×K belonging to at least one failure category among a plurality of failure categories that are normal and result in process failure, wherein the labeled images are from the classification of the image generation chip, positioning the J×K labeled images at a plurality of positions within the M×N frame, filling around the edges of a specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N frame, a step of further training a pre-trained convolutional neural network to generate a classification classifier using the training dataset, a step of storing the coefficients of the trained classifier to identify and classify images of the classification of the image generation chip from the production process cycle and including whereby the trained classifier can receive an image of the classification of the image generation chip and classify the image as normal or belonging to at least one failure category among a plurality of failure categories that result in process failure.

[0274] Clause 101. The step of filling around the edges of a specific J×K labeled image using at least a portion of the specific J×K labeled image is the step of applying a horizontal reflection to at least a portion of the specific J×K labeled image The method according to Clause 100, further including.

[0275] Clause 102. The step of filling around the edges of a specific J×K labeled image using at least a portion of the specific J×K labeled image is the step of applying a vertical reflection to at least a portion of the specific J×K labeled image The method according to Clause 100, further including.

[0276] Clause 103. The step of creating a training dataset using labeled images of dimension J×K is Generating a J×K labeled image by cutting out a portion from a large image and placing the cut-out J×K portion in an M×N frame The method according to clause 100, further comprising

[0277] Clause 104. The method according to clause 100, wherein the J×K labeled image is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image can be reduced to 1 / 2 to 1 / 50 times per side.

[0278] Clause 105. The method according to clause 104, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0279] Clause 106. The method according to clause 100, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0280] Clause 107. The method according to clause 100, wherein the plurality of failure categories includes the remaining failure categories indicating defective patterns on the image due to unidentified failure causes.

[0281] Clause 108. A non-transitory computer-readable storage medium having computer program instructions for training a convolutional neural network to identify and classify images of a section of an image generation chip that results in a process cycle failure, the instructions, when executed on a processor, Using a pre-trained convolutional neural network trained to extract image features, the pre-trained convolutional neural network accepting an image of dimension M×N, Creating a training data set using a J×K labeled image that is normal and belongs to at least one of the plurality of failure categories that result in a process failure, wherein the labeled image is from a section of the image generation chip, Position J×K labeled images at multiple positions within an M×N frame, Using at least a portion of a particular J×K labeled image, filling around the edge of the particular J×K labeled image, thereby filling the M×N frame, a step; Further training a pre-trained convolutional neural network to generate a classifier using a training dataset, a step; Storing the coefficients of the trained classifier to identify and classify images of the classification of image generation chips from a production process cycle, a step including, Thereby, a trained classifier can receive an image of the classification of an image generation chip and classify the image as belonging to at least one failure category among a plurality of failure categories that result in normal or process failure. A non-transitory computer-readable storage medium implementing a method.

[0282] Clause 109. The step of filling around the edge of a particular J×K labeled image using at least a portion of the particular J×K labeled image is The step of applying a horizontal reflection to at least a portion of the particular J×K labeled image A non-transitory computer-readable storage medium according to clause 108, implementing a method further including.

[0283] Clause 110. The step of filling around the edge of a particular J×K labeled image using at least a portion of the particular J×K labeled image is The step of applying a vertical reflection to at least a portion of the particular J×K labeled image A non-transitory computer-readable storage medium according to clause 108, implementing a method further including.

[0284] Clause 111. The step of creating a training dataset using a J×K labeled image of dimension J×K is Generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The non - transient computer - readable storage medium according to clause 108, implementing a method further comprising

[0285] The non - transient computer - readable storage medium according to clause 108, wherein a labeled image of dimension J×K is obtained by downsampling a high - resolution image from a scanner, and as a result, the resolution of the high - resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0286] The non - transient computer - readable storage medium according to clause 112, wherein the high - resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0287] The non - transient computer - readable storage medium according to clause 108, wherein the plurality of failure categories include at least hybridization failure, space - shift failure, offset failure, surface - wear failure, and reagent - flow failure.

[0288] The non - transient computer - readable storage medium according to clause 108, wherein the plurality of failure categories include the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure.

[0289] A system including one or more processors coupled to a memory, wherein the memory is loaded with computer instructions to train a convolutional neural network to identify and classify images of segments of an image generation chip that result in process - cycle failures, and when the instructions are executed on the processor, Using a pre - trained convolutional neural network pre - trained to extract image features, the pre - trained convolutional neural network accepting an image of dimension M×N, Creating a training data set using a labeled image of dimension J×K that is normal and belongs to at least one of the plurality of failure categories that result in process failures, wherein the labeled image is from a segment of the image generation chip Position J×K labeled images at multiple positions within an M×N frame, Using at least a portion of a specific J×K labeled image, fill around the edges of the specific J×K labeled image, thereby filling the M×N frame and creating, Further train a pre-trained convolutional neural network to generate a classifier using a training dataset, Remember the coefficients of the trained classifier to identify and classify images of the classification of image generation chips from the production process cycle including, Thereby, a trained classifier can receive an image of the classification of an image generation chip and classify the image as belonging to at least one failure category among a plurality of failure categories that result in normal or process failure, and perform an operation, a system.

[0290] Clause 117. Using at least a portion of a specific J×K labeled image to fill around the edges of the specific J×K labeled image is, Applying a horizontal reflection to at least a portion of the specific J×K labeled image The system according to clause 116, which further performs an operation including.

[0291] Clause 118. Using at least a portion of a specific J×K labeled image to fill around the edges of the specific J×K labeled image is, Applying a vertical reflection to at least a portion of the specific J×K labeled image The system according to clause 116, which further performs an operation including.

[0292] Clause 119. Using an image labeled with a dimension J×K to create a training dataset is, Generating a J×K labeled image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The system according to clause 116, which further performs an operation including.

[0293] Clause 120. The system according to clause 116, wherein a labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0294] Clause 121. The system according to clause 120, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0295] Clause 122. The system according to clause 116, wherein the plurality of failure categories include at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0296] Clause 123. The system according to clause 116, wherein the plurality of failure categories include the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure. Combined inference

[0297] Clause 124. A method for identifying a process cycle image of a section of an image generation chip that causes a failure in a process cycle, comprising: creating an input to a trained classifier, comprising: accessing an image of the section having dimension J×K; positioning the J×K image in an M×N analysis frame; filling around the edge of a specific labeled image of J×K using at least a portion of the specific labeled image of J×K, thereby filling the M×N analysis; inputting the M×N analysis frame to a trained classifier, comprising: the trained classifier classifying the image of the section of the image generation chip as normal or belonging to at least one of a plurality of failure categories; outputting a classification result of the section of the image generation chip; A non - transitory computer - readable storage medium for implementing a method, including

[0298] The method according to clause 124, further including the step of positioning a J×K image at the center of an M×N analysis frame.

[0299] In clause 126, the step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image is The step of applying a horizontal reflection to at least a portion of the specific J×K image The method according to clause 124, further including

[0300] In clause 127, the step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image is The step of applying a vertical reflection to at least a portion of the specific J×K image The method according to clause 124, further including

[0301] In clause 128, the step of creating an input to a trained classifier is The step of generating a J×K image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame The method according to clause 124, further including

[0302] The method according to clause 129, wherein the dimension J×K image is obtained by downsampling a high - resolution image from a scanner, and as a result, the resolution of the high - resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0303] The method according to clause 130, wherein the high - resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0304] The method according to clause 131, wherein the plurality of failure categories include at least hybridization failure, space - shift failure, offset failure, surface wear failure, and reagent flow failure.

[0305] The method according to clause 124, wherein a plurality of failure categories include the remaining failure categories indicating an unsound pattern on an image due to an unidentified cause of failure.

[0306] A non-transitory computer-readable storage medium having computer program instructions for identifying a process cycle image of a section of an image generation chip that causes a failure in a process cycle, the instructions, when executed on a processor, creating an input to a trained classifier, the step including: accessing a sectioned image having dimensions J×K; positioning the J×K image in an M×N analysis frame; filling around the edge of a specific J×K labeled image using at least a portion of the specific J×K labeled image, thereby filling the M×N analysis; inputting the M×N analysis frame to a trained classifier; the trained classifier classifying the sectioned image of the image generation chip as normal or belonging to at least one of a plurality of failure categories; outputting a classification result of the section of the image generation chip; and a non-transitory computer-readable storage medium for implementing the method.

[0307] The non-transitory computer-readable storage medium according to clause 133, further including positioning the J×K image at the center of the M×N analysis frame.

[0308] In clause 135, the step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image further includes applying a horizontal reflection to at least a portion of the specific J×K image. The non-transitory computer-readable storage medium according to clause 133, implementing a method further including this step.

[0309] The step of filling around the edge of a specific J×K image using at least a portion of the specific J×K image is the step of applying a vertical reflection to at least a portion of the specific J×K image A non - transitory computer - readable storage medium according to clause 133, which implements a method further including the above.

[0310] The step of creating an input to a trained classifier is the step of generating a J×K image by cutting a portion from a large image and placing the cut J×K portion in an M×N frame A non - transitory computer - readable storage medium according to clause 133, which implements a method further including the above.

[0311] A non - transitory computer - readable storage medium according to clause 133, wherein the J×K image is obtained by downsampling a high - resolution image from a scanner, and as a result, the resolution of the high - resolution image is reduced to 1 / 2 to 1 / 50 times per side.

[0312] A non - transitory computer - readable storage medium according to clause 138, wherein the high - resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0313] A non - transitory computer - readable storage medium according to clause 133, wherein the plurality of failure categories includes at least hybridization failure, space - shift failure, offset failure, surface wear failure, and reagent flow failure.

[0314] A non - transitory computer - readable storage medium according to clause 133, wherein the plurality of failure categories includes the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure.

[0315] Clause 142. A system including one or more processors coupled to a memory, wherein the memory is loaded with computer instructions to identify a process cycle image of a section of an image generation chip that causes a process cycle failure, and when the instructions are executed on the processor, to create an input to a trained classifier, to access an image of the section having dimensions J×K, to position the J×K image within an M×N analysis frame, to create, including filling around the edges of a particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N analysis, to input the M×N analysis frame to a trained classifier, wherein the trained classifier classifies the image of the section of the image generation chip as normal or belonging to at least one of a plurality of failure categories, the inputting, and to output a classification result of the section of the image generation chip and including performing operations.

[0316] Clause 143. The system according to Clause 142, further including positioning the J×K image at the center of the M×N analysis frame.

[0317] Clause 144. The system according to Clause 142, wherein filling around the edges of a particular J×K image using at least a portion of the particular J×K image further includes applying a horizontal reflection to at least a portion of the particular J×K image and performing operations.

[0318] Clause 145. The system according to Clause 142, wherein filling around the edges of a particular J×K image using at least a portion of the particular J×K image further includes applying a vertical reflection to at least a portion of the particular J×K image and performing operations.

[0319] Clause 146. Creating an input to a trained classifier includes generating a J×K image by cutting out parts from a large image and placing the cut-out J×K parts in an M×N frame The system according to clause 142, which further performs an operation including this.

[0320] Clause 147. The J×K image is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side. The system according to clause 142.

[0321] Clause 148. The high-resolution image obtained from the scanner has a size of 3600×1600 pixels. The system according to clause 147.

[0322] Clause 149. The system according to clause 142, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface wear failure, and reagent flow failure.

[0323] Clause 150. The system according to clause 142, wherein the plurality of failure categories includes the remaining failure categories indicating an unsound pattern on the image due to an unidentified cause of failure. Computer system

[0324] FIG. 17 is a simplified block diagram of a computer system 1700 that may be used to implement the disclosed technology. The computer system typically includes at least one processor 1772 that communicates with a number of peripheral devices via a bus subsystem 1755. These peripheral devices may include a storage subsystem 1710 that includes, for example, a memory subsystem 1722 and a file storage subsystem 1736, a user interface input device 1738, a user interface output device 1776, and a network interface subsystem 1774. The input and output devices enable user interaction with the computer system. The network interface subsystem provides an interface to an external network that includes an interface to a corresponding interface device within other computer systems.

[0325] In one embodiment, a good-versus-bad classifier 151 for classifying defective images is communicatively linked to a storage subsystem and a user interface input device.

[0326] The user interface input device 1738 may include a pointing device such as a keyboard, mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated in a display, an audio input device such as a voice recognition system and a microphone, and other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computer system.

[0327] The user interface output device 1776 can include a non-visual display such as a display subsystem, a printer, a fax machine, or an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or any other mechanism for generating a visible image. The display subsystem can also provide a non-visual display such as an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from a computer system to a user or another machine or computer system.

[0328] The memory subsystem 1710 stores programming and data constructs that provide some or all of the functionality of the modules and methods described herein. These software modules are generally executed by a processor alone or in combination with other processors.

[0329] The memory used in the memory subsystem can include several memories such as a main random access memory (RAM) 1732 for storing instructions and data during program execution, a read only memory (ROM) 1734 in which fixed instructions are stored. The file storage subsystem 1736 provides a persistent storage device for program and data files and can include a hard disk drive, a related removable medium, a CD-ROM drive, an optical drive, or a removable media cartridge. The modules implementing the functions of a particular embodiment may be stored by the file storage subsystem within the memory subsystem or in another machine accessible by the processor.

[0330] The bus subsystem 1755 provides a mechanism for various components and subsystems of a computer system to communicate with each other as intended. Although the bus subsystem is schematically shown as a single bus, alternative embodiments of the bus subsystem can use multiple buses.

[0331] The computer system itself can be of various types, including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a mainframe, a server farm, a loosely distributed set of loosely networked computers, or any other data processing system or user device. Since computers and networks are of a constantly changing nature, the description of the computer system illustrated in FIG. 17 is intended only as a specific example for the purpose of exemplifying the disclosed technology. Many other configurations of the computer system can have more or fewer components than the computer system illustrated in FIG. 17.

[0332] The computer system 1700 includes a GPU or an FPGA 1778. It may also include a machine learning processor hosted by a machine learning cloud platform such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's Tensor Processing Unit (TPU), rack-mounted solutions such as the GX4 Rackmount Series and GX8 Rackmount Series, NVIDIA DGX-1, Microsoft's Stratix V FPGA, Graphcore's Intelligent Processor Unit (IPU), Qualcomm's Zeroth Platform with Snapdragon processors, NVIDIA's Volta, NVIDIA's DRIVE PX, NVIDIA's JETSON TX1 / TX2 MODULE, Intel's Nirvana, Movidius VPU, Fujitsu DPI, ARM's DynamicIQ, IBM TrueNorth, and others.

Explanation of Signs

[0333] 100 System 111 Genotyping Device 115 Process Cycle Image Database 117 Failure Category Label Database 138 Training Database 138 Process Cycle Image Database 138 Database 151 Good vs. Bad Classifier 151 Classifier 151 Output 155 Network 168 Reference Database 168 Database 171 Root Cause Classifier 171 Classifier 171 Random Forest Classifier

Claims

1. A method for training a convolutional neural network to identify and classify images of a section of an image generation chip that results in process failures, the method comprising: using the pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting an image of dimension M×N; creating a training dataset using labeled images of dimension J×K that are smaller than dimension M×N and depict process success and failure, wherein the labeled images are from the section of the image generation chip, positioning the J×K labeled images at multiple positions within at least one M×N frame, using at least a portion of a particular J×K labeled image to fill around the edges of the particular J×K labeled image, thereby filling the at least one M×N frame; using the training dataset to further train the pre-trained convolutional neural network to generate a trained classifier; storing the coefficients of the trained classifier to identify and classify images of the section of the image generation chip from a production process cycle. A method comprising the above steps.

2. The method according to claim 1, wherein the trained classifier is configured to accept the image of the section of the image generation chip and classify the image as depicting process success and failure.

3. The step of using at least a portion of the particular J×K labeled image to fill around the edges of the particular J×K labeled image further comprises: applying a horizontal reflection to at least a portion of the particular J×K labeled image. The method according to claim 1.

4. The step of using at least a portion of the particular J×K labeled image to fill around the edges of the particular J×K labeled image further comprises: applying a vertical reflection to at least a portion of the particular J×K labeled image. The method according to claim 1.

5. The step of creating the training dataset using the labeled images of dimension J×K comprises: Generating the J×K labeled image by cutting a portion from a large image, and placing the cut J×K portion in the at least one M×N frame The method according to claim 1, further comprising

6. The method according to claim 1, wherein the labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side

7. The method according to claim 6, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels

8. Identifying a process cycle image of a section of the image generation chip that causes a failure in the process cycle, which is creating an input to the trained classifier, including accessing the image of the section having the dimension J×K, positioning the J×K image in an M×N analysis frame, inputting the M×N analysis frame to the trained classifier, applying the trained classifier to classify the image of the section of the image generation chip as good or failed, and outputting a classification result of the section of the image generation chip The method according to claim 1, comprising

9. The step of creating an input to the trained classifier is using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image, thereby filling the M×N analysis frame The method according to claim 8, further comprising

10. At least one non-transitory computer-readable storage medium having computer program instructions for identifying and classifying an image of a section of an image generation chip that results in a process failure, the instructions, when executed on at least one processor, causing the at least one processor to use a pre-trained convolutional neural network pre-trained to extract image features, the pre-trained convolutional neural network accepting an image of dimension M×N, create a training dataset using labeled images of dimension J×K smaller than dimension M×N that depict process success and failure, the labeled image is from the partition of the image generation chip, position the J×K labeled image at multiple positions within at least one M×N frame, using at least a portion of a specific J×K labeled image to fill around the edge of the specific J×K labeled image, thereby filling the at least one M×N frame, creating, using the training data set to further train the pre-trained convolutional neural network to generate a trained classifier, storing the coefficients of the trained classifier to identify and classify images of the partition of the image generation chip from the production process cycle A non-transitory computer-readable storage medium configured to cause the above to be performed. **Claim 11** The non-transitory computer-readable storage medium according to claim 10, wherein the trained classifier is configured to receive the image of the partition of the image generation chip and classify the image as depicting process success and failure. **Claim 12** The at least one processor configured to use at least a portion of the specific J×K labeled image to fill around the edge of the specific J×K labeled image, The non-transitory computer-readable storage medium according to claim 10, comprising the at least one processor configured to apply a horizontal reflection to at least a portion of the specific J×K labeled image. **Claim 13** The at least one processor configured to use at least a portion of the specific J×K labeled image to fill around the edge of the specific J×K labeled image, The non-transitory computer-readable storage medium according to claim 10, comprising the at least one processor configured to apply a vertical reflection to at least a portion of the specific J×K labeled image. **Claim 14** The at least one processor configured to create the training data set using the labeled image of dimension J×K, The non-transitory computer-readable storage medium according to claim 10, comprising the at least one processor configured to generate the J×K labeled image by cutting a portion from a large image and arranging the cut J×K portion in the at least one M×N frame. **Claim 15** The non-transitory computer-readable storage medium according to claim 10, wherein the labeled image of dimension J×K is configured to be obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

16. The non-transitory computer-readable storage medium according to claim 15, wherein the high-resolution image acquired from the scanner has a size of 3600×1600 pixels.

17. The instructions cause the at least one processor to create an input to the trained classifier, which includes accessing the image of the division having the dimension J×K, positioning the J×K image in an M×N analysis frame, inputting the M×N analysis frame to the trained classifier, applying the trained classifier to classify the image of the division of the image generation chip as good or failed, and outputting the classification result of the division of the image generation chip. The non-transitory computer-readable storage medium according to claim 10, which is further configured to cause the above operations.

18. The instructions cause the at least one processor to fill around the edge of the specific J×K labeled image using at least a part of the specific J×K labeled image, thereby filling the M×N analysis frame. The non-transitory computer-readable storage medium according to claim 17, which is further configured to cause the above operation.

19. A system including one or more processors coupled to a memory, wherein the memory has computer instructions loaded for identifying and classifying an image of a division of an image generation chip that results in a process failure, and when the instructions are executed on the one or more processors, the one or more processors use a pre-trained convolutional neural network pre-trained to extract image features, wherein the pre-trained convolutional neural network accepts an image of dimension M×N, create a training dataset using labeled images of dimension J×K smaller than dimension M×N that depict process success and failure, wherein the labeled images are from the division of the image generation chip. Position the J×K labeled image at multiple positions within at least one M×N frame, Using at least a portion of a specific J×K labeled image, fill around the edges of the specific J×K labeled image, thereby filling the at least one M×N frame, creating, Using the training data set, further train the pre-trained convolutional neural network to generate a trained classifier, Store the coefficients of the trained classifier to identify and classify the images of the section of the image generation chip from the production process cycle A system configured to cause to perform.

20. The system according to claim 19, wherein the trained classifier is configured to receive an image of the section of the image generation chip and classify it as depicting process success and failure.

21. The instruction configured to cause the one or more processors to fill around the edges of the specific J×K labeled image using at least a portion of the specific J×K labeled image, An instruction configured to cause the one or more processors to apply a horizontal reflection to at least a portion of the specific J×K labeled image The system according to claim 19, comprising.

22. The instruction configured to cause the one or more processors to fill around the edges of the specific J×K labeled image using at least a portion of the specific J×K labeled image, The system according to claim 19, comprising an instruction configured to cause the one or more processors to apply a vertical reflection to at least a portion of the specific J×K labeled image.

23. The instruction configured to cause the one or more processors to create the training data set using a labeled image of dimension J×K, The system according to claim 19, comprising an instruction configured to cause the one or more processors to generate the J×K labeled image by cutting a portion from a large image and place the cut J×K portion in the at least one M×N frame.

24. The system according to claim 19, wherein the labeled image of dimension J×K is configured to be obtained by downsampling a high-resolution image from a scanner, and as a result, the resolution of the high-resolution image is reduced to 1 / 2 to 1 / 50 times per side.

25. The system according to claim 24, wherein the high-resolution image is configured to be acquired from the scanner so as to have a size of 3600×1600 pixels.

26. The instructions cause the one or more processors to create an input to the trained classifier, including accessing the image of the segment having the dimension J×K, positioning the J×K image in an M×N analysis frame, inputting the M×N analysis frame to the trained classifier, applying the trained classifier to classify the image of the segment of the image generation chip as good or failed, and outputting the classification result of the segment of the image generation chip. The system according to claim 19, wherein the system is configured to perform the above operations.

27. The instructions further cause the one or more processors to fill around the edge of the specific J×K labeled image using at least a part of the specific J×K labeled image, thereby filling the M×N analysis frame. The system according to claim 26, wherein the system is further configured to perform the above operations.

Images