Deep Learning Based Root Cause Analysis of Process Cycle Images

Abstract

The disclosed technology relates to training a convolutional neural network (CNN) to identify and classify images of segments of an imaging chip that cause process cycle failures. The disclosed technology includes creating a training dataset of images of dimension M×N using labeled images of segments of an imaging chip of dimension J×K. The disclosed technology may fill the M×N frame using horizontal and vertical reflections along edges of the J×K labeled images positioned within the M×N frame. The pre-trained CNN is further trained using the training dataset. The trained CNN may classify the segment images as normal or as depicting a failure. The disclosed technology may train a root cause CNN to classify process cycle images of segments that cause process cycle failures. The trained CNN may classify the segment images by root cause of the process failure among multiple failure categories.

Description

[Technical field]

[0001] <<Priority Application>> This application claims the benefit of U.S. Provisional Patent Application No. 63 / 143,673, entitled "DEEP LEARNING-BASED ROOT CAUSE ANALYSIS OF PROCESS CYCLES," filed on January 29, 2021 (Attorney Docket No. ILLM 1044-1 / IP-2089-PRV), which is incorporated by reference for all purposes.

[0002] "Built-in" below, U.S. Patent Application No. 171 / 161,595, entitled “MACHINE LEARNING-BASED ROOT CAUSE ANALYSIS OF PROCESS CYCLE IMAGES,” filed on January 28, 2021 (Attorney Docket No.: ILLM 1026-2 / IP-1911-US); U.S. Patent Application No. 17 / 332,904, entitled “MACHINE LERNING-BASED ANALYSIS OF PROCESS INDICATORS TO PREDICT SAMPLE REEVALUATION SUCCESS,” filed on May 27, 2021 (Attorney Docket No.: ILLM 1027-2 / IP-1973-US); U.S. Patent Application No. 17 / 548,424, filed December 10, 2021, entitled "MACHINE LEARNING-BASED GENOTYPING PROCESS OUTCOME PREDICTION USING AGGREGATE METRICS" (Attorney Docket No. ILLM 1028-2 / IP-1978-US), is incorporated by reference as if fully set forth herein.

[0003] The disclosed technology relates to classification of images for production process assessment and root cause failure analysis. [Background technology]

[0004] The subject matter discussed in this section should not be assumed to be prior art merely as a result of its reference in this section. Similarly, it should not be assumed that the problems referenced in this section, or related to the subject matter provided as background, have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which as such may also correspond to embodiments of the claimed technology.

[0005] Genotyping is a process that can take multiple days to complete. The process is vulnerable to both mechanical and chemical processing errors. Collected samples for genotyping are extracted and distributed into sections and areas of an imaging chip. The samples are then chemically processed through multiple steps to generate a fluorescent image. The process generates a quality score for each section analyzed. This quality score may not provide insight into the root cause of a low quality process failure. In some cases, a failed section image still produces an acceptable quality score.

[0006] Therefore, an opportunity has arisen to introduce new methods and systems for evaluating section images and determining root cause analysis of failures during production genotyping. Summary of the Invention [Means for solving the problem]

[0007] To address the above problems, one aspect of the present invention provides a method for training a convolutional neural network to identify and classify images of sections of an imaging chip that result in process failures. The method includes using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training dataset using labeled images of dimension J×K less than M×N depicting process success and failure, the labeled images being from a section of an imaging chip, positioning the J×K labeled images at multiple locations within the M×N frame, and using at least a portion of a particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; further training the pre-trained convolutional neural network to generate a section classifier using the training dataset; and storing coefficients of the trained classifier to identify and classify images of the section of the imaging chip from a production process cycle, such that the trained classifier can accept images of the section of the imaging chip and classify the images as depicting process success and failure.

[0008] In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosed technology. In the following description, various embodiments of the disclosed technology are described with reference to the following drawings, in which: [Brief description of the drawings]

[0009] [Figure 1] 1 shows an architectural level schematic of a system in which process cycle images from a genotyping instrument are classified and the root cause of bad or failed images is determined. [Diagram 2] 2 illustrates subsystem components of the feature generator of FIG. 1 . [Diagram 3]The process steps for an exemplary genotyping process are presented. [Figure 4] An image of a section located within the imaging chip after successful process completion is presented. [Figure 5A] An example of a failed segmentation image due to hybridization failure during the genotyping process is presented. [Figure 5B] An example of a failed segmentation image due to hybridization failure during the genotyping process is presented. [Figure 5C] An example of a failed segmented image due to a spacer shift failure is presented. [Figure 5D] An example of a failed segmented image due to a spacer shift failure is presented. [Figure 5E] An example of a failed segment image due to offset failure is presented. [Figure 5F] An example of a failure segment image due to surface wear failure is presented. [Figure 5G] 13 provides an example of a failure segment image due to a reagent flow failure. [Figure 5H] 13 provides an example of a failure segment image due to a reagent flow failure. [Figure 5I] We present examples of failed segmentation images where the source of the failure is unknown. [Figure 6A] 1 is an example of 96 intrinsic image components selected by ranking principal components generated using Principal Component Analysis (PCA). [Figure 6B] 6B is a close-up of the top 40 intrinsic image components selected from the 96 images in FIG. 6A. [Figure 7A] 1 illustrates image rescaling and smoothing of segmented images for input to principal component analysis. [Figure 7B] 1 illustrates the creation of eigenimages using principal component analysis. [Figure 8A] 1 illustrates an example of feature generation using the intensity of areas of a partition. [Figure 8B] 1 illustrates another example of feature generation using the intensity of the area of ​​the partition. [Figure 9] 1 is a diagrammatic illustration of a one-vs-the-rest (OvR) classifier. [Figure 10A] We illustrate the training of binary (good vs. bad) and multi-class (root cause) classifiers using labeled training data that includes process cycle images. [Figure 10B] 1 illustrates a two-step process in which production process cycle images are classified as good versus bad and then a failure category is determined for the bad images. [Figure 11A] We present examples of feature engineering techniques applied to training data images. [Figure 11B] We present examples of feature engineering techniques applied to training data images. [Figure 12A] 1 illustrates a transfer learning step for training a convolutional neural network using fine tuning. [Figure 12B] 1 presents a network architecture for an exemplary convolutional neural network model. [Figure 12C] We present a first convolutional neural network example architecture for identifying and classifying bad process cycle images. [Figure 12D] We present an exemplary architecture of a second convolutional neural network for identifying and classifying bad process cycle images. [Figure 12E] We present an exemplary architecture of a second convolutional neural network for identifying and classifying bad process cycle images. [Figure 13] We present the performance of two example convolutional neural networks using different feature engineering techniques. [Figure 14] A summary of model performance metrics for failure categories is presented. [Figure 15] We present an example of a failure segmentation image with mixed defects showing manually annotated defect labels and predicted defect categories. [Figure 16]A process flow chart is presented for training a good vs. bad classifier and for applying the trained classifier to classify images of a section of an image generation chip. [Figure 17] FIG. 1 is a simplified block diagram of a computer system that can be used to implement the disclosed techniques. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following discussion is presented to enable those skilled in the art to make and use the disclosed technology and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Thus, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. Introduction

[0011] The disclosed technology applies vision systems and image classification for evaluation of production genotyping and root cause failure analysis. Three separate approaches are described, the first involving eigenimages, the second based on area thresholding, and the third using deep learning models such as convolutional neural networks (or CNNs). Principal component analysis (PCA) and non-negative matrix factorization (NMF) are among the disclosed technologies. Other dimensionality reduction techniques that may be applied to images include independent component analysis, dictionary learning, sparse principal component analysis, factor analysis, mini-batch K-means. Variations of image decomposition and dimensionality reduction techniques may be used. For example, PCA may be implemented using singular value decomposition (SVD) or as kernel PCA. Outputs from these techniques are given as input to a classifier. The applied classifiers may include random forests, K-nearest neighbors (KNN), multinomial logistic regression, support vector machines (SVM), gradient boosting decision trees, naive Bayes, etc. Since a large body of labeled images is available, convolutional neural networks such as ResNet, VGG, ImageNet, etc. can also be used as presented below in the description of the third image processing technique.

[0012] The genotyping production process is vulnerable to both mechanical and chemical processing errors. Collected samples are extracted and distributed into sections and areas of the BeadChip, then chemically processed through multiple steps to generate a fluorescent image. The final fluorescent image, or even intermediate fluorescent images, can be analyzed to monitor production and perform failure analysis.

[0013] The majority of the production analysis is a success. The current failure analysis is understood to fit into five categories and a remaining failure category. The five failure categories are hybridization or hyb failure, spacer shift failure, offset failure, surface wear failure, and reagent flow failure. The remaining category is unhealthy patterns due to mixing effects, unidentified causes, and weak signals. More different causes can then be identified, especially since root cause analysis leads to improved production.

[0014] The first image processing techniques applied to quality control and failure analysis evolved from face recognition by eigenface analysis. From tens of thousands of labeled images, a linear criterion of 40 to 100 image components was identified. One approach to forming the eigencriterion was principal component analysis (PCA) followed by a ranking of the components according to a measure of explained variability. It was observed that 40 components explained the majority of the variability. Beyond 100 components, additional components appeared to reflect patterns of noise or natural variability in sample processing. The number of relevant components is expected to be affected by the image resolution. Here, resolution reduction was applied such that the image production chip segments were analyzed at a resolution of 180 x 80 pixels. This was sufficient resolution to distinguish successful from unsuccessful productions and then classify the root causes of failure among the six failure categories. A formal sensitivity analysis was not applied, but images of slightly lower resolution also worked, and images at 4 to 22 times this resolution could be processed in the same manner, but with increased computational expense. Each image analyzed by eigenimage analysis is represented as a weighted linear combination of reference images. Each weight for the set of ordered reference components is used as a feature to train a classifier. For example, in one embodiment, 96 weights of components of labeled images were used to train a random forest classifier. A random forest classifier with 200 trees and a depth of 20 performed well. Two tasks were performed by the random forest classifier: separation of successful and unsuccessful production images, and then root cause analysis of unsuccessful production images. This two-stage classification was chosen due to the predominance of successful production runs, but one-stage classification could also be used.

[0015] The second image processing technique applied involved thresholding of the image areas. The production image of the section of the imaging chip captures several physically separated areas. Structures that border the section and separate the separate physical areas of the section are visible in the production image. The thresholding strategy involves separating the active areas from the border structures and then distinguishing the separated areas. Optionally, the structures that separate the physical areas can also be filtered out from the image. At least the active areas are subjected to a thresholding of the emission. The thresholding determines how much of the active area produces the desired signal strength. Each active area is evaluated after thresholding for success or failure. Patterns of failures between areas and sections of the imaging chip can be further evaluated for root cause classification.

[0016] Processing of production images to detect failed production runs and determine root cause can be performed immediately during production, and the results are read from the imaging chip and quality judged faster than the image size in pixels can be reduced. This image processing can be done quickly, since reducing the image size in pixels to 1 / 20 of the original size on the side greatly reduces computational requirements, and direct processing of reduced resolution images does not require correlation of individual light-emitting pixels with individual probes within an area. Short turnaround root cause analysis can be used to correct upstream processes before chemicals and processing time are wasted.

[0017] A third image processing technique involves applying deep learning models such as convolutional neural networks (CNNs). ResNet (He et al., CVPR 2016, <<arxiv.org / abs / 1512.03385> >) and VGG (Simonyan et al., 2015, <<arxiv.org / abs / 1409.1556> Figure 1 (available at: http: / / www.ncbi.nlm.nih.gov / pubmed / 12345133) is an example of a convolutional neural network (CNN) used to identify and classify. We applied ResNet-18 and VGG-16 architectures of the respective models to detect failure images and classify the failure images into the respective failure categories. The CNN model parameters are pre-trained on the ImageNet dataset (Deng et al., "ImageNet: A large-scale hierarchical image database," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255), which contains about 14 million images. The pre-trained model is fine-tuned using labeled images of the partition of the image generation chip. About 75,000 labeled images of the partition are used to fine-tune the pre-trained CNN. The training data consists of normal images from successful process cycles and abnormal (or bad or failed) images of the partition from failed process cycles. The images from failed process cycles belong to the five failure categories presented above.

[0018] The ResNet-18 and VGG-16 CNN models may use square input images of size 224x224 pixels. In one embodiment, the partitions of the image generation chip are rectangular, such as 180x80 pixels, as described above. Partitions of larger image sizes may be used. The disclosed technology applies feature engineering to create training data using rectangular labeled images that may be smaller than the 224x224 pixel size images required as input to the CNN models. The technology may apply three feature engineering techniques to create the input dataset, including cropping, zero padding, and reflection padding.

[0019] In cropping, the center portion of the rectangular shaped segmented image of the image generation chip is cropped to a size of 224x224 pixels. In this case, the input image of the segment is larger than 224x224 pixels. In one embodiment, the input segmented image is 504x224 pixels in size. Labeled images of other sizes larger than 224x224 can be used to crop the square portion. The input image is cropped to match the input image size (224x224 pixels) required by the CNN model.

[0020] In zero padding, the labeled input image size is smaller than 224×224 pixels. For example, the input image may be 180×80 pixels. The smaller sized input labeled image (such as 180×80 pixels) is placed in an analysis frame of 224×224 pixels. The image may be placed at any position in the analysis frame. The pixels in the surrounding area in the analysis frame may be zero padded, in other words, the surrounding pixels are assigned a zero image intensity value. The larger sized analysis frame may then be given as an input to the CNN model.

[0021] In reflection padding, a smaller size labeled input image may be placed at the center of a larger size analysis frame. The labeled image is then reflected horizontally and vertically along the edges to fill the surrounding pixels in the larger size analysis frame (224x224 pixels). The reflected labeled image is given as an input to the CNN model. Reflection padding may produce favorable results because features in the input labeled image are copied to multiple locations in the larger size analysis frame.

[0022] The disclosed techniques may perform data augmentation to increase the size of the training data. Horizontal and vertical translation may be performed by placing a rectangular labeled input image of smaller size (J×K pixels) at multiple locations in a larger size (M×N pixels) analysis frame. In one embodiment, the rectangular labeled input image is 180×80 pixels in size and the larger size analysis frame is 224×224 pixels. Other sizes of labeled input image and analysis frame may be used. In one embodiment, the J×K images may be systematically translated horizontally, vertically, or diagonally in the M×N analysis frame to generate additional training data. In one embodiment, the J×K images may be randomly positioned at different locations in the M×N analysis frame to generate additional training data.

[0023] A two-step detection and classification process may be applied using two separately trained convolutional neural networks (CNNs). The first CNN is trained for the detection task, and the trained classifier may classify the images as normal or as depicting a process failure. The failed process cycle images may be fed into a second CNN trained to classify the images by the root cause of the process failure. In one embodiment, the system may classify the images into the five different types of failure types listed above. The two-step process may be combined into a one-step process using a CNN to classify the production section images as normal or as belonging to one of the failure categories. environment

[0024] We describe a system for early prediction of failure in a genotyping system. Genotyping is the process of determining differences in an individual's genetic makeup (genotype) by using biological assays to examine the individual's DNA sequence and compare it to a reference sequence. Genotyping allows researchers to search for genetic variants, such as single nucleotide polymorphisms (SNPs) and structural changes in DNA. The system is described with reference to FIG. 1, which shows an architecture level schematic diagram of the system according to an embodiment. Because FIG. 1 is an architecture diagram, certain details have been intentionally omitted to improve clarity of description. The discussion of FIG. 1 is organized as follows: First, the elements of the diagram are described, followed by their interconnections. Then, the use of the elements in the system is described in more detail.

[0025] 1 includes a system 100. This paragraph names the labeled portions of the system 100. The figure illustrates a genotyping device 111, a process cycle image database 115, a failure category label database 117, a labeled process cycle image database 138, a trained good vs. bad classifier 151, a reference database of eigenimages 168, a trained root cause classifier 171, a feature generator 185, and a network 155.

[0026] The disclosed technology is applied to a variety of genotyping devices 111, also referred to as genotyping scanners and genotyping platforms. A network 155 communicatively couples the genotyping devices 111, the process cycle image database 115, the failure category label database 117, the labeled process cycle image database 138, the trained good vs. bad classifier 151, the reference database of eigenimages 168, the trained root cause classifier 171, and the feature generator 185.

[0027] The genotyping instrument may include an Illumina BeadChip imaging system, such as the ISCAN™ system. The instrument may detect the fluorescence intensity of hundreds to millions of beads arranged in segments of mapped locations on the imaging chip. The genotyping instrument may include an instrument control computer that controls various aspects of the instrument, such as laser control, precision machine control, detection of excitation signals, image registration, image extraction, and data output. Genotyping instruments may be used in a wide variety of physical environments and operated by technicians of various skill levels. Sample preparation may take 2-3 days and may include manual and automated handling of samples.

[0028] We illustrate the process steps of an exemplary genotyping process 300 in FIG. 3. This exemplary genotyping process is referred to as Illumina's INFINIUM™ assay workflow. The process is designed to interrogate many SNPs with broad locus multiplexing. Using a single bead type and a dual-color (red and green) channel approach, the process scales to genotyping hundreds to millions of SNPs per sample. The process begins with DNA sample receipt and extraction. The process can operate with relatively low input sample, such as 200 ng, which can assay millions of SNP loci. The sample is amplified. The amplification process can take several hours to overnight to complete. The amplified sample undergoes controlled enzymatic fragmentation. This is followed by alcohol precipitation and resuspension. The imaging chip is prepared for hybridization in the capillary flow chamber. The sample is then applied to the prepared imaging chip and incubated overnight. During this overnight hybridization, the sample is annealed to locus-specific 50-mers covalently attached to up to millions of bead types, one bead type for each allele per SNP locus. Allele specificity is imparted by enzymatic base extension followed by fluorescent staining. A genotyping instrument or scanner (such as the ISCAN™ system) detects the fluorescence intensity of the beads and performs the genotyping.

[0029] In one example, the genotyping results are presented using a metric called "call rate." This metric represents the percentage of genotypes correctly scanned on the imaging chip. A separate call rate is reported for each section of the imaging chip. A threshold can be used to accept or reject the results. For example, a call rate of 98% or greater can be used to accept the genotyping results for a section. Different thresholds such as less than 98% or more than 98% can be used. If the call rate of a section falls below the threshold, the genotyping process is considered a failure. The genotyping process can take many days and is therefore expensive to repeat. Failure in the genotyping process can occur due to operational errors (such as mechanical or handling errors) or chemical processing errors.

[0030] The genotyping system may provide process cycle images of the sections of the imaging chip upon completion of the genotyping process along with the call rates of each of the sections. The disclosed technology may process these section images to classify whether the genotyping process is successful (good images of the sections) or not (bad or failed images of the sections). The disclosed technology may further process the bad or failed images to determine the failure category. Currently, the system may classify the failed images into one of six failure categories: hybridization or hyb failure, spacer shift failure, offset failure, surface wear failure, reagent flow failure, and overall unhealthy images due to mixing effects, unknown causes, weak signals, etc. More different causes may then be identified, especially as root cause analysis leads to improved production.

[0031] We now refer to FIG. 1 to provide a description of the remaining components of the system 100. The failure category labels for the six failure types may be stored in a failure category label database 117. A training data set of labeled process cycle images is stored in a database 138. The labeled training examples may consist of successful (good) and unsuccessful (bad) process cycle images. The unsuccessful process cycle images are labeled as belonging to one of the six failure categories listed above. In one embodiment, the training database 138 consists of at least 20,000 training examples. In another embodiment, the size of the training data set is increased to 75,000 training examples using feature engineering techniques. The size of the training database may increase as more labeled image data is collected from the laboratory using genotyping equipment.

[0032] The disclosed technology includes three independent image processing techniques for extracting features from the process cycle images. A feature generator 185 may be used to apply one of the three techniques to extract features from the process cycle images for input to the machine learning model. The first image processing technique evolves from face recognition by eigenface analysis. A relatively small number of linear criteria, such as 40-100 or more image components, are identified from tens of thousands of labeled images. One approach to forming the eigencriteria is principal component analysis (PCA). The production cycle images are represented as a weighted linear combination of the criteria images for input to the classifier. For example, in one implementation, 96 weights of the components of the labeled images are used to train the classifier. The eigenimage criteria may be stored in the database 168.

[0033] A second image processing technique for extracting features involves thresholding of the segment image. A production image of a segment of the imaging chip captures several physically separated areas. Structures that border the segment and separate the distinct physical areas of the segment are visible in the production image. The thresholding technique determines how much of the active area produces a desired signal strength. Output from the thresholding technique can be provided as input to a classifier to distinguish good images from bad images. Patterns of failures among areas and segments of the imaging chip can be further evaluated for root cause analysis.

[0034] The third image processing technique includes various feature engineering techniques to prepare an image for input to a deep learning model. The segmented image of the image generation chip is rectangular. The image generation chip may have 12, 24, 48, or 96 segments arranged in two or more columns. The convolutional neural network (CNN) applied by the disclosed technology requires a square input image. Thus, the system includes logic to map the rectangular (J×K pixels) segmented image into a square (M×N pixels) analysis frame.

[0035] The system may apply one or more of the following feature engineering techniques: The system may apply zero padding to fill pixels surrounding the segmented image in a larger square analysis frame; The system may crop a center portion of the square dimensional segmented image and fill the analysis frame with the cropped segmented image, where the segmented image is of a larger dimension than the analysis frame; When the labeled segmented image is smaller than the analysis frame, the system may position the labeled input image in the analysis frame; The system may use horizontal and vertical reflections along the edges of the input segmented image to fill the larger size analysis frame; By using translations where the labeled input image is positioned at multiple locations in the analysis frame, the system may augment the labeled training data.

[0036] The image features of the production images generated by the feature generator 185 are provided as input to the trained classifiers 151 and 171. Two types of classifiers are trained: a good vs. bad classifier may predict successful and unsuccessful production images; and a root cause analysis classifier may predict failure categories for unsuccessful images. In one embodiment, the classifier used by the disclosed technology includes a random forest classifier. Other examples of classifiers that may be applied include K-nearest neighbors (KNN), multinomial logistic regression, and support vector machines. In another embodiment of the disclosed technology, a convolutional neural network (CNN) is applied to identify and classify images of sections of the image generation chip.

[0037] Completing the description of FIG. 1, all of the components of the system 100 described above are communicatively coupled to a network 155. The actual communication paths may be point-to-point over public and / or private networks. Communications may occur across a variety of networks, such as private networks, VPNs, MPLS circuits, or the Internet, and may use appropriate application programming interfaces (APIs) and data exchange formats, such as Representational State Transfer (REST), JavaScript Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), Java Message Service (JMS), and / or Java Platform Module System. All of the communications may be encrypted. Communications are typically over networks such as local area networks (LANs), wide area networks (WANs), telephone networks (Public Switched Telephone Network (PSTN), Session Initiation Protocol (SIP), wireless networks, point-to-point networks, star networks, token ring networks, hub networks, the Internet, including mobile Internet via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX. The engine or system components of FIG. 1 are implemented by software running on various types of computing devices. Exemplary devices are workstations, servers, computing clusters, blade servers, and server farms. Additionally, various authorization and authentication technologies may be used to secure communications, such as username / password, Open Authorization (OAuth), Kerberos, Secured, digital certificates, etc. Feature Generator - System Component

[0038] FIG. 2 is a high-level block diagram of the components of the feature generator 185. These components are computer implemented using a variety of different computer systems, as presented below in the description of FIG. 16. The illustrated components may be merged or further separated when implemented. The feature generator 185 consists of three high-level components that implement three image processing techniques: a principal component analysis or PCA-based feature generator 235, an image segmentation-based feature generator 255, and a training data generator for CNN 275. The PCA-based feature generator consists of an image scaler 237 and an eigenimage reference creator 239. The image segmentation-based feature generator 255 consists of an image transformer 257 and an intensity extractor 259. The training data generator for CNN 275 comprises an image cropper 277, an image translator 279, and an image reflector 281. In the following sections, we present further details of the implementation of these components. PCA-based feature generator

[0039] The first image processing technique evolved from face recognition by eigenface analysis. One approach to form an eigenbase is principal component analysis (PCA). A PCA-based feature generator 235 applies PCA to the resized process image. An image scaler component 237 resizes the process cycle image. The scaling reduces the size of the process image so that it can be processed in a computationally efficient manner by the eigenimage base creator component 239. We present details of these components in the following sections. Image Scaler

[0040] Higher resolution images obtained from a genotyping device or scanner may require more computational resources to process. The images obtained from the genotyping scanner are resized by an image scaler 237 so that the images of the sections of the image generating chip are analyzed at a reduced resolution, such as 180x80 pixels. We refer to scaling (or rescaling) as resampling the image throughout this specification. Resampling changes the number of pixels in the image displayed. Increasing the number of pixels in the initial image increases the image size and is referred to as upsampling. Reducing the number of pixels in the initial image decreases the image size and is referred to as downsampling. In one embodiment, the images of the sections obtained from the scanner have a resolution of 3600x1600 pixels. In another embodiment, the images of the sections obtained from the scanner have a resolution of 3850x1600 pixels. These original images are downsampled to reduce the size of the images in pixels by a factor of 1 / 20 per side from the original resolution. This is sufficient resolution to distinguish successful production images from unsuccessful production images and then classify the root cause of failure among the six failure categories. The image may be downsampled to 1 / 4 to 1 / 40 the number of pixels per side at the original resolution and processed in the same manner. In another embodiment, the image may be downsampled to 1 / 2 to 1 / 50 the number of pixels per side at the original resolution and processed in the same manner. Exemplary techniques for resampling high resolution images are presented below.

[0041] The disclosed techniques may apply various interpolation techniques to reduce the size of the produced image. In one embodiment, bilinear interpolation is used to reduce the size of the partitioned image. Linear interpolation is a method of curve fitting using linear polynomials to construct new data points within a discrete set of known data points. Bilinear interpolation is an extension of linear interpolation to interpolate functions of two variables (e.g., x and y) on a two-dimensional grid. Bilinear interpolation is performed using linear interpolation, first in one direction and then again in the second direction. Each step is linear in the sampled values ​​and positions, but the interpolation as a whole is not linear, but rather quadratic in the sample locations. Other interpolation techniques may also be used to reduce the size of the partitioned image (rescaling), such as neighborhood interpolation and resampling using pixel area relationships. Intrinsic image fiducial generator

[0042] The first image processing techniques applied to segmented images to generate input features for a classifier evolved from face recognition by eigenface analysis. From tens of thousands of labeled images, linear criteria of 40-100 image components are identified. One approach to forming criteria of eigenimages is Principal Component Analysis (PCA). A set B of elements (vectors) in a vector space V is called a criterion if every element of V can be written in a unique manner as a linear combination of the elements of B. Equivalently, B is a criterion if its elements are linearly independent and every element of V is a linear combination of the elements of B. A vector space may have several criteria. However, all criteria have the same number of elements, which is called the dimension of the vector space. In our technique, the criteria of the vector space are eigenimages.

[0043] PCA is often used to reduce the dimension of a d-dimensional dataset by projecting it onto a k-dimensional subspace, where k < d. For example, the resized labeled images in our inventors' training database represent vectors in a d = 14,400 dimensional space (180 x 80 pixels). In other words, an image is a point in a 14,400 dimensional space. The eigen-space based approach approximates the image vectors with lower dimensional feature vectors. The main assumption behind this technique is that the image space given by the feature vectors has a lower dimension than the image space given by the number of pixels in the image, and that image recognition can be performed in this reduced space. The sectional images of the image generation chip, which are similar in overall configuration, will not be randomly distributed in this large space and can thus be explained by a relatively low dimensional subspace. The PCA technique finds the vectors that best explain the distribution of the sectional images within the entire image space. These vectors define a subspace of the images, also referred to as the "image space". In our inventors' embodiments, each vector represents an 180 x 80 pixel image and is a linear combination of the images in the training data. In the following text, our inventors present details on how principal component analysis (PCA) can be used to create a criterion for eigen-images.

[0044] The PCA-based analysis of labeled training images can consist of the following five steps. Step 1: Accessing multi-dimensional correlation data

[0045] The first step in the application of PCA is to access the high-dimensional data. In one example, a PCA-based feature generator used 20,000 labeled images as training data. Each image was resized to 180 x 80 pixel resolution and represented as a point in a 14,400 dimensional space with one dimension per pixel. This technique can handle images with higher or lower resolution than specified above. The size of the training dataset is expected to increase as our inventors collect more labeled images from the laboratory. Step 2: Standardize the data

[0046] Standardization (or Z-score normalization) is the process of rescaling features so that they have the characteristics of a Gaussian distribution with mean equal to 0, i.e. μ=0, and standard deviation from the mean equal to 1, i.e. σ=1. Standardization is performed to construct features that have similar ranges to each other. The standard score of an image can be calculated by subtracting the mean(image) from the image and dividing the result by the standard deviation. Once PCA has obtained a feature subspace that maximizes the variance along an axis, it helps to standardize the data so that the data is centered on the axis. Step 3: Calculate the covariance matrix

[0047] A covariance matrix is ​​a dxd matrix in d-dimensional space, where each element represents the covariance between two features. The covariance of two features measures their tendency to vary together. The variation is the average of the squared deviations of the features from their mean. The covariance is the average of the products of the deviations of the features from their mean. Consider features k and j. Let {x(1,j),x(2,j),...,x(i,j)} be the set of i examples of feature j, and {x(1,k),x(2,k),...,x(i,k)} be the set of i examples of feature k. Similarly,

number

number

number

number

number

[0048] The mean vector is a d-dimensional vector, where each value in the vector represents a sample mean of a feature column in the training dataset. jk is "-(σ ij )(σ ik )"; that is, "+(σ ij )(σ ik ) can vary between linear correlation to inverse linear correlation. When there is no dependency between two features, σ jk The value of is zero. Step 4: Compute the eigenvectors and eigenvalues

[0049] The eigenvectors and eigenvalues ​​of the covariance matrix represent the core of PCA. The eigenvectors (or principal components) determine the orientation of the new feature space, and the eigenvalues ​​determine their magnitude. In other words, the eigenvalues ​​explain the variance of the data along the axes of the new feature space. Eigenvalue decomposition is a method of matrix decomposition by representing a matrix using its eigenvectors and eigenvalues. An eigenvector is defined as a vector that changes only by a scalar when a linear transformation is applied to the vector. If A is a matrix that represents a linear transformation, then v is an eigenvector and λ is the corresponding eigenvalue, which can be expressed as Av=λv. A square matrix can have as many eigenvectors because it has dimensions. If we represent all the eigenvectors as columns of a matrix V and the corresponding eigenvalues ​​as entries of a diagonal matrix L, then the above equation can be expressed as AV=VL. For a covariance matrix, all the eigenvectors are orthogonal to each other and are principal components of the new feature space. Step 5: Use the explained variance to select a criterion for the eigenimages

[0050] The above steps may result in 14,400 principal components for our implementation, equal to the dimension of the feature space. An eigenpair consists of an eigenvector and a scalar eigenvalue. We may filter the eigenpairs based on the eigenvalue and create a measure of the eigenimage using a metric called "explained variance". The explained variance indicates how much information (or variance) can be attributed to each of the principal components. We may plot the results of the explained measurements on a two-dimensional graph. The filtered principal components are represented along the x-axis. A graph showing the cumulative explained variance may be plotted. The first m components that represent the major part of the variance may be selected.

[0051] In our implementation, the first 40 components represent a high percentage of explained variance, and therefore we selected the first 40 principal components to form our new feature space baseline. In other implementations, 25-100 principal components or more than 100 principal components, up to 256 or 512 principal components, may be selected to create the eigenimage baseline. Each production image analyzed by eigenimage analysis is represented as a weighted linear combination of the baseline images. Each weight of the ordered set of baseline components is used as a feature to train the classifier. For example, in one implementation, 96 weights for the components of the labeled images were used to train the classifier.

[0052] The disclosed techniques may use other image decomposition and dimensionality reduction techniques. For example, non-negative matrix factorization (NMF), which learns a part-based representation of an image, compared to PCA, which learns a complete representation of the image. Unlike PCA, NMF learns to represent an image with a set of reference images that resemble parts of the image. NMF factors matrix X into two matrices W and H, with the property that all three matrices have no negative elements. We assume that matrix X is set so that there are n data points (such as images of a partition on an image generation chip), each with p dimensions (e.g., 14,400). Thus, matrix X has p rows and n columns. We wish to reduce the p dimensions to r dimensions, in other words, to create a rank-r approximation. NMF approximates matrix X as the product of two matrices: W (p rows and r columns) and H (r rows and n columns).

[0053] The interpretation of matrix W is that each column is a reference element. By reference elements we average several components present in the n original data points (or images). These are the components from which we can reconstruct an approximation to all of the original data points or images. The interpretation of matrix H is that each column gives the coordinates of a data point in the reference matrix W. In other words, it shows us how to reconstruct an approximation to the original data points from a linear combination of the components of matrix W. In the case of face images, the reference elements of matrix W (or reference images) may include features such as eyes, nose, lips, etc. The columns of matrix H indicate which features are present in which images. Image Segmentation Based Feature Generator

[0054] A second image processing technique for extracting features from the process cycle images is based on thresholding of image areas. The image segmentation based feature generator 255 first segments the image of the section of the imaging chip using the image segmenter 257, and then applies thresholding by extracting the intensity of the active area or region of interest of the section image. The thresholding determines how much of the active area produces the desired signal strength.

[0055] The image generation chip may consist of multiple partitions, such as 24, 48, 96 or more, organized into rows and columns. This design allows for the processing of multiple samples in one process cycle, since many samples (one per partition) may be processed in parallel. The partitions are physically separated from other partitions so that the samples are not mixed with each other. In addition, the partitions may be organized into multiple parallel regions, called "slots." Hence, the structure of the partitions and the slot borders are visible in the process cycle images from the genotyping scanner. We present below the details of two components of the image segmentation-based feature generator 255 that may implement the technique of transforming the partition image for the extraction of image features. Image Converter

[0056] The image converter 257 applies a series of image transformation techniques to prepare the segmented image for intensity extraction from the regions of interest. In one embodiment, this process of image transformation and intensity extraction is performed by some or all of the following five steps: Image transformation converts the grayscale image of the segment into a binary image consisting of black and light pixels. The average intensity values ​​of the active regions of the grayscale and binary images are given as input features to a classifier for classifying the image as a healthy (good) or unhealthy (bad) image. In the following text, we present details of the image transformation steps, including applying thresholding to convert the grayscale image into a binary image. Process steps include applying filters to remove noise.

[0057] The first step in the image transformation process is to apply a bilateral filter to the process cycle image of the segment. Bilateral filtering is a technique that smooths an image while preserving edges. It replaces the intensity of each pixel with a weighted average of the intensity values ​​from its neighboring pixels. Each neighborhood is weighted by a spatial component that penalizes distant pixels, and a range component that penalizes pixels with different intensities. The combination of both components ensures that only nearby similar pixels contribute to the final result. Thus, bilateral filtering is an efficient way to smooth an image while preserving its discontinuities or edges. Other filters, such as median filters and anisotropic diffusion, may also be used.

[0058] The second step of the image transformation involves applying thresholding to output the image from step 1. In one embodiment, we apply Otsu's method (Otsu, N., 1979, "A threshold selection method from gray-level histograms", IEEE Transactions on Systems, Man, and Cybernetics, Volume 9, Issue 1), which uses intensity histograms and searches for a threshold to maximize the weighted sum of the grayscale variance between pixels assigned to dark and light intensity classes. Otsu's method tries to maximize the variance between classes. The basic idea is that well-thresholded classes should differ with respect to the intensity values ​​of their pixels, and conversely, the best threshold is the threshold that gives the best separation between classes with respect to their intensity values. In addition, Otsu's method has the property that it is entirely based on calculations performed on the histogram of the image, which is an easily obtainable one-dimensional array. For further details on Otsu's method, see Section 10.3.3 of Gonzalez and Woods, "Digital Image Processing", 3 rd Please refer to the Edition.

[0059] The third step of the image transformation is the application of a noise-reducing Gaussian blur filter to remove speckle-like noise. Noise can contaminate process cycle images with small speckles. Gaussian filtering is a weighted average of the intensities of neighboring locations with weights that decrease with the spatial distance to the central location.

[0060] The fourth step of image transformation involves image morphology operations. The binary output image from the third step is processed by a morphology transformation to fill holes in the image. A hole can be defined as a background region (represented by 0s) surrounded by a connected border of foreground pixels (represented by 1s). The two basic image morphology operations are "erosion" and "dilation". In an erosion operation, a kernel slides (or moves) over the binary image. A pixel (1 or 0) in the binary image is considered to be 1 if all pixels under the kernel are 1s. Otherwise, it is eroded (changed to 0). The erosion operation is useful for removing isolated 1s in a binary image. However, erosion also shrinks clusters of 1s by shrinking their edges. The dilation operation is the opposite of erosion. In this operation, as the kernel slides over the binary image, the values ​​of all pixels in the binary image area overlapped by the kernel are changed to 1 if the value of at least one pixel under the kernel is 1. When a dilation operation is applied to a binary image followed by an erosion operation, the effect is to close small holes in the clusters of 1s (represented by 0s in the image). The output from this step is provided as input to an intensity extractor component 259, which implements the fifth step of this image transformation technique. intensity extractor

[0061] The intensity extractor 259 divides the segmented image into active regions or segments by filtering out structures at the boundaries of the segments and slots. The intensity extractor may apply different segmentations to divide the segmented image into 8 to 17 active areas. Examples of areas in the segmented image include the four slots, the four corners, the four edges between the corners, and the various vertical and horizontal lines at the boundaries of the segments and slots. Areas corresponding to known structures that separate the active areas are then removed from the image. The remaining image portions of the active areas are processed by the intensity extractor 259. Intensity values ​​are extracted and averaged for each active area in the transformed image and the corresponding untransformed image. For example, if intensity values ​​are extracted from 17 active areas in the transformed image, the intensity extractor also extracts intensity values ​​from the same 17 active areas in the untransformed image. Thus, a total of 34 features are extracted per segmented image.

[0062] For binary images, the average intensity of the active area can be between 1 and 0. For example, consider that the intensity of a black pixel is 0 and the intensity of a light (or blank) pixel is 1. If all pixels in the active area are black, the average intensity of the active area will be 0. Similarly, if all pixels in the active area are light, the intensity of that area will be 1. The active area of ​​a healthy image will appear as blank or light in the binary image, while the black pixels represent an unhealthy image. The average intensity of the corresponding active area in the grayscale image is also extracted. The average intensity of the active areas from both the grayscale image and the converted binary image is given as input to a good vs. bad classifier. In one embodiment, the classification confidence score from the classifier is compared to a threshold to classify the image as a healthy (or good or success) image or an unhealthy (or bad or failure) image. An example threshold is 80%. A higher value of the threshold may result in more images being classified as unhealthy. A training data generator for convolutional neural networks (CNNs)

[0063] A third image processing technique may use cropping, translation, and reflection to prepare input images for training a convolutional neural network (CNN). The segmented images are rectangular, and ResNet (He et al., CVPR 2016, <<arxiv.org / abs / 1512.03385> >) and VGG (Simonyan et al., 2015, <<arxiv.org / abs / 1409.1556> CNNs such as the CNN 275 available at http: / / www.CNN.org / (available at: http: / / www.CNN.org / ) use square input images of size 224 x 224 pixels. The training data generator 275 for CNNs includes an image cropper 277, an image translator 279, and an image reflector 281 to prepare input images for the CNN. The image translator 279 may also augment the labeled input images to increase the size of the training data. Image Cropper

[0064] The image cropper 277 includes logic for cropping the image of the image generating chip segment to match the input image size required by the convolutional neural network (CNN). In one embodiment, the high resolution image obtained from the genotyping device has dimensions of 3600×1600 pixels. As explained above, high resolution images can require more computational resources to process, and therefore the disclosed technique downsamples the image to reduce its size. The size of the image is reduced by 1 / 4 to 1 / 40 from the pixel size of the original image, thus resulting in an image with a size ranging from 964×400 pixels to 90×40 pixels. In one example, the segment image is downsampled by a factor of 1 / 20 from the original image size of 3600×1600 pixels, resulting in a size of 180×80 pixels (J×K image). The downsampled image can be fed to the image cropper to prepare the input as required by the convolutional neural network (CNN).

[0065] The image cropper may crop various portions of the rectangular segmented image to prepare a square input image for the CNN. For example, a central portion of a rectangular shaped segmented image of an image generation chip may be cropped and placed within an analysis frame of 224×224 pixels. If the input image of the segment is larger than 224×224 pixels, such as 504×224 pixels, the image cropper may crop 224×224 pixels from the image to fill the M×N analysis frame. The image cropper may crop portions smaller than 224×224 pixels for placement within the M×N analysis frame. Image Translator

[0066] The image translator 279 contains logic to position the labeled input image of the image generation chip's segment at multiple locations in the analysis frame. "Translation" may refer to moving the shape or segment image in this case without rotating or flipping. Translation may also refer to sliding the segment image within the analysis frame. After translation, the segment image looks the same and has the same dimensions, but is positioned at a different location in the analysis frame. The image translator contains logic to move or slide the segment image horizontally, vertically, or diagonally at different locations in the analysis frame.

[0067] The analysis frame may be larger than the size of the labeled input image. In one case, the analysis frame is a square with a size of 224×224 pixels. Analysis frames of other sizes and shapes may be used. The labeled input image of the segments is positioned at multiple locations within the analysis frame. When the image size is 180×80 pixels and the analysis frame is 224×224 pixels, the image may be translated horizontally and translated vertically. In another example, when the image size is 224×100 pixels, the image may be translated only horizontally within the 224×224 pixel analysis frame. In another example, when the image size is 200×90 pixels, the image may be translated horizontally and vertically within the 224×224 pixel analysis frame.

[0068] Pixels in the analysis frames surrounding the labeled input image may be zero-padded, where pixels surrounding the labeled input image are assigned a "0" value for their respective intensities. Translation of one input labeled image may result in multiple analysis frames containing the input image at different locations within the frame. This process may augment the training data and therefore increase the number of training examples. Image Reflector

[0069] The image reflector 281 includes logic to horizontally and vertically reflect the smaller sized input labeled image positioned in the larger sized analysis frame. Reflection refers to the image of the object, or in this case the segmented image, as seen in a mirror. The smaller sized labeled input image may be placed in the center of the larger sized analysis frame. The image reflector 281 includes logic to horizontally and vertically reflect the segmented image along the edges to fill the surrounding pixels in the larger sized analysis frame (e.g., 224x224 pixels). Reflection padding may increase the probability of detecting failed process images, since the image or parts of the image are copied to multiple locations in the analysis frame. Process Cycle Image

[0070] Here we present examples of successful and unsuccessful production images of segments on an imaging chip. FIG. 4 is an example 400 of a production image of 24 segments on an imaging chip. The segments are arranged in 12 rows and 2 columns. Each segment has 4 slots. Example 400 shows a segment image of a successful production cycle. Imaging chips with other configurations of segments, such as including 48, 96, or more segments, can also be used. In the following figures, we present examples of segment images of unsuccessful production cycles. The production process is vulnerable to both operation and chemical processing errors. Operation defects can be caused due to mechanical or sample handling problems. Chemical processing errors can be caused by problems in the sample or chemical processing of the sample. The disclosed technique attempts to classify bad process image cycles that occur due to both operation and chemical processing errors.

[0071] FIG. 5A shows an example of a section image 510 from an unsuccessful production cycle. The image of section 512 in the second column and seventh row of the imaging chip in FIG. 5A is dark in the bottom half and slightly light in the top. The cause of this failure is related to the hybridization process. Therefore, the failed image of the section is labeled as a "Hyb" failure. Hybridization failure can also occur due to failure of the robot handling the sample during the sample preparation process on the imaging chip. The call rate of this section is "97.545", which is below the 98 percent pass threshold. In some cases, the call rate for the section from the genotyping instrument can exceed the pass threshold, and then the section image can also fail due to hybridization errors.

[0072] In example 510, it can be noted that the image of section 514 at row 11 and column 2 has a dark area on the right wall. This may also indicate a processing problem, but the overall call rate of this image is above the pass threshold and is not labeled as a failed image. There is sufficient redundancy of samples on the section such that small areas of the section with apparent failures can be ignored and may not cause errors in the results. For example, in one case, the scanner reads fluorescence from about 700K probes on a section with a redundancy of 10. Therefore, the call rate is based on the reading of about 7 million probes. We present a further example of hybridization failure in example 515 of FIG. 5B. The four sections on the imaging chip with dashed borders show poorly produced images of the sections due to hybridization failures. Note that the call rate values ​​for these four sections are above the pass threshold, but the images of these sections are labeled as failed due to hybridization errors.

[0073] FIG. 5C presents an example 520 of nine section images showing unsuccessful processing due to spacer shift failure. When the samples are prepared on the sections on the imaging chip, dark markers are placed around the section. The spacers separate the samples in each section from other samples in adjacent sections. If the markers are not placed correctly, they may block part of the image signal. Offset errors can occur across multiple adjacent sections, as shown in FIG. 5C. The top parts of the nine sections in this figure appear dark. The dark areas at the top of the sections increase as you move from left to right. Space shift issues are caused by incorrect placement of markers by the lab technician during preparation of the samples on the imaging chip, and are therefore operational errors. FIG. 5D presents three more examples of failed images of sections due to spacer shift failure. Box 525 shows five section images with spacer shift failures when the top parts of the section images are dark with increasing width from top right to top left. Box 527 shows two section images showing unsuccessful processing due to spacer shift issues at the bottom parts of the sections. Similarly, box 529 shows an image of two segments that failed due to space shift problems.

[0074] 5E shows an example of a failed image of a section due to an unsuccessful process caused by an offset failure. In an offset failure, the image of the section on the imaging chip is shifted to one side. For example, in example 530, all the section images on the imaging chip are shifted toward the left side, so that the dark outer border of the imaging chip on the left edge is cut off from the image. An offset failure can be caused by a scanning error, such as a scanner misalignment or misplacement of the imaging chip on the chip carrier.

[0075] FIG. 5F shows an example of a failed segment image due to surface wear failure. Surface wear is caused by scratches on the surface of a segment of an imaging chip during the manufacturing process or during preparation of samples on the segment. The scratches are visible as lines on the image of the segment, as shown in example 535. Note that the call rate values ​​are above the passing threshold for the three segments in the dashed box on the left, yet they are labeled as failed due to surface wear failure.

[0076] FIG. 5G is an example 540 of a failed section image due to a reagent flow failure. The 10 section images in box 542 are labeled as failed images due to reagent flow failure. The section images failed due to an unsuccessful process caused by improper reagent flow. During the genotyping process, the reagent is introduced into the imaging chip from one side. The reagent flows from one end of the imaging chip to the other end, completely covering all sections. Sometimes there is a problem with the reagent flow and it may not propagate evenly to all sections. In this case, when a sufficient amount of reagent does not cover the section, the reagent may dry out. An improper reagent flow may reduce the intensity of the emission signal from some sections because the fluorescent dye may not be distributed evenly across all sections, thus affecting the image quality. The failed image due to reagent flow failure may appear darker in color compared to the section image representing a successful process cycle. FIG. 5H shows a further example of a failed section image due to reagent flow failure in example 545. A reagent flow failure may affect multiple adjacent sections within a region of the imaging chip, as shown in FIG. 5G and FIG. 5H.

[0077] FIG. 5I presents an example of a failed image due to unknown reasons. The failed segment image is labeled as "unhealthy." Failed images in the unhealthy class of failures may be due to mixed or unidentified causes and weak signals. Segment image example 550 also shows an example of a spacer failure for the top left segment of the imaging chip. The image segment on the top left location (row 1 and column 2) is labeled as a spacer failure. It can be seen that the top portion of the failed segment image is dark. The portion of the top dark region increases from the right corner to the left corner of the segment image. Principal Component Analysis-Based Feature Generation

[0078] Here, we present examples of eigenimages, referred to as eigenfaces in the field of face recognition. From tens of thousands of labeled images, a linear measure of 40-100 image components is identified. Figure 6A presents an example of 96 eigenimages obtained by applying Principal Component Analysis (PCA). The 96 eigenimages are selected based on a ranking of the components according to the explained variability measures, as presented above. Figure 6B shows the top 40 ranked eigenimages from the 96 eigenimages of Figure 6A. In one implementation, it was observed that 40 components explained the majority of the variability. The additional components selected appeared to reflect patterns of noise or natural variability in sample processing.

[0079] Here we describe the creation of dimensionality reduction and eigenimage criteria using PCA. The first step is to reduce the resolution of the segmented image and prepare the reduced image for input to PCA. Figure 7A presents an example 710 illustrating dimensionality reduction. A segmented image with a resolution of 3600x1600 pixels is downsampled by a factor of 1 / 20 per side, resulting in a segmented image with a reduced size of 180x80 pixels. The downsampled segmented images are flattened. The resulting flattened segmented images are each a one-dimensional array, i.e., 14,400x1 pixels.

[0080] As an alternative to the 1 / 20 size at each edge, other dimensionality reductions may be used. The principle is that much less information, much less pixel density, is needed to assess the overall health of the flow cell than calling out individual clusters or balls within the flow cell. Thus, 1 / 2 倍 ~1 / 50 倍 Range or 1 / 4 倍 ~1 / 40 倍 A range of reductions can be used, with more extreme resolution reductions expected as the initial resolution of the segmented image increases. In particular, when transfer learning may be applied to leverage the pre-training of the deep learning framework, it is desirable to select a resolution reduction that fits the captured segmentation to the input aperture of the deep learning framework. A 1 / 20 reduction to 180x80 pixels with both horizontal and vertical reflections is preferred. 倍 Downsampling of 224x224 pixels proved to be a good choice with an input aperture of 224x224 pixels. Other reductions are evident for different segmented images and different deep learning input apertures.

[0081] In an alternative embodiment, PCA is applied to the downsampled image as described in the previous application. The flattened segmented image is standardized as described above, thus resulting in the standardized, flattened and rescaled segmented image shown in example 740 of FIG. 7B, which is given as input to the PCA. Thus, PCA generates 14,400 principal components or eigenimages. Each input image is a vector in a 14,400 dimensional space. We then use the explained variance to rank the principal components or eigenimages and create a criterion, for example a criterion of 40-100 components. The components form a criterion in a linear space. Image Segmentation-Based Feature Generation

[0082] A second image processing technique for generating features from an image of a segment involves thresholding of an image area or segment. FIG. 8A shows an example 810 of a segmentation-based feature applied to a segment image. An example 812 of the example 810 is a production image of a segment of an image generation chip. We apply an image transform to convert this grayscale production image 812 to generate a companion binary image 814. In one embodiment, some or all of the five steps presented above with reference to the image converter 257 may be implemented to convert a grayscale image to a binary image. Black pixels in the binary image 814 indicate unhealthy or bad image pixels, while light pixels indicate healthy or good image pixels.

[0083] In FIG. 8A, the right-hand illustration 816 is an exemplary schematic diagram of a segment showing various areas of the segment and the borders or lines around these areas. The areas where the intensity of the fluorescent signal is recorded are also referred to as active areas or regions of interest. For example, the segment schematic 816 shows four slot active areas that run parallel to each other from top to bottom. Areas of the segment image that are not active areas are filtered out of the image. For example, the slot border areas that are separated from each other by vertical lines that indicate the slot borders or boundaries. Similarly, the borders on the four sides of the segment image may be filtered out. Segmentation techniques may divide the segment image into 4 to 20 or more segments or active areas. Thresholding determines how much of the active area produces the desired signal intensity.

[0084] The number of active areas determines the number of features generated per image. For example, if the segmented image is segmented into 8 active areas, the image intensity from the 8 active areas of the transformed image and the image intensity values ​​from the same 8 active areas of the original segmented image before transformation are provided as inputs to the classifier. Thus, in this example, a total of 16 features are provided to the classifier per segmented image. The average intensity of the signal intensity from the active areas may be used as input to the classifier. For example, if the segmented image is segmented into 8 active areas, the average intensity of these 8 active areas is calculated for both the grayscale and binary images. These 16 intensity values ​​are provided as inputs to the classifier to classify the segmented image as good versus bad. Other segmentation schemes that divide into fewer or more segments, such as 4, 12, 17 or more segments per image may be used. When provided as inputs to the random forest classifier, a subset of features is randomly selected for each decision tree. The decision tree votes the image as healthy (or successful) or unhealthy (or failed). The majority of votes in the random forest are used to classify the image. In one embodiment, the value of the number of trees in the random forest classifier is in the range of 200 to 500 and the value of the model depth is in the range of 5 to 40. Patterns of failures across areas and segments of the imaging chip can be further evaluated for root cause classification.

[0085] Figure 8B presents an example 820 of three pairs of segmented images 822, 824, and 826. The left image of each pair is a pre-transformed grayscale segmented image, and the right image of each pair is a processed binary image after applying image transformation as described with reference to Figure 2. The first image pair 822 is the product image of a successful genotyping process. The second image pair 824 is the product image of a failed production image due to hybridization (or hyb) failure. The third image pair 826 is a failed image due to a surface wear problem. One-vs-other (OvR) classification

[0086] FIG. 9 presents an illustrated example 900 of running a one-vs-others classifier. The graphs show an example of running a one-vs-others (OvR) classifier on a dataset consisting of samples belonging to three classes (squares, circles, and triangles), as shown in the graph 931 on the left. As shown in the graph 915 on the top right, the first hyperplane 916 shows the hyperplane determination of the square class as ground truth. The hyperplane 916 partitions the data points of the square class from the rest of the data points (circles and triangles). Similarly, graphs 936 and 955 partition the data points of the circle and triangle classes, respectively, to the other classes of data via hyperplanes 937 and 956, respectively. The position of the hyperplane is determined by a weight vector. The training algorithm attempts to maximize the margin of the hyperplane from the ground truth classes for generalization, which may result in misclassification of one or more data points. We apply OvR classification to distinguish segmented images from a process cycle that belong to a good class from images that belong to multiple bad (or failed) classes. Random Forest Classifier

[0087] The disclosed technology may apply various classifiers to distinguish images from good or healthy images from bad or unhealthy images that belong to multiple failure classes. The applied classifiers include Random Forest, K-Nearest Neighbors, Multinomial Logistic Regression, and Support Vector Machine. We present an embodiment of the disclosed technology using the Random Forest classifier as an example.

[0088] Random forest classifiers (also called random decision forests) are an ensemble machine learning technique. Ensemble techniques or algorithms combine more than one technique of the same or different kinds to classify objects. A random forest classifier consists of multiple decision trees operating as an ensemble. Each individual decision tree in the random forest acts as a base classifier and outputs a class prediction. The class with the most votes becomes the prediction of the random forest model. The basic idea behind random forests is that a large number of relatively uncorrelated models (decision trees) operating as a committee will favor one of the individual constituent models.

[0089] The disclosed technique applies a random forest classifier in a two-stage classification process. A first trained random forest classifier performs the task of separating unsuccessful production images from successful production images. A second trained random forest classifier performs the task of root cause analysis of unsuccessful production images by predicting the failure class of the unsuccessful images. This two-stage classification was chosen due to the predominance of successful production runs, but one-stage classification can also be used. Another reason for choosing a two-stage approach is that it allows us to control the sensitivity threshold for classifying images as healthy or successful production images versus unhealthy or failed production images. We can increase the threshold in the first stage classification, thus causing the classifier to classify more production images as failed images. These failed images are then processed by the second stage classifier for root cause analysis by identifying the failure class. Training a Random Forest Classifier

[0090] FIG. 10A illustrates the training of two random forest classifiers as shown in example 1000. The training data consists of input features for labeled process cycle images stored in training database 138 as shown in FIG. 1. In one exemplary training of the classifiers, we used 20,000 partitions of labeled production images. The labeled images include both good images from successful production cycles and failed images from failed production cycles. The size of the training database 138 will grow as labeled production images are received from laboratories performing the genotyping process.

[0091] In one embodiment, we used 96 weights of components of labeled production images to train the random forest classifier. A random forest classifier with 200 decision trees and a depth of 20 performed well. It is understood that a random forest classifier with a range of 200-500 decision trees and a depth range of 10-40 is expected to provide good results for this embodiment. We used randomized search cross-validation to tune the hyperparameters. The search range for depth was 5-150 and the search range for number of trees was 100-500. Increasing the number of trees may increase the performance of the model, but may also increase the time required for training. A training database 1001 containing features for 20,000 production cycle images is used to train a binary classifier labeled as good vs. bad classifier 151. The same training database may be used to train a root cause classifier 171 to predict failure classes. The root cause classifier 171 is trained on a training database 1021 consisting only of bad or failed production images, as shown in FIG. 10A.

[0092] Decision trees are prone to overfitting. To overcome this problem, bagging techniques are used to train decision trees in random forests. Bagging is a combination of bootstrap and aggregation techniques. In bootstrap, during training, we take a row sample from our training database and use it to train each decision tree in random forests. For example, a subset of the selected row features may be used to train decision tree 1. Hence, the training data for decision tree 1 may be referred to as row sample 1 with column sample 1, or RS1+CS1. The columns or features may be selected randomly. Decision tree 2 and subsequent decision trees in the random forest are trained in a similar manner by using a subset of the training data. Note that the training data for decision trees is generated with replacement, i.e., the same row data may be used in training multiple decision trees.

[0093] The second part of the bagging technique is the aggregation part that is applied during production. Each decision tree outputs a classification for each class. For binary classification, it can be 1 or 0. The output of the random forest is the aggregation of the outputs of the decision trees in the random forest with the majority of votes selected as the output of the random forest. By using votes from multiple decision trees, the random forest reduces the high variance in the decision tree results, thus resulting in good prediction results. By using row and column sampling to train each individual decision tree, each decision tree becomes an expert on the training records with the selected features.

[0094] During training, the output of the random forest is compared to the ground truth labels and the prediction error is calculated. During backpropagation, the weights of the 96 components (or eigenimages) are adjusted such that the prediction error is reduced. The number of components or eigenimages depends on the number of components selected from the output of a principal component analysis (PCA) using a measure of explained variance. During binary classification, the good vs. bad classifier uses image description features from the training data and applies one-vs-other (OvR) classification of a good class (or healthy labeled image) versus multiple bad classes (images labeled with one of six failure classes). The parameters of the trained random forest classifier (such as the component weights) are stored for use in the good vs. bad classification of production cycle images during inference.

[0095] Training of the root cause classifier 171 is performed in a similar manner. The training database 1021 consists of features from labeled process cycle images from faulty process cycles belonging to multiple failure classes. The random forest classifier 171 is trained using the image description features for one-vs-other (OvR) classification of each failure class versus the rest of the labeled training examples. Classification using the Random Forest classifier

[0096] Here, we describe the classification of production images using trained classifiers 151 and 171. FIG. 10B presents a two-stage classification 1080 of production images using a good vs. bad classifier 151 in a first stage and a root cause classifier 171 in a second stage. The process is presented using a series of process flow steps labeled 1-9. The process begins in step 1 by accessing a trained random forest classifier, labeled as a good vs. bad classifier 151. Input features of the production images stored in the database 1030 are provided as input to the classifier 151. The classifier distinguishes good images that belong to successful process cycles from bad images that belong to unsuccessful process cycles. The bad images may, for example, belong to multiple failure classes, each image may belong to one of the six failure classes described above. The trained classifier accesses criteria of eigenimages used to analyze the production images. The trained classifier creates image description features for the production images based on a linear combination of the eigenimages. The eigenimage weights are learned during training of the classifier as described above.

[0097] When we apply one-versus-all classification, every decision tree in the random forest classifier predicts an output for each class, i.e., whether the image belongs to one of the seven classes (one good class and six failure classes). Therefore, each decision tree in the random forest will output seven probability values, i.e., one value per class. The results from the decision trees are aggregated and the majority of votes are used to predict the image as good or bad. For example, if more than 50% of the decision trees in the random forest classify the image as good, the image is classified as a good image belonging to a successful production cycle. The sensitivity of the classifier can be adjusted, for example, by setting the threshold higher, which will result in more images being classified as bad. In process step 2, the output 151 from the classifier is checked. If the image is classified as a good image (step 3), the process ends (step 4). If, instead, the image is classified as a bad image indicating a failed process cycle (step 5), the system calls the root cause classifier 171 (step 6).

[0098] The root cause classifier is applied in the second stage of the two-stage process to determine the class of failure for the bad image. The process continues in the second stage by accessing the production image input features for the bad image (step 7) and providing the input features to the trained root cause classifier 171 (step 8). Each decision tree in the root cause classifier 171 votes for the input image features by applying one-versus-all classification. In this case, the classification determines whether the image belongs to one of the six failure classes versus the rest of the five failure classes. Each decision tree provides a classification for each class. The majority votes from the decision trees determine the failure class of the image (step 9).

[0099] We can use other classifiers to classify good vs. bad segmented images and perform root cause analysis. For example, the disclosed technique may apply a K-nearest neighbor (k-NN or KNN) algorithm to classify segmented images. The k-NN algorithm assumes that similar examples (or segmented images in our embodiment) exist in close proximity. The k-NN algorithm captures the idea of ​​similarity (also called proximity or closeness) by calculating the distance between data points or images. For this purpose, linear distance (or Euclidean distance) is commonly used. In k-NN classification, the output is a class membership, e.g., a good image class or a bad image class. An image is classified by the multiple votes of its neighbors, and an object is assigned to the most common class among its k nearest neighbors. The value of k is a positive integer.

[0100] To select the right value of k for our data, we run the k-NN algorithm several times with different values ​​of k and select a value of k that reduces the number of errors we encounter while maintaining the ability of the algorithm to make predictions accurately when the algorithm is given data that it has not seen before. We assume that we set the value of k to 1. This can result in inaccurate predictions. We consider that we have two clusters of data points: good images and bad images. Suppose we have a query example surrounded by many good image data points, but it is close to one bad image data point that is also within the cluster of good image data points. With k=1, the k-NN will inaccurately predict that the query example is a bad image. As we increase the value of k, the predictions of the k-NN algorithm become more stable due to majority voting (in classification) and averaging (in regression). Thus, the algorithm is more likely to make accurate predictions up to a certain value of k. As the value of k increases, we start to observe an increase in the number of errors. Values ​​of k in the range of 6 to 50 are expected to work.

[0101] Examples of other classifiers that can be trained and applied by the disclosed technology include multinomial logistic regression, support vector machines (SVM), gradient boosting trees, naive Bayes, etc. We evaluated the performance of the classifiers using three criteria: training time, accuracy, and interpretability of results. The random forest classifier performed better than the other classifiers. We briefly present the other classifiers in the following text.

[0102] Support vector machine classifiers also performed equally well as random forest classifiers. The SVM classifier locates a hyperplane between the feature vector for the good class versus the feature vectors for multiple bad classes. The disclosed techniques may include training a multinomial logistic regression. A multinomial regression model may be trained to predict the probability of different possible outcomes (multi-class classification). The model is used when the output is categorical. Hence, the model may be trained to predict whether an image belongs to a good class or one of multiple bad classes. The performance of the logistic regression classifier was lower than the random forest and SVM classifiers. The disclosed techniques may include training a gradient boosting model, which is an ensemble of predictive models such as decision trees. The model attempts to optimize a cost function over a function space by iteratively selecting a function that points in the negative gradient direction. For example, the model may be trained to minimize the mean squared error over the training dataset. The gradient boosting model required more training time compared to other classifiers. The disclosed techniques may include training a naive Bayes classifier that assumes that the value of a particular feature is independent of the value of any other feature. The naive Bayes classifier considers each of the features to contribute independently to the probability of an example belonging to a class. The naive Bayes classifier may be trained to classify images into a good class versus multiple bad classes. Example of Feature Engineering for Deep Learning-Based Classification

[0103] We present an example of feature engineering of an image for input to a convolutional neural network (CNN). Using feature engineering techniques shown in FIG. 11A, such as cropping, zero padding, and reflection operations, analysis frames 1105, 1107, and 1109 with a size of M×N pixels are generated. The analysis frames are sized to match the input image size used by the particular convolutional neural network. In one example, the size of the analysis frames is 224×224 pixels, which matches the size of the input images required by the ResNet-18 and VGG-16 models. The dimensions used in this example are 3600×1600 pixels for the segmented image. This example shows how a segmented image of 3600×1600 pixels is fitted into a square frame of size 224×224 pixels for input to a convolutional neural network (CNN). The first step in this process is to downsample the 3600x1600 pixel segmented image by 1 / 20 on each side to obtain a reduced size image of 180x80 pixels. The second step is to place the 180x80 pixel image into a 224x224 pixel frame. The placed image is then reflected horizontally to fill the 224x224 pixel square frame to the left and right of the 180x80 pixel placed image. The 180x80 pixel image is then reflected vertically to fill the top and bottom of the 224x224 square frame.

[0104] FIG. 11A includes an example 1100 illustrating cropping, zero padding, and reflection operations on a segmented image. An example segmented image 1103 is shown. The segmented image 1103 is rectangular. In cropping, a central portion or any other portion of the segmented image is cropped. FIG. 11A illustrates a cropped portion from the segmented image 1103 placed in the analysis frame 1105. The cropped portion of the segmented image may be of square dimensions, i.e., the cropped portion has the same number of pixels along the height and width of the cropped portion. For example, the cropped image placed in the analysis frame 1105 may be an 80×80 pixel cropped portion of the J×K image 1103. The cropped portion may be located at the center of the analysis frame. In other implementations, the cropped portion may be located near one side of the analysis frame. In another implementation, the cropped portion may be resampled to make the size of the cropped segment equal to the size of the analysis frame, for example, 224×224 pixels. This ensures that the cropped segment completely fills the analysis frame.

[0105] 11A illustrates the zero-padding operation after a segmented image 1103 is placed in an analysis frame 1107. The segmented image may be positioned in the center of the analysis frame as shown in example 1107. The segmented image may also be positioned at any other location within the analysis frame. Surrounding pixels in the analysis frame may be given a "0" image intensity value.

[0106] FIG. 11A also illustrates the application of reflection to fill pixels in the analysis frame 1109. The segmented image may be positioned at the center of the analysis frame 1107 and then reflected horizontally to fill the analysis frame along the left and right edges of the analysis frame. It may also be reflected vertically along the top and bottom edges filling the top and bottom of the analysis frame 1109. Evaluation of various strategies, detailed below, revealed that reflection to fill frames performs well in both training and production without requiring any changes in the input aperture of the image processing framework being evaluated. Surprisingly, reflection significantly outperformed zero padding.

[0107] FIG. 11B illustrates training data augmentation by applying a translation to the training example segmented image 1103. Applying a translation for data augmentation includes positioning the segmented image at multiple locations within the analysis frame. For example, as shown in analysis frame 1111, the segmented image is positioned at the center of the analysis frame. Horizontal and vertical translation of the segmented image may generate multiple analysis frames as shown in examples 1113, 1115, 1117, etc. As shown in FIG. 11B, the surrounding area may be initially zero-padded and depicted by a dark color (or black). However, the surrounding area does not necessarily have to be black. The disclosed technique may apply horizontal and vertical reflection padding to fill the surrounding area with the reflected segmented image as shown in FIG. 11A. Transfer Learning with Finetuning

[0108] In many real-world applications, entire convolutional neural networks (CNNs) are not trained from scratch with random initialization. This is because the training datasets are often small.<image-net.org> It is common to pre-train a CNN on a large dataset, such as ImageNet, which contains about 14 million images (available at http: / / www.imagenet.org / ), and then use the pre-trained CNN as an initialization or fixed feature extractor for the target task. This process is known as transfer learning to transfer knowledge learned from a source dataset to a target dataset. A commonly used transfer learning technique is called fine-tuning.<d2l.ai / chapter_computer-vision / fine-tuning.html> FIG. 12A, reproduced from, illustrates fine-tuning (1200). There are four steps to fine-tuning a deep learning model:

[0109] Step 1: Pre-train a neural network model, i.e., the source model, on a source dataset (e.g., the ImageNet dataset).

[0110] Step 2: The second step is to create a new neural network model, the target model, which replicates all the model design and their parameters on the source model, except for the output layer. We assume that these model parameters contain knowledge learned from the source dataset, which is equally applicable to the target dataset. We also assume that the output layer of the source model is closely related to the labels of the source dataset, and therefore is not used in the target model.

[0111] Step 3: The third step is to add an output layer to the target model, whose output size is the number of target dataset categories, and randomly initialize the model parameters of this layer. Thus, for a detection task, the output layer of our model may have two categories (normal and failure). For a classification task, the output layer of our model may have five categories corresponding to the five defect categories. The model may have an additional category to classify images with unknown failure types that are not classified into existing known failure categories.

[0112] Step 4: The fourth step is to train the target model on the target dataset. In this case, the training dataset contains about 75,000 labeled segmented images. We train the output layer from scratch while the parameters of all the remaining layers are fine-tuned based on the parameters of the source model. Network architecture of the VGG-16 model

[0113] VGG architecture (Simonyan et al., 2015, <<arxiv.org / abs / 1409.1556> The VGG-16 architecture (available at: http: / / www.vgg-16.org / ) has been widely used in computer vision in recent years. It includes stacked convolutional and max-pooling layers. We used a smaller, and therefore faster, 16-layer architecture known as VGG-16. The architecture (1210) is presented in FIG. 12B. The model consists of five convolutional layers: conv1 (1220), conv2 (1222), conv3 (1224), conv4 (1226) and conv5 (1228). One way to analyze the output of the convolutions is via a fully connected (FC) network. Thus, the output of the convolutional layers is fed into a fully connected (FC) layer. There are three fully connected layers: fc6 (1240), fc7 (1242) and fc8 (1244). The model uses "max pooling" between the convolutional layers and between the last convolutional layer and the fully connected layer. There are five "max pooling" layers 1230, 1232, 1234, 1236, and 1238. We extract features from the "conv5" layer (1228) which has a stride of 16 pixels.

[0114] The model takes as input an image size of 224x224 pixels. The input image may be an RGB image. The image is passed through a stack of convolutional (conv) layers, and filters are used with very small receptive fields: 3x3 (to capture the notion of left / right, top / bottom, center). One configuration utilizes 1x1 convolution filters, which can be viewed as a linear transformation (followed by a nonlinearity) of the input channels. The convolution stride is fixed at 1 pixel, i.e., the spatial padding of the convolutional layer input is such that the spatial resolution is preserved after the convolution, i.e., the padding is 1 pixel for a 3x3 convolutional layer.

[0115] Spatial pooling is performed by five "max pooling" layers 1230, 1232, 1234, 1236, and 1238 as shown in FIG. 12B, which follow some of the convolutional layers. The output of the convolution is also called the feature map. This output is given as the input to the max pooling layers. The goal of the pooling layers is to reduce the dimensionality of the feature map. For this reason, it is also called downsampling. The factor by which the downsampling is done is called the "stride" or "downsampling factor". The pooling stride is denoted by "s". In one type of pooling called "max pooling", the maximum value is selected for each stride. For example, consider max pooling with s=2 applied to a 12-dimensional vector x=[1, 10, 8, 2, 3, 6, 7, 0, 5, 4, 9, 2]. A max pooling vector x with stride s=2 means that we select the maximum value of every two values ​​starting from index 0, resulting in the vector [10, 8, 6, 7, 5, 9]. Thus, a max pooling vector x with stride s=2 results in a 6-dimensional vector. In the example VGG-16 architecture, max pooling is performed over a 2x2 pixel window with stride 2.

[0116] FIG. 12B illustrates that the image size is reduced after passing through each of the five convolutional layers in the VGG model and the respective max pooling layers. The input image has dimensions of 224×224 pixels. After passing through the first convolutional layer labeled conv1 (1220) and the max pooling layer (1230), the image size is reduced to 112×112 pixels. After passing through the second convolutional layer labeled conv2 (1222) and the max pooling layer (1232), the image size is reduced to 56×56 pixels. The third convolutional layer labeled conv3 (1224) and the max pooling layer (1234) reduces the image size to 28×28 pixels. The fourth convolutional layer labeled conv4 (1226) and the max pooling layer (1236) reduces the image size to 14×14 pixels. Finally, a fifth convolutional layer, labeled conv5 (1228), and a max pooling layer (1238) reduce the image size to 7x7 pixels.

[0117] Three fully connected (FC) layers 1240, 1242, and 1244 follow the stack of convolutional layers. The first two FC layers 1240 and 1242 each have 4096 channels, and the third FC layer 1244 performs 1000-way classification and therefore contains 1000 channels (one for each class). The last layer is a softmax layer. The depth of the convolutional layers may vary in different architectures of the VGG model. The configuration of the fully connected layers may be the same in different architectures of the VGG model.

[0118] FIG. 12C illustrates a VGG-16 architecture 1270 applied by the disclosed techniques for a detection task. The model architecture illustrates parameter values ​​for five convolutional layers, conv1-conv5, and three fully connected layers, fc6-fc8. The output from the final FC layer 1244 produces two outputs that classify the segmented image as normal or failed. The labels in FIG. 12C correspond to the respective elements in the architecture diagram in FIG. 12B. Network architecture of the RESNET-18 model

[0119] ResNet architecture (He et al., CVPR 2016, <<arxiv.org / abs / 1512.03385> The ResNet-18 architecture (available at: http: / / www.resnet-18.org / ) was designed to avoid problems with ultra-deep neural networks. Most advantageously, the use of residual connections helps overcome the vanishing gradient problem. We used a ResNet-18 architecture with 18 trainable layers. Figures 12D and 12E illustrate an exemplary ResNet-18 architecture 1280. The ResNet-18 architecture is organized into four layers: 1260, 1262 shown in Figure 12D, and 1264, 1268 shown in Figure 12E. Each layer comprises two blocks labeled "0" and "1". Each block comprises two convolutional layers labeled "conv1" and "conv2". There is one convolutional layer 1259 before the layer 1 block (1260). Thus, there are a total of 17 convolutional layers separated by batch normalization and ReLU. One fully connected (FC) layer (1270) at the end produces two outputs. This architecture is used for detection tasks where the network classifies segmented images into normal (or good) images and failure (or bad) images.

[0120] Training deep learning models with dozens of convolutional layers is difficult because the distribution of inputs to the deep layers of the network can change after each mini-batch when the weights are updated. This slows down the convergence speed of the model. Batch normalization (Ioffe and Szegedy 2015,<arxiv.org / abs / 1502.03167> The batch normalization technique (available at: http: / / www.data.net / data / articles / 0210233 / ) can overcome this problem. Batch normalization standardizes the inputs to a layer for each mini-batch, reducing the number of epochs required to train a deep learning model. Performance Comparison - Feature Engineering and Deep Learning Models

[0121] We compared the performance of two models, ResNet-18 and VGG-16, using the augmented input data generated by applying the feature engineering techniques presented above. Example 1300 in FIG. 13 presents the results of the performance comparison using the macro F1 score. The macro F1 performance score may be considered to reflect a combination of precision and recall values. For example, the F1 score may combine the precision (p) and recall (r) values ​​with equal weighting, as shown in Equation (1) below. The macro F1 score may be calculated as the average of the F1 scores for all failure categories.

number

[0122] Referring back to the results in FIG. 13, the top two bars 1305 and 1310 present the scores of the VGG-16 and ResNet-18 models, respectively. Both models produced the same macro F1 (or F1) performance score of 96.2 when trained using a fine-tuning approach to transfer learning. Both the VGG-16 and ResNet-18 models were trained on the full segmented images of the image generation chip. Reflection padding was used to fill the larger size analysis frames as described above. The training data is augmented by creating copies of the images. The system may augment the training data by creating multiple copies of the labeled images in one class (such as a failure class) to balance the dataset when images belonging to this class are scarce compared to images belonging to one or more other classes. Alternatively, image copies may be used multiple times during training to balance the training data.

[0123] The graph in FIG. 13 also illustrates the performance of the ResNet-18 model using different feature engineering techniques in the third through fifth bars of the performance comparison graph. The third bar 1315 illustrates that the ResNet-18 model achieves a score of 77.2 when trained using only cropping techniques for feature engineering. The results are poor when only cropping is used for training data generation because not many defects occur in the center of the segmented image. Thus, such defects may be missed by the model during training. The fourth bar 1320 shows that the performance score is 92.3 when the ResNet-18 model is trained using training data generated with zero padding used to fill areas in the analysis frame surrounding the segmented image. Zero padding does not perform well due to unnatural (dark) areas in the analysis frame surrounding the location of the segmented image. The fifth bar from the top, labeled 1325, shows that when the ResNet-18 model is trained using training data that has been augmented by creating copies of the labeled images, the performance score is 94.9. In this case, no translation is used to create the additional training data transformations.

[0124] The bottom bar 1335 of the graph 1300 presents the performance of the base model. The base model is a ResNet-18 CNN used as the feature extractor with no fine-tuning. No feature engineering techniques such as reflection padding are used. No data augmentation is used. The base model scores 74.3.

[0125] Figure 14 presents a breakdown of performance scores by failure category for the best performing model disclosed. The best model achieved an overall macro F1 score of 97% and an accuracy of 96%. Table 1401 in Figure 14 presents the F1 scores for different failure categories, examples of which are presented above in Figures 5A-5I.

[0126] The confusion matrix 1420 illustrates the performance of the model in terms of predicted labels versus true labels for five failure categories. The majority of defect categories are correctly predicted as indicated by the values ​​on the diagonal (labeled 1425). The vertical bars on the right indicate the number of samples with different failure categories. The number of samples with a particular type of failure can be calculated by summing the numbers in a row (labeled 1430) for that failure type. For example, the number of samples containing offset and spacer type failures are 38 and 26, respectively. The correct prediction of the failure category can be determined by looking at the values ​​on the diagonal. For example, the model correctly predicted 36 samples with offset failures and 24 samples with space shift failures. Similarly, of the 129 samples with hybridization or "hyb" failures, 123 were correctly predicted. The model correctly predicted 163 samples with reagent flow failures out of a total of 170 samples with reagent flow failures. The largest number of samples contained surface abrasion failures. A total of 428 samples were correctly predicted out of a total of 437 samples with wear failures. Misclassified samples

[0127] Figure 15 presents an analysis of 6 images from the 26 misclassifications in the confusion matrix presented in Figure 14. For each segmented image, the manually annotated failure labels are listed in the top row, and the predicted labels from the model are listed in the bottom row. Some images were misclassified by the deep learning model, while others were misclassified by the humans who labeled the training set.

[0128] Four sample images from the left 1503, 1505, 1507, and 1509 have multiple defects that belong to different failure categories. Manual annotations represent only one of the multiple defects. For example, the first image on the left (labeled 1503) is labeled by a human annotator as having surface wear. The surface wear is present in the upper left portion of the image, as shown within the bounding box. The model predicted that the image has hybridization or hyb failures. It can be seen that the image has hyb failures in two locations, in the upper right portion of the image, and near the bottom of the image, as pointed to by the arrows.

[0129] The last two images 1511 and 1513 were incorrectly labeled by the human annotators, and the deep learning model predicted these images correctly. For example, the fifth image from the left (labeled 1511) is labeled as having a spacer shift failure. However, the model predicted the image to have a surface wear failure. The human annotators may have misidentified the failure category due to the dark areas of the image being located close to the bottom edge of the image. Similarly, the sixth image from the left (labeled 1513) was labeled by the human annotators as having a hybridization or hyb failure. The model predicted the failure category as a reagent flow failure, which is the correct failure category for the image. Thus, the performance of the machine learning is even better than indicated by the F1 score. Improving performance using deep learning

[0130] We compared the performance of our deep learning-based approach (deepIBEX) with a previous solution using a random forest model (IBEX) on the same split of the dataset. The results show that deepIBEX performs better than IBEX as measured by macro F1 score and accuracy in both anomaly detection (separating good vs. bad images) and classification (identifying the root cause of bad images) tasks. methodology

[0131] The segmented image dataset was split into training, validation, and test sets with sample ratios of 70%, 15%, and 15%. For the two models compared here (deepIBEX and IBEX), we used the same split of the dataset for parameter tuning, training, and evaluation. We used random grid search to tune model hyperparameters on the training and validation sets. We finally evaluated model performance on the test set. Tested parameters

[0132] A set of 10 hyperparameters was tested for each model through a random grid search. In IBEX, the main hyperparameters we tuned were the final dimension of the PCA and the depth of the trees in the random forest. In deepIBEX, we tuned the learning rate, batch size, and momentum. Macro F1 score and accuracy measures

[0133] The deepIBEX model performed better than the IBEX model, as shown by the macro F1 score and precision. The F1 score for each category may be defined as the harmonic mean of the recall and precision of that category. The F1 score may also be calculated using equal weights for precision and recall, as shown in equation (1) above. The macro F1 score may be calculated as the average of the F1 scores of all failure categories. The macro F1 scores and precision, which measure the performance of the model on the test set, are presented in the following table (Table 1). A high macro F1 score, or high precision, means good performance of the model on the test data. Precision is defined as the proportion of correctly labeled samples among all test samples. For both detection and classification, deepIBEX outperformed IBEX. [Table 1] Table 1: Comparison of DeepIBEX and IBEX models task Model Parameters Macro F1 score (%) accuracy(%) detection deepIBEX Fine Tune 88.9 94.4 detection IBEX 200 components, depth=5 80.5 89.1 detection Gauss NB 50 ingredients 76.2 89.0 classification deepIBEX Fine Tune 96.2 96.8 classification IBEX 150 components, depth=50 68.5 75.4 classification Gauss NB 50 ingredients 69.7 72.5 Process Flow for Training and Applying Good vs. Bad Classifiers

[0134] FIG. 16 presents a flow chart 1601 presenting process steps for training good versus bad classifiers and applying the trained classifiers to classify production images of a segment of an image generating chip.

[0135] The process steps presented in the flowchart 1601 may be implemented using a processor accessible to a computer system and programmed using a computer program stored in executable memory by a processor, by dedicated logic hardware including field programmable integrated circuits, and by a combination of dedicated logic hardware and a computer program. As with all flowcharts herein, it will be understood that many of the steps may be combined, performed in parallel, or performed in a different order without affecting the functionality achieved. Furthermore, it will be understood that the flowcharts herein show only steps relevant to an understanding of the present technology, and that numerous additional steps to achieve other functionality may be performed before, after, and between the steps shown.

[0136] The process begins at step 1602. Training data creation process step 1610 may include a number of operations (1612, 1614, and 1616) that may be performed to create training data including labeled images of sections of an image generation chip. Training data creation may include generating a J×K labeled image (step 1612). This step may include cropping portions from a larger image to generate the J×K labeled image. The J×K labeled image may be positioned at a number of locations within an M×N analysis frame (step 1614). The M×N analysis frame is sized to match the input image size required by the convolutional neural network (CNN). The J×K image may be small in size relative to the M×N analysis frame and may not completely fill the M×N analysis frame. Additionally, the M×N analysis frame may be square and the J×K image may be rectangular. In step 1616, a portion of the J×K labeled image located within the M×N analysis frame may be used to fill around the edges of the J×K labeled image. Horizontal reflections may be used to fill the M×N analysis frame along the left and right edges of the analysis frame. Vertical reflections may be used to fill the M×N analysis frame along the top and bottom edges of the M×N analysis frame. The steps presented above may be repeated to generate many training examples by varying the position of the same J×K labeled image within the M×N analysis frame.

[0137] A convolutional neural network (CNN) may be trained in step 1620 using training data generated by performing the process steps presented above. The system may train a pre-trained CNN, such as a VGG-16 or ResNet-18 model. The trained CNN model may then be applied to production images of the section to classify the images as good or bad (step 1624). Feature engineering techniques, such as reflection padding, may be applied to production images of the section of the image generation chip to fill the M×N input frames into the CNN model. Images classified by the classifier as good or normal may indicate a successfully completed process. Images classified as bad or failed may indicate a process failure (step 1626). The process may continue to step 1628 where the bad process cycle images may be further classified to determine the root cause of the failure. If the production images are classified as good, the process may end in step 1630. Specific Implementations

[0138] The disclosed technology applies image classification for evaluation and root cause analysis of the genotyping process. Two tasks are performed by a classifier: separation of successful and unsuccessful production images, and then root cause analysis of the unsuccessful images. Training and inference of good vs. bad classifiers

[0139] We first present the classification of successful and unsuccessful production images. In one embodiment of the disclosed technology, a method is described for training a convolutional neural network (CNN) to identify and classify images of a section of an image-producing chip from bad or failed or unsuccessful processes that result in process cycle failure. The method includes using a pre-trained convolutional neural network (CNN) to extract image features. The pre-trained CNN may accept images of dimension M×N. Examples of image dimensions for input to the CNN may include 224×224 pixels, 227×227 pixels, 299×299 pixels. Alternatively, the size of the input image to the CNN may be in the range of 200-300 pixels on a side, or in the range of 75-550 pixels on a side. The input image may be square or rectangular. The method includes creating a training dataset using labeled images of dimension J×K that are smaller than M×N, are normal, and depict process failures. Exemplary dimensions of a J×K size labeled image are 180×80 pixels, 224×100 pixels, 200×90 pixels, 120×120 pixels, 224×224 pixels, 504×224 pixels, etc. The method includes creating a training data set using labeled images of dimension J×K, where dimension J×K is smaller than M×N and is normal and depicts a process failure. The images are from a section of an image generating chip. The method includes positioning the J×K labeled images at multiple locations within an M×N (224×224) frame. The method includes using at least a portion of a particular J×K labeled image to fill around an edge of a particular J×K labeled image, thereby filling the M×N frame.

[0140] The method includes further training the pre-trained CNN to generate a segment classifier using the training data set. The method includes storing coefficients of the trained classifier to identify and classify images of the segments of the imaging chip from a production process cycle. The trained classifier can accept images of the segments of the imaging chip and classify the images as normal or as depicting a process failure.

[0141] In a production embodiment, the trained CNN can be applied to identify bad process cycle images of a section of an image generation chip. Here, we present a method for identifying bad process cycle images of a section of an image generation chip that cause a process cycle failure. The method includes creating an input to a trained classifier.

[0142] The input may fill the M×N input aperture of the image processing framework, or may have a smaller dimension J×K and be reflected to fill the M×N analysis frame. Taking the latter approach, creating the input includes accessing an image of the partition having dimension J×K. The method includes positioning the J×K image within the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned within the M×N analysis frame. Depending on the relative sizes of J×K vs. M×N, some zero padding may be applied, for example to fill narrow strips along the top and bottom of the analysis frame, but reflection has been found to work better. The method includes inputting the M×N analysis frame to a trained classifier. The method includes using the trained classifier to classify an image of the partition of the imaging chip as normal or as depicting a process failure. The method includes outputting a classification result of the partition of the imaging chip.

[0143] Embodiments of this method, as well as other methods disclosed, optionally include one or more of the following features. The method may also include features described in connection with the methods presented above. For brevity, alternative combinations of method features are not individually listed. Features applicable to methods, systems, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0144] In one embodiment, the method further comprises centering the J×K image in the M×N analysis frame.

[0145] In one embodiment, the method includes applying horizontal reflection to at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image in the M×N analysis frame.

[0146] In one embodiment, the method includes applying a vertical reflection to at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image in the M×N analysis frame.

[0147] In one embodiment, a J×K labeled image is generated by cropping a portion from a larger image and placing the cropped J×K portion into an M×N frame. Examples of large image sizes include images with dimensions of 504×224 pixels or even larger.

[0148] A labeled image of dimension J×K is obtained by downsampling a high resolution image from a scanner, so that the resolution of the high resolution image may be reduced by 1 / 2 to 1 / 50 times per side of the original pixel resolution. A reduction of 1 / 25 times per side reduces the number of pixels to 1 / 625 of the original number of pixels. In one embodiment, the high resolution image of the segment obtained from a scanner or genotyping device has a size of 3600×1600 pixels.

[0149] The above computer-implemented methods may be executed on a system including computer hardware. The computer-implemented system may execute one or more of the above methods. The computer-implemented system may incorporate any of the method features described immediately above or throughout this application that apply to the method performed by the system. For brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0150] As an article of manufacture rather than a method, program instructions executable by a processor may be stored on a non-transitory computer readable medium (CRM). When executed, the program instructions perform one or more of the computer-implemented methods described above. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may become one or more components of a computer-implemented system that performs the disclosed methods.

[0151] Each of the features discussed in this specific embodiment section for the method embodiment applies equally to the CRM and system embodiments, and as noted above, all method features are not repeated here but should be considered again by reference. Training and inference root cause analysis

[0152] In one embodiment of the disclosed technology, a method is described for training a convolutional neural network (CNN) to classify images of a section of an image generation chip by root cause of process failure. The method includes using a pre-trained CNN to extract image features. The pre-trained CNN can accept images of dimension M×N. An example of an image dimension includes 224×224 pixels. The method includes creating a training data set using labeled images of dimension J×K that belong to at least one failure category of a plurality of failure categories that result in a process failure. Exemplary dimensions of the J×K size labeled images are 180×80 pixels, 200×90 pixels, etc. The images are from a section of the image generation chip. The method includes positioning the J×K labeled images at multiple locations within an M×N (224×224) frame. The method includes using at least a portion of a particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame. The method includes further training the pre-trained CNN to generate a segment classifier using the training data set. The method includes storing coefficients of the trained classifier to identify and classify images of segments of the image generating chip from a production process cycle. The trained classifier may accept images of segments of the image generating chip and classify the images by a root cause of process failure among a plurality of failure categories.

[0153] In a production implementation, the trained CNN can be applied to classify bad process cycle images of a section. We present here a method for identifying and classifying bad process cycle images of a section of an image generating chip that cause a process cycle failure. The method includes creating an input to a trained classifier.

[0154] The input may fill the M×N input aperture of the image processing framework, or may have a smaller dimension J×K and be reflected to fill the M×N analysis frame. Taking the latter approach, creating the input includes accessing an image of the partition having dimension J×K. The method includes positioning the J×K image within the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along edges of the J×K image positioned within the M×N analysis frame. Depending on the relative sizes of J×K vs. M×N, some zero padding may be applied, for example to fill narrow strips along the top and bottom of the analysis frame, but reflection has been found to work better. The method includes inputting the M×N analysis frame to a trained classifier. The method includes using the trained classifier to classify the image of the partition of the imaging chip by root cause of process failure among a plurality of failure categories. The method includes outputting the classification results of the partition of the imaging chip.

[0155] Embodiments of this method, as well as other methods disclosed, optionally include one or more of the following features. The method may also include features described in connection with the methods presented above. For brevity, alternative combinations of method features are not individually listed. Features applicable to methods, systems, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0156] In one embodiment, the method further comprises centering the J×K image in the M×N analysis frame.

[0157] In one embodiment, the method includes applying horizontal reflection to at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image in the M×N analysis frame.

[0158] In one embodiment, the method includes applying a vertical reflection to at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image in the M×N analysis frame.

[0159] In one embodiment, a J×K labeled image is generated by cropping a portion from a larger image and placing the cropped J×K portion into an M×N frame. Examples of large image sizes include images with dimensions of 504×224 pixels or even larger.

[0160] The labeled image of dimensions J×K is obtained by downsampling the high-resolution image from the scanner, so that the resolution of the high-resolution image may be reduced by 1 / 2 to 1 / 50 of the original resolution. In one embodiment, the high-resolution image of the segments obtained from the scanner or genotyping device has a size of 3600×1600 pixels.

[0161] The multiple failure categories can include at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0162] The multiple failure categories may include a residual failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0163] The above computer-implemented methods may be executed on a system including computer hardware. The computer-implemented system may execute one or more of the above methods. The computer-implemented system may incorporate any of the method features described immediately above or throughout this application that apply to the method performed by the system. For brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0164] As an article of manufacture rather than a method, program instructions executable by a processor may be stored on a non-transitory computer readable medium (CRM). When executed, the program instructions perform one or more of the computer-implemented methods described above. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may become one or more components of a computer-implemented system that performs the disclosed methods.

[0165] Each of the features discussed in this specific embodiment section for the method embodiment applies equally to the CRM and system embodiments, and as noted above, all method features are not repeated here but should be considered again by reference. Combined, single-pass good vs. bad detection and root cause of process failure

[0166] In one embodiment of the disclosed technology, a method is described for training a convolutional neural network (CNN) to identify and classify images of a section of an image generation chip from faulty process cycles resulting in process cycle failures. The method includes using a pre-trained CNN to extract image features. The pre-trained CNN can accept images of dimension M×N. The method includes creating a training data set using labeled images of dimension J×K that are normal and belong to at least one failure category of a plurality of failure categories resulting in process failures. The images are from a section of the image generation chip. The method includes positioning the J×K labeled images at multiple locations within an M×N (224×224) frame. The method includes using at least a portion of a particular J×K labeled image to fill around an edge of a particular J×K labeled image, thereby filling M×N. The method includes further training the pre-trained CNN to generate a section classifier using the training data set. The method includes storing coefficients of the trained classifier to identify and classify images of the section of the image generation chip from production process cycles. The trained classifier may accept images of a section of the imaging chip and classify the images as normal or as belonging to at least one of a plurality of failure categories that result in a process failure.

[0167] In a production embodiment, the trained CNN can be applied to classify bad process cycle images of a section. We present a method for identifying bad process cycle images of a section of an image generating chip that cause a process cycle failure. The method includes creating an input to a trained classifier.

[0168] The input may fill the M×N input aperture of the image processing framework, or may have a smaller dimension J×K and be reflected to fill the M×N analysis frame. Taking the latter approach, creating the input includes accessing an image of the partition having dimension J×K. The method includes positioning the J×K image within the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned within the M×N analysis frame. Depending on the relative sizes of J×K vs. M×N, some zero padding may be applied, for example to fill narrow strips along the top and bottom of the analysis frame, but reflection has been found to work better. The method includes inputting the M×N analysis frame to a trained classifier. The method includes using the trained classifier to classify an image of the partition of the image generation chip as normal or as belonging to at least one of a plurality of failure categories. The method includes outputting a classification result of the partition of the image generation chip.

[0169] The method may also include features described in connection with the methods presented above. For the sake of brevity, alternative combinations of method features are not individually listed. Features applicable to methods, systems, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0170] The above computer-implemented methods may be executed on a system including computer hardware. The computer-implemented system may execute one or more of the above methods. The computer-implemented system may incorporate any of the method features described immediately above or throughout this application that apply to the method performed by the system. For brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0171] As an article of manufacture rather than a method, program instructions executable by a processor may be stored on a non-transitory computer readable medium (CRM). When executed, the program instructions perform one or more of the computer-implemented methods described above. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may become one or more components of a computer-implemented system that performs the disclosed methods.

[0172] Each of the features discussed in this specific embodiment section for the method embodiment applies equally to the CRM and system embodiments, and as noted above, all method features are not repeated here but should be considered again by reference. item

[0173] The following clauses describe various aspects of the technology described herein. Separate good vs. bad detections and then classify the root causes. Good vs. bad training

[0174] Clause 1. A method for training a convolutional neural network to identify and classify images of segments of an imaging chip that result in process failure, comprising: using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimensions M×N; creating a training data set using labeled images of dimension J×K smaller than M×N depicting process successes and failures; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, This allows a trained classifier to accept images of sections of an imaging chip and classify the images as depicting process success and failure.

[0175] Clause 2. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, applying horizontal reflections to at least a portion of the particular J×K labeled image; 2. The method of claim 1, further comprising:

[0176] Clause 3. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, Applying vertical reflections to at least a portion of the particular J×K labeled image. 2. The method of claim 1, further comprising:

[0177] Clause 4. Creating a training dataset using labeled images of dimension J×K, generating J×K labeled images by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 2. The method of claim 1, further comprising:

[0178] Clause 5. The method of clause 1, wherein a labeled image of dimension J × K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0179] Clause 6. The method of clause 5, wherein the high-resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0180] Clause 7. A non-transitory computer-readable storage medium having computer program instructions thereon, the instructions, when executed on a processor, for identifying and classifying images of segments of an imaging chip that result in process failures, using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimensions M×N; creating a training data set using labeled images of dimension J×K smaller than M×N depicting process successes and failures; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, This allows the trained classifier to accept images of a section of the imaging chip and classify the images as either normal or depicting a process failure. A non-transitory computer-readable storage medium embodying a method.

[0181] Clause 8. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, applying horizontal reflections to at least a portion of the particular J×K labeled image; 8. A non-transitory computer-readable storage medium according to clause 7, further comprising:

[0182] Clause 9. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, Applying vertical reflections to at least a portion of the particular J×K labeled image. 8. A non-transitory computer-readable storage medium according to clause 7, further comprising:

[0183] Clause 10. The method of claim 1, further comprising the step of creating a training data set using labeled images of dimension J×K, generating J×K labeled images by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 8. A non-transitory computer-readable storage medium according to clause 7, further comprising:

[0184] Clause 11. The non-transitory computer-readable storage medium of clause 7, wherein the labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0185] Clause 12. The non-transitory computer-readable storage medium of clause 11, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0186] Clause 13. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions that, when executed on the processor, identify and classify images of sections of an imaging chip that result in process failures, the instructions comprising: using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training data set using labeled images of dimension J×K smaller than M×N depicting process successes and failures; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; Further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; and storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, This allows the trained classifier to accept images of a section of the imaging chip and classify the images as either normal or depicting a process failure. A system that performs an action.

[0187] Clause 14. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, Applying horizontal reflections to at least a portion of a particular J×K labeled image 14. The system of claim 13, further performing operations including:

[0188] Clause 15. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, Applying vertical reflections to at least a portion of a particular J×K labeled image. 14. The system of claim 13, further performing operations including:

[0189] Clause 16. Creating a training dataset using labeled images of dimension J × K Generate J×K labeled images by cropping parts from a larger image and placing the cropped J×K parts into an M×N frame. 14. The system of claim 13, further performing operations including:

[0190] Clause 17. The system of clause 13, wherein the labeled image of dimension J×K is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0191] Clause 18. The system of clause 17, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels. training root cause analysis

[0192] Clause 19. A method for training a convolutional neural network to classify images of a section of an imaging chip according to a root cause of a process failure, comprising: using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimensions M×N; creating a training data set using labeled images of dimension J×K that belong to at least one failure category of a plurality of failure categories that result in a process failure; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; Further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; and storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, This allows the trained classifier to accept images of a section of an image generating chip and classify the images according to root cause of process failure from among a number of failure categories.

[0193] Clause 20. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, applying horizontal reflections to at least a portion of the particular J×K labeled image; 20. The method of claim 19, further comprising:

[0194] Clause 21. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, Applying vertical reflections to at least a portion of the particular J×K labeled image. 20. The method of claim 19, further comprising:

[0195] Clause 22. A step of creating a training data set using labeled images of dimension J×K, generating J×K labeled images by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 20. The method of claim 19, further comprising:

[0196] Clause 23. The method of clause 19, wherein a labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0197] Clause 24. The method according to clause 23, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0198] Clause 25. The method of clause 19, wherein the plurality of failure categories includes at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0199] Clause 26. The method of clause 19, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0200] Clause 27. A non-transitory computer-readable storage medium having computer program instructions, which, when executed on a processor, are for training a convolutional neural network to classify images of a section of an image generating chip by root cause of a process failure, using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimensions M×N; creating a training data set using labeled images of dimension J×K that belong to at least one failure category of a plurality of failure categories that result in a process failure; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; Further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; and storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, Thus, the trained classifier can accept images of a section of the imaging chip and classify the images according to root cause of process failure among multiple failure categories. A non-transitory computer-readable storage medium embodying a method.

[0201] Clause 28. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, applying horizontal reflections to at least a portion of the particular J×K labeled image; 28. The non-transitory computer-readable storage medium of claim 27, further comprising:

[0202] Clause 29. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, Applying vertical reflections to at least a portion of the particular J×K labeled image. 28. A non-transitory computer-readable storage medium according to clause 27, further comprising:

[0203] Clause 30. A step of creating a training data set using labeled images of dimension J×K, generating J×K labeled images by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 28. A non-transitory computer-readable storage medium according to clause 27, further comprising:

[0204] Clause 31. The non-transitory computer-readable storage medium of clause 27, wherein a labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0205] Clause 32. The non-transitory computer-readable storage medium of clause 31, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0206] Clause 33. The non-transitory computer-readable storage medium of clause 27, wherein the plurality of failure categories includes at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0207] Clause 34. The non-transitory computer-readable storage medium of clause 27, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0208] Clause 35. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions to train a convolutional neural network to classify images of a section of an image generating chip by root cause of process failure, the instructions, when executed on the processor, using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training data set using labeled images of dimension J×K that belong to at least one failure category of a plurality of failure categories that result in a process failure; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; Further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; and storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, Thus, the trained classifier can accept images of a section of the imaging chip and classify the images according to root cause of process failure among multiple failure categories. A system that performs an action.

[0209] Clause 36. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, Applying horizontal reflections to at least a portion of a particular J×K labeled image 36. The system of claim 35, further performing operations including:

[0210] Clause 37. Using at least a portion of the particular J×K labeled image to fill in around an edge of the particular J×K labeled image, Applying vertical reflections to at least a portion of a particular J×K labeled image. 36. The system of claim 35, further performing operations including:

[0211] Clause 38. Creating a training dataset using labeled images of dimension J × K Generate J×K labeled images by cropping parts from a larger image and placing the cropped J×K parts into an M×N frame. 36. The system of claim 35, further performing operations including:

[0212] Clause 39. The system of clause 35, wherein the labeled image of dimension J×K is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0213] Clause 40. The system of clause 39, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0214] Clause 41. The system of clause 35, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0215] Clause 42. The system of clause 35, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0216] Good vs. Bad Inference Clause 43. A method for identifying process cycle images of a section of an image generating chip causing a process cycle failure, comprising: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis frame; inputting an M×N analysis frame to a trained classifier; a trained classifier classifying images of the sections of the imaging chip as either normal or representative of a process failure; outputting the classification result of the image generating chip; A method comprising:

[0217] Clause 44. The method of clause 43, further comprising the step of centering the J×K image in the M×N analysis frame.

[0218] Clause 45. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a horizontal reflection to at least a portion of the particular J×K image; 44. The method of claim 43, further comprising:

[0219] Clause 46. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a vertical reflection to at least a portion of the particular J×K image; 44. The method of claim 43, further comprising:

[0220] Clause 47. The step of creating an input to a trained classifier comprises: generating a J×K image by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 44. The method of claim 43, further comprising:

[0221] Clause 48. The method of clause 43, wherein the J×K image is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0222] Clause 49. The method according to clause 48, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0223] Clause 50. When the image of the section of the imaging chip is determined to depict a process failure, accessing a second convolutional neural network, the second convolutional neural network being trained to classify the image by a root cause of the process failure from among a plurality of failure categories; applying a second convolutional neural network to the images of the segments to classify among a plurality of failure categories to select among the plurality of failure categories as a likely root cause of the failed process cycle; 44. The method of claim 43, further comprising:

[0224] Clause 51. The method of clause 50, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0225] Clause 52. The method of clause 50, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0226] Clause 53. A non-transitory computer readable storage medium having computer program instructions, which when executed on a processor, perform a process cycle image identification process for identifying a process cycle image of a section of an image generating chip that causes a process cycle failure. Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis frame; inputting an M×N analysis frame to a trained classifier; a trained classifier classifying images of the sections of the imaging chip as either normal or representative of a process failure; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0227] Clause 54. A non-transitory computer-readable storage medium according to clause 53, implementing a method further comprising centering the J×K image in the M×N analysis frame.

[0228] Clause 55. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a horizontal reflection to at least a portion of the particular J×K image; 54. A non-transitory computer-readable storage medium according to clause 53, further comprising a method for:

[0229] Clause 56. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a vertical reflection to at least a portion of the particular J×K image; 54. A non-transitory computer-readable storage medium according to clause 53, further comprising a method for:

[0230] Clause 57. The step of creating an input to a trained classifier comprises: generating a J×K image by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 54. A non-transitory computer-readable storage medium according to clause 53, further comprising a method for:

[0231] Clause 58. The non-transitory computer-readable storage medium of clause 53, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 times per side.

[0232] Clause 59. The non-transitory computer-readable storage medium of clause 58, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0233] Article 60. accessing a second convolutional neural network upon determining that the image of the segment of the imaging chip depicts a process failure, the second convolutional neural network being trained to classify the image by a root cause of the process failure from among a plurality of failure categories; applying a second convolutional neural network to the images of the segments to classify among a plurality of failure categories to select among the plurality of failure categories as a likely root cause of the failed process cycle; 54. A non-transitory computer-readable storage medium according to clause 53, further comprising a method for:

[0234] Clause 61. The non-transitory computer-readable storage medium of clause 60, wherein the plurality of failure categories includes at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0235] Clause 62. The non-transitory computer-readable storage medium of clause 60, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0236] Clause 63. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions, which when executed on the processor, identify process cycle images of a section of an image generating chip causing a process cycle failure, the instructions comprising: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; mapping the J×K images into an M×N analysis frame; and filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image to thereby fill the M×N analysis frame. inputting an M×N analysis frame to a trained classifier; inputting an image of a section of an imaging chip, the image being classified by a trained classifier as either normal or representative of a process failure; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0237] Clause 64. The system of clause 63, further performing an operation including centering the J×K image in the M×N analysis frame.

[0238] Clause 65. Using at least a portion of the particular J×K image to fill in around an edge of the particular J×K image Applying horizontal reflection to at least a portion of a particular J×K image 64. The system of claim 63, further performing operations including:

[0239] Clause 66. Using at least a portion of the particular J×K image to fill in around an edge of the particular J×K image, Applying a vertical reflection to at least a portion of a particular J×K image 64. The system of claim 63, further performing operations including:

[0240] Clause 67. Creating an input to a trained classifier Generate a J×K image by cropping a portion from a larger image and place the cropped J×K portions in an M×N frame. 64. The system of claim 63, further performing operations including:

[0241] Clause 68. The system of clause 63, wherein the J×K image is obtained by downsampling a high resolution image from the scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0242] Clause 69. The system of clause 68, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0243] Article 70. accessing a second convolutional neural network when the image of the segment of the imaging chip is determined to depict a process failure, the second convolutional neural network being trained to classify the image by a root cause of the process failure from among a plurality of failure categories; applying a second convolutional neural network to the image of the segments to classify among a plurality of failure categories to select among the plurality of failure categories as a probable root cause of the failed process cycle; 64. The system of claim 63, further performing operations including:

[0244] Clause 71. The system of clause 70, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0245] Clause 72. The system of clause 70, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause. reasoning root cause

[0246] Clause 73. A method for identifying and classifying process cycle images of a section of an image generating chip that causes a process cycle failure, comprising: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis; inputting an M×N analysis frame to a trained classifier; a trained classifier classifying the images of the image generating chip segments according to root causes of process failures among a plurality of failure categories; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0247] Clause 74. The method of clause 73, further comprising the step of centering the J×K image in the M×N analysis frame.

[0248] Clause 75. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a horizontal reflection to at least a portion of the particular J×K image; 74. The method of claim 73, further comprising:

[0249] Clause 76. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a vertical reflection to at least a portion of the particular J×K image; 74. The method of claim 73, further comprising:

[0250] Clause 77. The step of creating an input to a trained classifier comprises: generating a J×K image by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 74. The method of claim 73, further comprising:

[0251] Clause 78. The method of clause 73, wherein the J×K image is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0252] Clause 79. The method according to clause 78, wherein the high resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0253] Clause 80. The method of clause 73, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0254] Clause 81. The method of clause 73, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0255] Clause 82. A non-transitory computer-readable storage medium having computer program instructions thereon for identifying and classifying process cycle images of a section of an image generating chip causing a process cycle failure, the instructions being executed on a processor to: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis; inputting an M×N analysis frame to a trained classifier; a trained classifier classifying the images of the image generating chip segments according to root causes of process failures among a plurality of failure categories; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0256] Article 83. Centering the J x K images in the M x N analysis frame 83. A non-transitory computer readable storage medium as recited in clause 82, further comprising a method for:

[0257] Clause 84. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a horizontal reflection to at least a portion of the particular J×K image; 83. A non-transitory computer readable storage medium as recited in clause 82, further comprising a method for:

[0258] Clause 85. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a vertical reflection to at least a portion of the particular J×K image; 83. A non-transitory computer readable storage medium as recited in clause 82, further comprising a method for:

[0259] Clause 86. The step of creating an input to a trained classifier comprises: generating a J×K image by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 83. A non-transitory computer readable storage medium as recited in clause 82, further comprising a method for:

[0260] Clause 87. The non-transitory computer-readable storage medium of clause 82, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 times per side.

[0261] Clause 88. The non-transitory computer-readable storage medium of clause 87, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0262] Clause 89. The non-transitory computer-readable storage medium of clause 82, wherein the plurality of failure categories includes at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0263] Clause 90. The non-transitory computer-readable storage medium of clause 82, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0264] Clause 91. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions that, when executed on the processor, identify and classify process cycle images of a section of an image generating chip causing a process cycle failure, the instructions: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; and filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N analysis. inputting an M×N analysis frame to a trained classifier; inputting an image of a segment of an imaging chip into a trained classifier, the trained classifier classifying the image of a segment of an imaging chip according to a root cause of a process failure from among a plurality of failure categories; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0265] Article 92. Centering J×K images in an M×N analysis frame 92. The system of claim 91, further performing operations including:

[0266] Clause 93. Using at least a portion of the particular J×K image to fill in around an edge of the particular J×K image Applying horizontal reflection to at least a portion of a particular J×K image 92. The system of claim 91, further performing operations including:

[0267] Clause 94. Using at least a portion of a particular J×K image to fill in around an edge of the particular J×K image. Applying a vertical reflection to at least a portion of a particular J×K image 92. The system of claim 91, further performing operations including:

[0268] Clause 95. Creating an input to a trained classifier Generate a J×K image by cropping a portion from a larger image and place the cropped J×K portions in an M×N frame. 92. The system of claim 91, further performing operations including:

[0269] Clause 96. The system of clause 91, wherein the J×K image is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0270] Clause 97. The system described in clause 96, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0271] Clause 98. The system of clause 91, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0272] Clause 99. The system of clause 91, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause. Combined, single-pass good vs. bad detection and root cause of process failure Training combinations

[0273] Clause 100. A method for training a convolutional neural network to identify and classify images of segments of an image generating chip that result in process cycle failures, comprising: using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimensions M×N; creating a training data set using labeled images of dimension J×K that are normal and belong to at least one failure category of a plurality of failure categories that result in a process failure; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, This allows the trained classifier to accept images of a section of an image generating chip and classify the images as normal or as belonging to at least one failure category of a plurality of failure categories that result in a process failure.

[0274] Clause 101. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, applying horizontal reflections to at least a portion of the particular J×K labeled image; The method of claim 100, further comprising:

[0275] Clause 102. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, Applying vertical reflections to at least a portion of the particular J×K labeled image. The method of claim 100, further comprising:

[0276] Clause 103. A step of creating a training data set using labeled images of dimension J×K, generating J×K labeled images by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; The method of claim 100, further comprising:

[0277] Clause 104. The method of clause 100, wherein a labeled image of dimension J×K is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image may be reduced by a factor of 1 / 2 to 1 / 50 per side.

[0278] Clause 105. The method of clause 104, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0279] Clause 106. The method of clause 100, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0280] Clause 107. The method of clause 100, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0281] Clause 108. A non-transitory computer-readable storage medium having computer program instructions thereon for training a convolutional neural network to identify and classify images of segments of an image generating chip that result in process cycle failures, the instructions, when executed on a processor, using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimensions M×N; creating a training data set using labeled images of dimension J×K that are normal and belong to at least one failure category of a plurality of failure categories that result in a process failure; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, A non-transitory computer-readable storage medium implementing a method whereby a trained classifier can accept an image of a section of an image generating chip and classify the image as normal or as belonging to at least one failure category of a plurality of failure categories that result in a process failure.

[0282] Clause 109. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, applying horizontal reflections to at least a portion of the particular J×K labeled image; 109. A non-transitory computer-readable storage medium as recited in Clause 108, further comprising:

[0283] Clause 110. The step of filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, Applying vertical reflections to at least a portion of the particular J×K labeled image. 109. A non-transitory computer-readable storage medium as recited in Clause 108, further comprising:

[0284] Clause 111. A step of creating a training data set using labeled images of dimension J×K, generating J×K labeled images by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 109. A non-transitory computer-readable storage medium as recited in Clause 108, further comprising:

[0285] Clause 112. The non-transitory computer-readable storage medium of clause 108, wherein the labeled image of dimension J×K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0286] Clause 113. The non-transitory computer-readable storage medium of clause 112, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0287] Clause 114. The non-transitory computer-readable storage medium of clause 108, wherein the plurality of failure categories includes at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0288] Clause 115. The non-transitory computer-readable storage medium of clause 108, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0289] Clause 116. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions to train a convolutional neural network to identify and classify images of segments of an image generating chip that result in process cycle failures, the instructions, when executed on the processor, using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training data set using labeled images of dimension J×K that are normal and belong to at least one failure category of a plurality of failure categories that result in a process failure; the labeled images are from a section of an imaging chip; Position the J×K labeled images at multiple positions in the M×N frame. using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N frame; Further training the pre-trained convolutional neural network to generate a classification classifier using the training dataset; and storing the coefficients of the trained classifier to identify and classify images of a section of image generating chips from a production process cycle; Including, This results in a system in which the trained classifier performs an operation in which it accepts an image of a section of an image generating chip and can classify the image as normal or as belonging to at least one of a plurality of failure categories that result in a process failure.

[0290] Clause 117. Using at least a portion of a particular J×K labeled image to fill in around an edge of a particular J×K labeled image, Applying horizontal reflections to at least a portion of a particular J×K labeled image The system of clause 116, further performing operations including:

[0291] Clause 118. Using at least a portion of a particular J×K labeled image to fill in around an edge of a particular J×K labeled image, Applying vertical reflections to at least a portion of a particular J×K labeled image. The system of clause 116, further performing operations including:

[0292] Clause 119. A training data set is created using labeled images of dimension J × K. Generate J×K labeled images by cropping parts from a larger image and placing the cropped J×K parts into an M×N frame. The system of clause 116, further performing operations including:

[0293] Clause 120. The system of clause 116, wherein the labeled image of dimension J×K is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0294] Clause 121. The system of clause 120, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0295] Clause 122. The system of clause 116, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0296] Clause 123. The system of clause 116, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause. Combined reasoning

[0297] Clause 124. A method for identifying process cycle images of a section of an image generating chip causing a process cycle failure, comprising: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis; inputting an M×N analysis frame to a trained classifier; a trained classifier classifying images of the sections of the image generating chip as normal or as belonging to at least one of a plurality of failure categories; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0298] Clause 125. The method of clause 124, further comprising the step of centering the J×K image in the M×N analysis frame.

[0299] Clause 126. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a horizontal reflection to at least a portion of the particular J×K image; 125. The method of claim 124, further comprising:

[0300] Clause 127. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a vertical reflection to at least a portion of the particular J×K image; 125. The method of claim 124, further comprising:

[0301] Clause 128. The step of creating an input to a trained classifier comprises: generating a J×K image by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 125. The method of claim 124, further comprising:

[0302] Clause 129. The method of clause 124, wherein an image of dimension J x K is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0303] Clause 130. The method according to clause 129, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0304] Clause 131. The method of clause 124, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0305] Clause 132. The method of clause 124, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0306] Clause 133. A non-transitory computer-readable storage medium having computer program instructions thereon, the instructions, when executed on a processor, for identifying a process cycle image of a section of an image generating chip that causes a process cycle failure, Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis; inputting an M×N analysis frame to a trained classifier; a trained classifier classifying images of the sections of the image generating chip as normal or as belonging to at least one of a plurality of failure categories; outputting the classification result of the image generating chip; A non-transitory computer-readable storage medium implementing a method comprising:

[0307] Clause 134. The non-transitory computer-readable storage medium of clause 133, further comprising the step of centering the J×K image in the M×N analysis frame.

[0308] Clause 135. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a horizontal reflection to at least a portion of the particular J×K image; 134. A non-transitory computer-readable storage medium according to claim 133, further comprising:

[0309] Clause 136. The step of filling around an edge of the particular J×K image using at least a portion of the particular J×K image, applying a vertical reflection to at least a portion of the particular J×K image; 134. A non-transitory computer-readable storage medium according to claim 133, further comprising:

[0310] Clause 137. The step of creating an input to a trained classifier comprises: generating a J×K image by cropping portions from a larger image and placing the cropped J×K portions into an M×N frame; 134. A non-transitory computer-readable storage medium according to claim 133, further comprising:

[0311] Clause 138. The non-transitory computer-readable storage medium of clause 133, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 times per side.

[0312] Clause 139. The non-transitory computer-readable storage medium of clause 138, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0313] Clause 140. The non-transitory computer-readable storage medium of clause 133, wherein the plurality of failure categories includes at least hybridization failures, space shift failures, offset failures, surface abrasion failures, and reagent flow failures.

[0314] Clause 141. The non-transitory computer-readable storage medium of clause 133, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0315] Clause 142. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions, which when executed on the processor, identify process cycle images of a section of an image generating chip causing a process cycle failure, the instructions comprising: Creating an input to a trained classifier, comprising: accessing an image of a partition having dimensions J×K; positioning the J×K images into an M×N analysis frame; and filling around an edge of the particular J×K labeled image using at least a portion of the particular J×K labeled image, thereby filling the M×N analysis. inputting an M×N analysis frame to a trained classifier; inputting an image of a section of the image generating chip, the image being classified by a trained classifier as being normal or belonging to at least one of a plurality of failure categories; Outputting the classification result of the image generating chip. A system for performing an operation, comprising:

[0316] Clause 143. The system of clause 142, further comprising centering the J×K image in the M×N analysis frame.

[0317] Clause 144. Using at least a portion of a particular J×K image to fill in around an edge of the particular J×K image. Applying horizontal reflection to at least a portion of a particular J×K image The system of claim 142, further performing operations including:

[0318] Clause 145. Using at least a portion of a particular J×K image to fill in around an edge of a particular J×K image Applying a vertical reflection to at least a portion of a particular J×K image The system of claim 142, further performing operations including:

[0319] Clause 146. Creating inputs to a trained classifier Generate a J×K image by cropping a portion from a larger image and place the cropped J×K portions in an M×N frame. The system of claim 142, further performing operations including:

[0320] Clause 147. The system of clause 142, wherein the J×K image is obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

[0321] Clause 148. The system of clause 147, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

[0322] Clause 149. The system of clause 142, wherein the plurality of failure categories includes at least hybridization failure, space shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0323] Clause 150. The system of clause 142, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause. Computer Systems

[0324] 17 is a simplified block diagram of a computer system 1700 that may be used to implement the disclosed techniques. A computer system typically includes at least one processor 1772 that communicates with a number of peripheral devices via a bus subsystem 1755. These peripheral devices may include, for example, a storage subsystem 1710 including a memory subsystem 1722 and a file storage subsystem 1736, user interface input devices 1738, user interface output devices 1776, and a network interface subsystem 1774. The input and output devices enable user interaction with the computer system. The network interface subsystem provides an interface to external networks, including interfaces to corresponding interface devices in other computer systems.

[0325] In one embodiment, a good vs. bad classifier 151 for classifying bad images is communicatively linked to the storage subsystem and the user interface input device.

[0326] User interface input devices 1738 may include pointing devices such as keyboards, mice, trackballs, touch pads, or graphics tablets, scanners, touch screens integrated into displays, audio input devices such as voice recognition systems and microphones, as well as other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and manners for inputting information into a computer system.

[0327] The user interface output devices 1776 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem can also provide non-visual displays such as an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system to a user or to another machine or computer system.

[0328] The storage subsystem 1710 stores programming and data constructs that provide the functionality of some or all of the modules and methods described herein. These software modules are generally executed by the processor alone or in combination with other processors.

[0329] The memory used in the storage subsystem may include several memories, such as a main random access memory (RAM) 1732, a read only memory (ROM) 1734 with fixed instructions stored therein, for storing instructions and data during program execution. The file storage subsystem 1736 provides persistent storage for program and data files and may include a hard disk drive, associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored by the file storage subsystem in the storage subsystem or in other machines accessible by the processor.

[0330] The bus subsystem 1755 provides a mechanism for allowing the various components and subsystems of the computer system to communicate with each other as intended. Although the bus subsystem is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0331] The computer system itself can be of various types, including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a mainframe, a server farm, a loosely distributed set of loosely networked computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the description of the computer system depicted in Figure 17 is intended only as a specific example for purposes of illustrating the disclosed technology. Many other configurations of a computer system can have more or fewer components than the computer system depicted in Figure 17.

[0332] The computer system 1700 includes a GPU or FPGA 1778. It may also include machine learning processors hosted by machine learning cloud platforms such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's Tensor Processing Unit (TPU), rackmount solutions such as the GX4 Rackmount Series, GX8 Rackmount Series, NVIDIA DGX-1, Microsoft' Stratix V FPGA, Graphcore's Intelligent Processor Unit (IPU), Qualcomm's Zeroth Platform with Snapdragon processors, NVIDIA's Volta, NVIDIA's DRIVE PX, NVIDIA's JETSON TX1 / TX2 MODULE, Intel's Nirvana, Movidius VPU, Fujitsu DPI, ARM's DynamicIQ, IBM TrueNorth, and others. [Explanation of symbols]

[0333] 100 Systems 111 Genotyping Equipment 115 Process Cycle Image Database 117 Failure Category Label Database 138 Training Database 138 Process Cycle Image Database 138 Databases 151 Good vs. Bad Classifiers 151 Classifier 151 Output 155 Network 168 Reference Database 168 Databases 171 Root cause classifier 171 Classifier 171 Random Forest Classifier

Claims

1. 1. A method for training a convolutional neural network to identify and classify images of segments of an imaging chip that result in process failures, comprising: using the pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training data set using labeled images of dimension J×K smaller than said dimensions M×N depicting process successes and failures; the labeled image is from the section of the imaging chip; positioning the J×K labeled images at multiple locations within at least one M×N frame; using at least a portion of a particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the at least one M×N frame; further training the pre-trained convolutional neural network using the training data set to generate a trained classifier; storing coefficients of the trained classifier to identify and classify images of segments of the imaging chip from a production process cycle; A method comprising:

2. The method of claim 1, wherein the trained classifier is configured to accept images of the section of the image generating chip and classify the images as depicting process success and failure.

3. The step of filling around the edge of the particular JxK labeled image using the at least a portion of the particular JxK labeled image, applying a horizontal reflection to said at least a portion of said particular J×K labeled image. The method of claim 1 further comprising:

4. The step of filling around the edge of the particular JxK labeled image using the at least a portion of the particular JxK labeled image comprises: applying a vertical reflection to said at least a portion of said particular J×K labeled image. The method of claim 1 further comprising:

5. The step of creating the training data set using labeled images of dimension J×K, generating the J×K labeled image by cropping a portion from a larger image and placing the cropped J×K portion in the at least one M×N frame; The method of claim 1 further comprising:

6. The method of claim 1, wherein the labeled image of dimension J x K is obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

7. The method of claim 6 , wherein the high resolution image acquired from the scanner has a size of 3600×1600 pixels.

8. The method of claim 7, further comprising the step of identifying a process cycle image of a section of the image generating chip that causes a process cycle failure, the identifying step comprising: Creating an input to the trained classifier, accessing the image of the partition having the dimension J×K; positioning the J×K images into an M×N analysis frame; inputting the M×N analysis frames into the trained classifier; applying the trained classifier to classify the images of the section of the imaging chip as good or bad; outputting a classification result of the section of the image generating chip; The method of claim 1 , comprising:

9. The method of claim 8, further comprising: using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis frame. The method of claim 8 further comprising:

10. At least one non-transitory computer readable storage medium having computer program instructions thereon for identifying and classifying images of sections of an imaging chip that result in process failure, the instructions, when executed on at least one processor, causing the at least one processor to: using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training data set using labeled images of dimension J×K smaller than said dimension M×N depicting process successes and failures; the labeled image is from the section of the imaging chip; positioning the J×K labeled images at multiple locations within at least one M×N frame; using at least a portion of a particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the at least one M×N frame; further training the pre-trained convolutional neural network using the training data set to generate a trained classifier; storing coefficients of the trained classifier to identify and classify images of a section of the imaging chip from a production process cycle; A non-transitory computer-readable storage medium configured to cause

11. The non-transitory computer-readable storage medium of claim 10, wherein the trained classifier is configured to accept the images of the section of the image generating chip and classify the images as depicting process success and failure.

12. The at least one processor configured to fill around the edges of the particular JxK labeled image using the at least a portion of the particular JxK labeled image, 11. The non-transitory computer-readable storage medium of claim 10, further comprising the at least one processor configured to apply a horizontal reflectance to the at least a portion of the particular JxK labeled image.

13. The at least one processor configured to fill around the edges of the particular JxK labeled image using the at least a portion of the particular JxK labeled image, 11. The non-transitory computer-readable storage medium of claim 10, further comprising the at least one processor configured to apply a vertical reflectance to the at least a portion of the particular JxK labeled image.

14. The at least one processor configured to create the training data set using labeled images of dimension J×K, 11. The non-transitory computer-readable storage medium of claim 10, further comprising: the at least one processor configured to generate the JxK labeled image by cropping a portion from a larger image and place the cropped JxK portion in the at least one MxN frame.

15. The non-transitory computer-readable storage medium of claim 10, wherein the labeled image of dimension J x K is configured to be obtained by downsampling a high-resolution image from a scanner, such that the resolution of the high-resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

16. 16. The non-transitory computer-readable storage medium of claim 15, wherein the high resolution image acquired from the scanner has a size of 3600 x 1600 pixels.

17. The instructions to the at least one processor: Creating an input to the trained classifier, accessing the image of the partition having the dimension J×K; positioning the J×K images in an M×N analysis frame; inputting the M×N analysis frames into the trained classifier; applying the trained classifier to classify the images of the section of the imaging chip as good or bad; outputting a classification result of the division of the image generating chip; The non-transitory computer-readable storage medium of claim 10 , further configured to:

18. The instructions to the at least one processor: using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis frame.

20. The non-transitory computer-readable storage medium of claim 17, further configured to:

19. 1. A system including one or more processors coupled to a memory, the memory loaded with computer instructions for identifying and classifying images of sections of an imaging chip that result in process failure, the instructions, when executed on the one or more processors, causing the one or more processors to: using a pre-trained convolutional neural network to extract image features, the pre-trained convolutional neural network accepting images of dimension M×N; creating a training data set using labeled images of dimension J×K smaller than said dimension M×N depicting process successes and failures; the labeled image is from the section of the imaging chip; positioning the J×K labeled images at multiple locations within at least one M×N frame; using at least a portion of a particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the at least one M×N frame; further training the pre-trained convolutional neural network using the training data set to generate a trained classifier; storing coefficients of the trained classifier to identify and classify images of a section of the imaging chip from a production process cycle; A system configured to:

20. The system of claim 19, wherein the trained classifier is configured to accept images of sections of the image generating chip and classify them as depicting process success and failure.

21. The instructions configured to cause the one or more processors to fill around the edge of the particular JxK labeled image using the at least a portion of the particular JxK labeled image, comprising: instructions configured to cause the one or more processors to apply a horizontal reflection to the at least a portion of the particular J×K labeled image; 20. The system of claim 19, comprising:

22. The instructions configured to cause the one or more processors to fill around the edge of the particular JxK labeled image using the at least a portion of the particular JxK labeled image, comprising:

20. The system of claim 19, comprising instructions configured to cause the one or more processors to apply a vertical reflectance to the at least a portion of the particular JxK labeled image.

23. the instructions configured to cause the one or more processors to create the training data set using labeled images of dimension J×K, 20. The system of claim 19, further comprising instructions configured to cause the one or more processors to generate the JxK labeled image by cropping a portion from a larger image and place the cropped JxK portion in the at least one MxN frame.

24. The system described in claim 19, wherein the labeled image of dimension J x K is configured to be obtained by downsampling a high resolution image from a scanner, such that the resolution of the high resolution image is reduced by a factor of 1 / 2 to 1 / 50 per side.

25. The system described in claim 24, configured so that the high-resolution image is obtained from the scanner to have a size of 3600 x 1600 pixels.

26. The instructions to the one or more processors: Creating an input to the trained classifier, accessing the image of the partition having the dimension J×K; positioning the J×K images in an M×N analysis frame; inputting the M×N analysis frames into the trained classifier; applying the trained classifier to classify the images of the section of the imaging chip as good or bad; outputting a classification result of the division of the image generating chip; 20. The system of claim 19, configured to:

27. The instructions to the one or more processors: using at least a portion of the particular J×K labeled image to fill around an edge of the particular J×K labeled image, thereby filling the M×N analysis frame.

27. The system of claim 26, further configured to:

Images