Root Cause Analysis of Process Cycle Images Based on Deep Learning

By combining principal component analysis and deep learning models with feature engineering techniques, the problem of identifying failure causes in the genotyping process was solved, enabling rapid and accurate failure image classification and cause analysis, thus improving production efficiency.

CN115812223BActive Publication Date: 2024-11-15ILLUMINA INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280005244.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-01-29
Filing Date
2022-01-28
Publication Date
2024-11-15
Estimated Expiration
2042-01-28

AI Technical Summary

Technical Problem

Existing technologies struggle to quickly and accurately identify the causes of failure in the genotyping process, resulting in insufficient in-depth analysis of the reasons for failure in low-quality processes.

Method used

By combining principal component analysis (PCA), image segmentation, and deep learning models (such as convolutional neural networks, CNN) with feature engineering techniques, we analyze failed images during the genotyping process through image processing and classifier analysis to identify failure categories.

Benefits of technology

It enables rapid classification and accurate identification of failed images during genotyping, allowing for early detection of the causes of failure and reducing chemical waste and processing time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115812223B_ABST
    Figure CN115812223B_ABST
Patent Text Reader

Abstract

The disclosed technology involves training a convolutional neural network (CNN) to identify and classify images of segments of an image generating chip that cause process cycle failures. The disclosed technology includes creating a training data set of images of size M×N using labeled images of segments of an image generating chip of size J×K. The disclosed technology can fill the M×N frames along the edges of the J×K labeled images positioned in the M×N frames using horizontal and vertical reflections. The pre-trained CNN is further trained using the training data set. The trained CNN can classify the segment images as normal or depict failures. The disclosed technology can train a root cause CNN to classify process cycle images of segments that cause process cycle failures. The trained CNN can classify the segment images in multiple failure categories by the root cause of the process failure.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Priority application

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 143,673 (Attorney Docket No. ILLM1044-1 / IP-2089-PRV), entitled “DEEP LEARNING-BASED ROOT CAUSE ANALYSIS OF PROCESS CYCLES,” filed on January 29, 2021. This provisional application is incorporated herein by reference for all purposes.

[0003] Literature Incorporation

[0004] The following materials are incorporated by reference as if fully set forth herein:

[0005] U.S. Patent Application No. 171 / 161,595, entitled “MACHINE LEARNING-BASED ROOT CAUSE ANALYSIS OF PROCESS CYCLE IMAGES,” filed on January 28, 2021 (Attorney Docket No. ILLM 1026-2 / IP-1911-US);

[0006] U.S. Patent Application No. 17 / 332,904, entitled “MACHINE LERNING-BASED ANALYSIS OF PROCESSINDICATORS TO PREDICT SAMPLE REEVALUATION SUCCESS,” filed on May 27, 2021 (Attorney Docket No. ILLM 1027-2 / IP-1973-US);

[0007] U.S. patent application No. 17 / 548,424 (attorney docket No. ILLM 1028-2 / IP-1978-US), filed on December 10, 2021, and entitled “MACHINE LEARNING-BASED GENOTYPING PROCESS OUTCOME PREDICTION USING AGGREGATE METRICS.” Technical Field

[0008] The disclosed technology relates to classification of images for evaluation and root cause failure analysis of production processes. Background Art

[0009] The subject matter discussed in this section should not be considered to be prior art simply because it is mentioned in this section. Similarly, problems mentioned in this section or associated with the subject matter provided as background technology should not be considered to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which themselves may also correspond to specific implementations of the technology protected by the claims.

[0010] Genotyping is a multi-day process. The process is susceptible to mechanical and chemical handling errors. The collected samples for genotyping are extracted and distributed in segments and regions of the image generation chip. The samples are then chemically processed through multiple steps to generate fluorescent images. The process generates a quality score for each segment analyzed. The quality score does not provide insight into the root cause of failure of low-quality processes. In some cases, failed segment images still produce acceptable quality scores.

[0011] Therefore, there is an opportunity to introduce new methods and systems to evaluate segment images and determine the root cause analysis of failures during production genotyping. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the drawings, similar reference symbols generally refer to similar parts throughout the different views. In addition, the drawings are not necessarily drawn to scale, but rather focus on illustrating the principles of the disclosed technology. In the following description, various specific implementations of the disclosed technology are described with reference to the following drawings, in which:

[0013] Figure 1 An architectural level schematic of a system is shown in which process cycle images from a genotyping instrument are classified and the root cause of bad or failed images is determined.

[0014] Figure 2 Illustrated Figure 1 Subsystem components of the feature generator.

[0015] Figure 3 The process steps of an exemplary genotyping process are presented.

[0016] Figure 4 An image of the segment arranged in the image generating chip after successful processing completion is presented.

[0017] Figure 5A and Figure 5B Examples of failed segment images due to hybridization failure during the genotyping process are presented.

[0018] Figure 5C and Figure 5D An example of a failed segment image due to a failed spacer shift is presented.

[0019] Figure 5E An example of a failed segment image due to a migration failure is presented.

[0020] Fig. 5F Examples of segment images of failures due to surface abrasion failure are presented.

[0021] Figure 5G and Figure 5H An example of a segment image of a failure due to reagent flow failure is presented.

[0022] Fig.5I An example of a failed segment image where the source of the failure is unknown is presented.

[0023] Fig. 6A is an example of 96 intrinsic image components selected by using ranked principal components generated using principal component analysis (PCA).

[0024] Figure 6B is from Fig. 6A A magnified image of the first 40 intrinsic image components selected from the 96 images in .

[0025] Fig. 7A Illustrated is image scaling and flattening of segment images for input to principal component analysis.

[0026] Figure 7B The use of principal component analysis to create a basis for eigenimages is illustrated.

[0027] Fig. 8A An example of feature generation using the intensity of the region of the segment is illustrated.

[0028] Figure 8B Another example of feature generation using the intensity of the region of the segment is illustrated.

[0029] Fig. 9 is a graphical illustration of an One-vs-All (OvR) classifier.

[0030] Fig. 10A The use of labeled training data comprising process cycle images to train a binary (good vs. bad) classifier and a multi-class (root cause) classifier is illustrated.

[0031] Fig. 10B A two-step process is illustrated, where production process cycle images are classified as good or bad, and the failure class of the bad images is further determined.

[0032] Fig.11A and Fig. 11B Examples of feature engineering techniques applied to training data images are presented.

[0033] Fig. 12A Illustrate the transfer learning steps for training a convolutional neural network using fine-tuning.

[0034] Fig. 12B The network architecture of an exemplary convolutional neural network model is presented.

[0035] Fig. 12C An exemplary architecture of a first convolutional neural network for identifying and classifying bad process cycle images is presented.

[0036] Fig.12D and Fig.12E An exemplary architecture of a second convolutional neural network for identifying and classifying bad process cycle images is presented.

[0037] Fig.13 The performance of two exemplary convolutional neural networks using different feature engineering techniques is presented.

[0038] Fig.14 A summary of model performance metrics for failure categories is presented.

[0039] Fig.15 An example of a failed section image with mixed defects indicating manually annotated defect labels and predicted defect categories is presented.

[0040] Fig.16 A process flow diagram for training good and bad classifiers and applying the trained classifiers to classify images of segments of an image generating chip is presented.

[0041] Fig.17 is a simplified block diagram of a computer system that can be used to implement the disclosed techniques. DETAILED DESCRIPTION

[0042] The following discussion is presented to enable any person skilled in the art to make and use the disclosed technology, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the specific implementations shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0043] Introduction

[0044] The disclosed technology applies visual systems and image classification to the assessment and root cause failure analysis of production genotyping. Three different methods are described, the first method involves intrinsic images, the second method is based on threshold processing by region, and the third method uses deep learning models such as convolutional neural networks (or CNN). Principal component analysis (PCA) and non-negative matrix factorization (NMF) belong to the disclosed technology. Other dimensionality reduction techniques that can be applied to images include independent component analysis, dictionary learning, sparse principal component analysis, factor analysis, small batch K means. Variations of image decomposition and dimensionality reduction techniques can be used. For example, PCA can be implemented using singular value decomposition (SVD) or as kernel PCA. The output of these techniques is provided to a classifier as input. The applied classifier may include random forests, K nearest neighbors (KNN), multinomial logistic regression, support vector machines (SVM), gradient boosted trees, naive Bayes, etc. As the larger body of labeled images becomes available, convolutional neural networks such as ResNet, VGG, ImageNet can also be used as presented in the description of the third image processing technology below.

[0045] The genotyping production process is susceptible to mechanical and chemical handling errors. The collected sample is extracted, distributed in segments and areas of the BeadChip, and then chemically processed through multiple steps to generate a fluorescent image. The final fluorescent image, and even intermediate fluorescent images, can be analyzed to monitor production and perform failure analysis.

[0046] The vast majority of production assays are successful. Failed assays are currently understood to fall into five categories plus a remaining failure category. The five failure categories are hybridization or mixing failure, spacer shift failure, excursion failure, surface abrasion failure, and reagent flow failure. The remaining categories are unhealthy patterns due to mixing effects, unidentified causes, and weak signals. Over time, more distinct causes may be identified, especially as root cause analysis leads to improved production.

[0047] The first image processing techniques applied to quality control and failure analysis evolved from facial recognition by intrinsic facial analysis. From tens of thousands of labeled images, a linear basis of 40 to 100 or more image components was identified. One way to form the eigenbasis is to perform principal component analysis (PCA) and subsequently rank the components according to a measure of explainable variability. It was observed that 40 components explained most of the variability. Beyond 100 components, additional components appear to reflect noise patterns or natural variability in sample processing. The number of relevant components is expected to be affected by the image resolution. Here, resolution reduction was applied so that a segment of the image generation chip was analyzed at a resolution of 180×80 pixels. This was sufficient to resolve the distinction between successful and unsuccessful production and then classify the root causes of failure in six failure categories. No formal sensitivity analysis was applied, but it is expected that images with slightly lower resolution can also work, and images with resolutions 4 to 22 times this resolution can be processed in the same way, but with increased computational cost. Each image to be analyzed by intrinsic image analysis is represented as a weighted linear combination of the basis images. Each weight for the ordered set of basis components is used as a feature for training a classifier. For example, in one implementation, a random forest classifier is trained using 96 weights of the components of the labeled images. The random forest classifier has 200 trees and a depth of 20. The random forest classifier performs two tasks: separation of successful and unsuccessful production images and then root cause analysis of unsuccessful production images. This two-stage classification was chosen because successful production runs predominate, but one-stage classification could also be used.

[0048] The second image processing technique applied involves thresholding of image regions. A production image of a segment of an image generating chip captures several physically separated regions. Structures bordering the segments and structures of the physical regions separating the segments can be seen in the production image. The thresholding strategy includes separating the valid regions from the boundary structures and then distinguishing between the separated regions. Optionally, structures separating the physical regions can also be filtered out of the image. At least the valid regions are thresholded for luminescence. The thresholding determines how many valid regions are producing the expected signal intensity. After thresholding, the success or failure of each valid region is evaluated. The failure modes between regions and segments of the image generating chip can be further evaluated for root cause classification.

[0049] Processing of production images to detect failed production runs and determine root cause can be performed immediately during the production process, even faster than reading the results from the image generating chip and judging the quality. This image processing can be done faster because reducing the pixel size of the image to 1 / 20 of the original size greatly reduces the computational requirements, and direct processing of the reduced resolution image does not require correlating individual luminescent pixels in a region with individual probes. The rapid turnaround of root cause analysis can be used to correct upstream processes before chemicals and processing time are wasted.

[0050] The third image processing technique involves applying deep learning models, such as convolutional neural networks (CNNs). ResNet (He et al., CVPR 2016, in<arxiv.org / abs / 1512.03385> >) and VGG (Simonyan et al., 2015, available at <<arxiv.org / abs / 1409.1556> > Available at http: / / www.cnbc.com / research / convolutional-neural-networks / ) is an example of a convolutional neural network (CNN) for recognition and classification. We applied the ResNet-18 and VGG-16 architectures of the corresponding models to detect failed images and classify failed images into corresponding failure categories. The CNN model parameters are pre-trained on the ImageNet dataset (Deng et al., 2009, "ImageNet: A large-scale hierarchical image database", published in the conference paper of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 248-255), which contains about 14 million images. The pre-trained model is fine-tuned using labeled images of segments of the image generation chip. Labeled images of approximately 75,000 segments are used to fine-tune the pre-trained CNN. The training data consists of normal images from successful process cycles and abnormal (or bad, or failed) images from failed process cycles. Images from failed process cycles belong to the five failure categories presented above.

[0051] ResNet-18 and VGG-16 CNN models can use square input images of size 224×224 pixels. In one specific implementation, the segments of the image generation chip are rectangular, such as 180×80 pixels as described above. Segments of larger image sizes can be used. The disclosed technology applies feature engineering to create training data using rectangular labeled images, which can be smaller than the 224×224 pixel size images required as input to the CNN model. The technology can apply three feature engineering techniques to create the input data set, including cropping, zero padding, and reflection padding.

[0052] In cropping, the center portion of the rectangular-shaped segment image of the image generation chip is cropped to a size of 224×224 pixels. In this case, the input image of the segment is larger than 224×224 pixels. In one specific implementation, the input segment image has a size of 504×224 pixels. Other sizes of the labeled image larger than 224×224 can be used to crop out the square portion. The input image is cropped to match the input image size required by the CNN model (224×224 pixels).

[0053] In zero padding, the input image size of the marker is less than 224×224 pixels. For example, the input image can be 180×80 pixels. The smaller size (such as 180×80 pixels) of the input marker image is placed in an analysis frame of 224×224 pixels. The image can be placed at any position inside the analysis frame. The pixels in the surrounding area in the analysis frame can be zero-padded, or in other words, the surrounding pixels are assigned zero image intensity values. The larger size analysis frame can then be given as input to the CNN model.

[0054] In reflection filling, a smaller sized labeled input image can be placed in the center of a larger sized analysis frame. The labeled image is then reflected horizontally and vertically along the edges to fill the surrounding pixels in the larger sized analysis frame (224×224 pixels). The reflected labeled image is given as input to the CNN model. Reflection filling can produce better results when features in the input labeled image are replicated at multiple locations in the larger sized analysis frame.

[0055] The disclosed technology can perform data augmentation to increase the size of the training data. Horizontal and vertical translation can be performed by placing a rectangular labeled input image of a smaller size (J×K pixels) at multiple locations in an analysis frame of a larger size (M×N pixels). In one specific implementation, the rectangular labeled input image has a size of 180×80 pixels, and the larger size analysis frame is 224×224 pixels. Labeled input images and analysis frames of other sizes can be used. In one specific implementation, J×K images can be systematically translated horizontally, vertically, or diagonally in an M×N analysis frame to generate additional training data. In one specific implementation, J×K images can be randomly positioned at different locations in an M×N analysis frame to generate additional training data.

[0056] A two-step detection and classification process can be applied using two separately trained convolutional neural networks (CNNs). The first CNN is trained for the detection task, where the trained classifier can classify an image as normal or depicting a process failure. The failed process cycle images can be fed to the second CNN, trained to classify the images by the root cause of the process failure. In one specific implementation, the system can classify the images into the five different types of failure types listed above. The two-step process can be combined in a one-step process using the CNN to classify the production section image as normal or belonging to one of the failure categories.

[0057] environment

[0058] A system for early prediction of failures in genotyping systems is described. Genotyping is the process of determining differences in the genetic makeup (genotype) of individuals by examining their DNA sequence using a bioassay and comparing it to a reference sequence. Genotyping enables researchers to explore genetic variants, such as single nucleotide polymorphisms (SNPs) and structural changes in DNA. Reference Figure 1 To describe the system, the figure shows an architectural level diagram of the system according to a specific implementation. Figure 1 This is an architectural diagram, so some details are intentionally omitted to improve the clarity of the description. Figure 1 The discussion of the system is organized as follows. First, the elements of the drawings are described, followed by a description of their interconnections. Then, the purpose of the elements in the system is described in more detail.

[0059] Figure 1 Included is system 100. This paragraph names the labeled portions of system 100. The figure illustrates genotyping instrument 111, process cycle image database 115, failure class label database 117, labeled process cycle image database 138, trained good and bad classifier 151, feature image base database 168, trained root cause classifier 171, feature generator 185, and network 155.

[0060] The disclosed technology is applicable to various genotyping instruments 111, also referred to as genotyping scanners and genotyping platforms. A network 155 communicatively couples the genotyping instruments 111, the process cycle image database 115, the failure class label database 117, the labeled process cycle image database 138, the trained good and bad classifiers 151, the feature image base database 168, the trained root cause classifiers 171, and the feature generator 185.

[0061] Genotyping instruments may include Illumina's BeadChip imaging systems, such as the ISCAN TMSystem. The instrument can detect the fluorescence intensity of hundreds to millions of microbeads, which are arranged in sections on the mapped positions on the image generation chip. The genotyping instrument may include an instrument control computer, which controls various aspects of the instrument, such as laser control, precision mechanical control, detection of excitation signals, image registration, image extraction, and data output. The genotyping instrument can be used in a variety of physical environments and is operated by technicians of different skill levels. Sample preparation can take two to three days and can include manual and automatic processing of samples.

[0062] Illustrated Figure 3 The exemplary genotyping process 300 is referred to as Illumina's INFINIUM TM Determination workflow. The process is designed to study many SNPs at a wide range of locus multiplexing levels. Using a single microbead type and a dual-color (such as red and green) channel method, the process scales genotyping from hundreds to millions of SNPs per sample. The process starts with the addition and extraction of DNA samples. The process can be operated with relatively low input samples such as 200ng, which can determine millions of SNP sites. The sample is amplified. The amplification process may take several hours to overnight to complete. The amplified sample undergoes controlled enzyme fragmentation. Then alcohol precipitation and resuspension are performed. An image generation chip is prepared in a capillary flow chamber for hybridization. The sample is then applied to the prepared image generation chip and incubated overnight. During this overnight hybridization, the sample is annealed to a locus-specific 50-mer covalently linked to up to millions of microbead types. One microbead type corresponds to each allele of each SNP locus. Allele specificity is conferred by enzymatic base extension and subsequent fluorescent staining. Genotyping instruments or scanners (such as ISCAN TM The system detects the fluorescence intensity of the beads and performs genotyping calls.

[0063] In one example, the results of genotyping are presented using a metric called "call rate". This metric represents the percentage of genotypings that were correctly scanned on the image generating chip. A separate call rate is reported for each segment of the image generating chip. Thresholds can be used to accept or reject results. For example, a call rate of 98% or higher can be used to accept the genotyping results for a segment. Different thresholds can be used, such as below 98% or above 98%. If the call rate for a segment is below a threshold, the genotyping process is considered to have failed. The genotyping process can span many days, and therefore the cost of repetition is very high. Failures in the genotyping process can occur due to operational errors (such as mechanical or handling errors) or chemical processing errors.

[0064] The genotyping system can provide process cycle images of segments of the image generating chip and the corresponding call rates of the segments when the genotyping process is completed. The disclosed technology can process these segment images to classify whether the genotyping process is successful (good image of the segment) or unsuccessful (bad or failed image of the segment). The disclosed technology can further process the bad or failed images to determine the failure category. Currently, the system can classify failed images into one of six failure categories: hybridization or mixing failure due to mixing effects, unknown reasons, weak signals, etc., spacer shift failure, offset failure, surface abrasion failure, reagent flow failure, and overall unhealthy images. Over time, especially when root cause analysis leads to improved production, more different reasons can be identified.

[0065] Reference now Figure 1 100. The failure class labels of the six failure types can be stored in the failure class label database 117. The training data set of the labeled process image cycle is stored in the database 138. The labeled training examples can include successful (good) and unsuccessful (bad) process cycle images. Unsuccessful process cycle images are marked as belonging to one of the six failure categories listed above. In one specific implementation, the training database 138 includes at least 20,000 training examples. In another specific implementation, the size of the training data set is increased to as many as 75,000 training examples using feature engineering techniques. The size of the training database can be increased because more labeled image data is collected from the laboratory using a genotyping instrument.

[0066] The disclosed technology includes three independent image processing techniques for extracting features from process cycle images. A feature generator 185 can be used to apply one of the three techniques to extract features from process cycle images for input to a machine learning model. The first image processing technique is developed from facial recognition through eigenfacial analysis. A relatively small number of linear bases, such as 40 to 100 or more image components, are identified from tens of thousands of labeled images. One method of forming the eigenbasis is principal component analysis (PCA). The production cycle images are represented as weighted linear combinations of the basis images for input to a classifier. For example, in one specific implementation, 96 weights for the components of the labeled images are used to train the classifier. The basis of the eigenimages can be stored in a database 168.

[0067] A second image processing technique to extract features includes thresholding of segment images. Production images of segments of an image generating chip capture several physically separated regions. Structures bordering the segments and structures separating the physical regions of the segments can be seen in the production images. The thresholding technique determines how many valid regions are producing the expected signal strength. The output of the thresholding technique can be provided as input to a classifier to distinguish between good and bad images. Failure modes between regions and segments of the image generating chip can be further evaluated for root cause analysis.

[0068] The third image processing technique includes various feature engineering techniques to prepare images for input to the deep learning model. The segment image of the image generation chip is rectangular in shape. The image generation chip may have 12, 24, 48, or 96 segments arranged in two or more columns. The convolutional neural network (CNN) applied by the disclosed technology requires a square-shaped input image. Therefore, the system includes logic to position the segment image of a rectangular shape (J×K pixels) into an analysis frame of a square shape (M×N pixels).

[0069] The system may apply one or more of the following feature engineering techniques. The system may apply zero padding to fill pixels surrounding the segment image in a larger square-shaped analysis frame. The system may crop a center portion of the segment image in a square size and fill the analysis frame with the cropped segment image. In this case, the segment image has a larger size than the analysis frame. When the labeled segment image is smaller than the analysis frame, the system may position the labeled input image within the analysis frame. The system may use horizontal reflections and vertical reflections along the edges of the input segment image to fill a larger sized analysis frame. The system may augment the labeled training data by using translations in which the labeled input image is positioned at multiple locations in the analysis frame.

[0070] The image features of the production images generated by the feature generator 185 are provided as input to the trained classifiers 151 and 171. Two types of classifiers are trained. The good and bad classifiers can predict successful and unsuccessful production images. The root cause analysis classifier can predict the failure category of unsuccessful images. In one specific implementation, the classifiers used by the disclosed technology include random forest classifiers. Other examples of classifiers that can be applied include K nearest neighbors (KNN), multinomial logistic regression, and support vector machines. In another specific implementation of the disclosed technology, a convolutional neural network (CNN) is applied to identify and classify images of segments of the image generating chip.

[0071] Finish Figure 1Description, the components of the above system 100 are all coupled to the network 155 in communication. The actual communication path can be point-to-point on a public and / or private network. Communication can occur on various networks, such as private networks, VPNs, MPLS circuits or the Internet, and can use appropriate application programming interfaces (APIs) and data exchange formats, such as representational state transfer (REST), JavaScript object notation (JSON), extensible markup language (XML), simple object access protocol (SOAP), Java message service (JMS) and / or Java platform module system. All communications can be encrypted. Communication is usually carried out via protocols such as EDGE, 3G, 4G LTE, Wi-Fi and WiMAX, through networks such as LAN (local area network), WAN (wide area network), telephone network (public switched telephone network (PSTN), session initiation protocol (SIP), wireless network, point-to-point network, star network, token ring network, hub network, Internet (including mobile Internet). Figure 1 The engines or system components are implemented by software running on different types of computing devices. Exemplary devices are workstations, servers, computing clusters, blade servers, and server farms. In addition, various authorization and authentication techniques, such as username / password, open authorization (OAuth), Kerberos, secure, digital certificates, etc., can be used to protect communications.

[0072] Feature Generator - System Components

[0073] Figure 2 is a high-level block diagram of the components of the feature generator 185. These components are used as follows in Fig.16 The feature generator 185 is computer-implemented using a variety of different computer systems presented in the description of . The illustrated components may be combined or further separated when implemented. The feature generator 185 consists of three high-level components that implement three image processing techniques: a feature generator 235 based on principal component analysis or PCA, a feature generator 255 based on image segmentation, and a training data generator for CNN 275. The PCA-based feature generator includes an image scaler 237 and an intrinsic image basis creator 239. The image segmentation-based feature generator 255 includes an image converter 257 and an intensity extractor 259. The training data generator for CNN 275 includes an image cropper 277, an image transformer 279, and an image reflector 281. In the following sections, further details of the specific implementation of these components are presented.

[0074] PCA-based feature generator

[0075] The first image processing technique is developed from face recognition through intrinsic face analysis. One method of forming an eigenbase is principal component analysis (PCA). A PCA-based feature generator 235 applies PCA to the resized processed image. An image scaler component 237 resizes the process cycle image. Scaling reduces the size of the processed image so that they can be processed by an intrinsic image base creator component 239 in a computationally efficient manner. The details of these components are presented in the following sections.

[0076] Image Resizer

[0077] Higher resolution images obtained from a genotyping instrument or scanner may require more computing resources to process. The images obtained from the genotyping scanner are resized by the image scaler 237 so that the image of the segment of the chip is analyzed at a reduced resolution such as 180×80 pixels. Throughout the text, we refer to scaling (or rescaling) when resampling the image. Resampling changes the number of pixels in the displayed image. When the number of pixels in the initial image increases, the image size increases, and it is called upsampling. When the number of pixels in the initial image decreases, the image size decreases, and it is called downsampling. In one specific implementation, the image of the segment obtained from the scanner is at a resolution of 3600×1600 pixels. In another specific implementation, the image of the segment obtained from the scanner is at a resolution of 3850×1600 pixels. These original images are downsampled to reduce the pixel size of the image from the original resolution to 1 / 20 on each side. This resolution is sufficient to distinguish between successful production images and unsuccessful production images, and then classify the root causes of failure in six failure categories. The image may be downsampled to 1 / 4 to 1 / 40 of the number of pixels on each side at the original resolution and processed in the same manner. In another specific implementation, the image may be downsampled to 1 / 2 to 1 / 50 of the number of pixels on each side at the original resolution and processed in the same manner. An exemplary technique for resampling a high resolution image is presented below.

[0078] The disclosed technology can apply various interpolation techniques to reduce the size of the production image. In one specific implementation, bilinear interpolation is used to reduce the size of the segment image. Linear interpolation is a curve fitting method that uses linear polynomials to construct new data points with a range of discrete sets of known data points. Bilinear interpolation is an extension of linear interpolation for interpolating functions of two variables (e.g., x and y) on a two-dimensional grid. Bilinear interpolation is first performed using linear interpolation in one direction and then similarly in the second direction. Although each step is linear in sample values ​​and positions, the interpolation as a whole is not linear but quadratic in sample positions. Other interpolation techniques can also be used to reduce the size (rescaling) of the segment image, such as nearest neighbor interpolation and resampling using pixel area relationships.

[0079] Intrinsic Image Basis Creator

[0080] The first image processing technique applied to segment images to generate input features for classifiers was developed from facial recognition through eigenfacial analysis. From tens of thousands of labeled images, a linear basis of 40 to 100 or more image components is identified. One method to form the basis of an eigenimage is principal component analysis (PCA). A set B of elements (vectors) in a vector space V is called a basis if each element of V can be written as a linear combination of elements of B in a unique way. Equivalently, if the elements of B are linearly independent, then B is a basis, and each element of V is a linear combination of elements of B. A vector space can have several bases. However, all bases have the same number of elements, called the dimension of the vector space. In our technique, the basis of the vector space is an eigenimage.

[0081] PCA is typically used to reduce the dimensionality of a d-dimensional dataset by projecting it onto a k-dimensional subspace, where k < d. For example, the resized and labeled images in a training database describe vectors in a 14,400-dimensional space (180 × 80 pixels), i.e., d = 14,400. In other words, an image is a point in a 14,400-dimensional space. Eigenspace-based methods approximate the image vectors with lower-dimensional feature vectors. The main assumption behind this technique is that the image space given by the feature vectors has a lower dimension than the image space given by the number of pixels in the image, and image recognition can be performed in this reduced space. The images of the sections of the image generation chip are similar in overall configuration and will not be randomly distributed in this huge space, and thus can be described by a relatively low-dimensional subspace. The PCA technique finds the vectors that best account for the distribution of the section images within the entire image space. These vectors define the subspace of the image, also known as the "image space". In our specific implementation, each vector describes an image of 180 × 80 pixels and is a linear combination of the images in the training data. In the following text, details of how to use principal component analysis (PCA) to create a basis of eigenimages are presented.

[0082] The PCA-based analysis of the labeled training images may include the following five steps.

[0083] Step 1: Access multidimensional related data

[0084] The first step in applying PCA is to access the high-dimensional data. In one instance, a PCA-based feature generator uses 20,000 labeled images as training data. The size of each image is resized to a resolution of 180 × 80 pixels and represented as a point in a 14,400-dimensional space, one dimension for each pixel. This technique can handle images with higher or lower resolutions than those specified above. As more labeled images are collected from the laboratory, it is expected that the size of the training dataset will increase.

[0085] Step 2: Standardization of data

[0086] Normalization (or Z-score normalization) is the process of rescaling features so that they have the properties of a Gaussian distribution with a mean equal to zero or μ = 0 and a standard deviation from the mean equal to 1 or σ = 1. Normalization is performed to build features that have similar ranges to each other. The standard score of an image can be calculated by subtracting the mean (of the image) from the image and dividing the result by the standard deviation. Since PCA produces a feature subspace that maximizes the variance along the axes, it helps to normalize the data so that it is centered on the axes.

[0087] Step 3: Calculate the covariance matrix

[0088] The covariance matrix is ​​a d×d matrix in d-dimensional space, where each element represents the covariance between two features. The covariance of two features measures their tendency to vary together. The variation is the average of the squared deviations of the feature from its mean. The covariance is the average of the product of the deviations of the feature values ​​from their mean. Consider feature k and feature j. Let {x(1,j),x(2,j),…,x(i,j)} be the set of i examples of feature j, and let {x(1,k),x(2,k),…,x(i,k)} be the set of i examples of feature k. Similarly, let is the mean of feature j, and is the mean of feature k. The covariance of feature j and feature k is calculated as follows:

[0089]

[0090] The calculation of the covariance matrix can be expressed via the following matrix equation:

[0091]

[0092] The average vector can be expressed as:

[0093]

[0094] The mean vector is a d-dimensional vector, where each value in the vector represents the sample mean of the feature column in the training data set. jk It can be found in “–(σ ij )(σ ik )" (i.e., inverse linear correlation) to "+(σ ij )(σ ik )” linear correlation. When there is no correlation between the two features, σ jk The value of is zero.

[0095] Step 4: Calculate eigenvectors and eigenvalues

[0096] The eigenvectors and eigenvalues ​​of the covariance matrix represent the core of PCA. The eigenvectors (or principal components) determine the direction of the new feature space, and the eigenvalues ​​determine their magnitude. In other words, the eigenvalues ​​explain the changes in the data along the axis of the new feature space. Eigendecomposition is a matrix decomposition method that represents a matrix by using the eigenvectors and eigenvalues ​​of the matrix. An eigenvector is defined as a vector that changes only in a scalar when a linear transformation is applied to it. If A is a matrix representing a linear transformation, v is an eigenvector and λ is the corresponding eigenvalue, then it can be expressed as Av=λv. A square matrix can have as many eigenvectors as its dimensions. If all eigenvectors are represented as columns of a matrix V and the corresponding eigenvalues ​​are represented as entries of a diagonal matrix L, the above equation can be expressed as AV=VL. In the case of a covariance matrix, all eigenvectors are orthogonal to each other and are the main components of the new feature space.

[0097] Step 5: Use explained variance to select the basis of the intrinsic image

[0098] For our specific implementation, the above steps can produce 14,400 principal components, equal to the dimension of the feature space. An eigenpair consists of an eigenvector and a scalar eigenvalue. The eigenpairs can be classified based on the eigenvalues, and a basis of eigenimages is created using a measure called "explained variance". The explained variance indicates how much information (or variance) can be attributed to each principal component. The results of the explainable measure values ​​can be plotted on a two-dimensional plot. The classified principal components are represented along the x-axis. A graph indicating the cumulative explained variance can be plotted. The top m components that represent the main part of the variance can be selected.

[0099] In our implementation, the first 40 components represent a high percentage of explained variance, therefore, the first 40 principal components were selected to form the basis of the new feature space. In other implementations, 25 to 100 principal components or more than 100 principal components, up to 256 or 512 principal components, can be selected to create a basis of intrinsic images. Each production image to be analyzed by intrinsic image analysis is represented as a weighted linear combination of the basis images. Each weight of the ordered set of basis components is used as a feature for training a classifier. For example, in one implementation, 96 weights of the components of the labeled image are used to train the classifier.

[0100] The disclosed technology can use other image decomposition and dimensionality reduction techniques. For example, compared to PCA, which learns a complete representation of an image, non-negative matrix factorization (NMF) learns a part-based representation of an image. Unlike PCA, NMF learns to represent an image with a set of basis images that are similar to parts of the image. NMF decomposes the matrix X into two matrices W and H, with the property that none of the three matrices have negative elements. Assume that the matrix X is set so that there are n data points (such as an image of a segment on an image generation chip), each data point has p dimensions (e.g., 14,400). Therefore, the matrix X has p rows and n columns. It is desirable to reduce the p dimension to r dimensions, or in other words, to create an r-order approximation. NMF approximates the matrix X as the product of two matrices: W (p rows and r columns) and H (r rows and n columns).

[0101] The interpretation of the matrix W is that each column is a basis element. By basis elements, we mean some components that exist in the n original data points (or images). These are building blocks from which approximations to all original data points or images can be reconstructed. The interpretation of the matrix H is that each column gives the coordinates of the data points in the basis matrix W. In other words, it is explained how to reconstruct an approximation to the original data points based on linear combinations of the building blocks in the matrix W. In the case of facial images, the basis elements (or basis images) in the matrix W may include features such as eyes, noses, lips, etc. The columns of the matrix H indicate which features exist in which image.

[0102] Feature generator based on image segmentation

[0103] The second image processing technique for extracting features from the process cycle image is based on thresholding of image regions. The image segmentation based feature generator 255 applies thresholding by first segmenting the image of a segment of the image generating chip using the image segmentor 257 and then extracting the intensity of the valid area or area of ​​interest of the segment image. The thresholding determines how much of the valid area is producing the expected signal intensity.

[0104] The image generation chip may include multiple sections organized into rows and columns, such as 24, 48, 96 or more. This design enables multiple samples to be processed in one process cycle because many samples (one sample per section) can be processed in parallel. The sections are physically separated from other sections so that the samples do not mix with each other. In addition, the sections can be organized into multiple parallel regions called "grooves". Therefore, the structure at the boundary of the sections and grooves is visible in the process cycle image from the genotyping scanner. The details of the two components of the feature generator 255 based on image segmentation are presented below, which can implement the technology of converting segment images to extract image features.

[0105] Image Converter

[0106] The image converter 257 applies a series of image conversion techniques to prepare a segment image for extracting intensity from the region of interest. In one specific implementation, the process of image conversion and intensity extraction is performed by some or all of the following five steps. The image conversion converts the grayscale image of the segment into a binary image consisting of dark pixels and bright pixels. The average intensity value of the valid area of ​​the grayscale image and the binary image is provided as an input feature to the classifier to classify the image as a healthy (good) or unhealthy (bad) image. In the text below, the details of the image conversion steps are presented, including applying thresholding to convert the grayscale image into a binary image. The processing steps include applying filters to remove noise.

[0107] The first step in the image conversion process is to apply a bidirectional filter to process the periodic image of the segment. Bidirectional filtering is a technique for smoothing an image while preserving edges. Bidirectional filtering replaces the intensity of each pixel with a weighted average of the intensity values ​​of its neighboring pixels. Each neighboring pixel is weighted by a spatial component that penalizes distant pixels and a range component that penalizes pixels with different intensities. The combination of these two components ensures that only similar pixels nearby contribute to the final result. Therefore, bidirectional filtering is an effective way to smooth an image while preserving its discontinuities or edges. Other filters such as median filters and anisotropic diffusion can be used.

[0108] The second step in the image conversion includes applying thresholding to the output image from step 1. In one specific implementation, Otsu's method (Otsu, N., 1979, "A threshold selection method from gray-level histograms", IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9, No. 1) is applied, which uses an intensity histogram and searches for a threshold to maximize the weighted sum of the grayscale variances between pixels assigned to dark and light intensity levels. Otsu's method attempts to maximize the inter-class variance. The basic idea is that well-thresholded classes should be distinct with respect to the intensity values ​​of their pixels, and conversely, the threshold that gives the best separation between classes according to the class intensity values ​​will be the optimal threshold. In addition, Otsu's method has the property of being entirely based on calculations performed on the histogram of the image, which is a one-dimensional array that is readily available. For more details on Otsu's method, see Section 10.3.3 of "Digital Image Processing" (3rd Edition) by Gonzalez and Woods.

[0109] The third step of image conversion is to apply a noise reduction Gaussian blur filter to remove speckle noise. Noise can contaminate the process periodic image with small specks. Gaussian filtering is a weighted average of the intensities of neighboring locations with weights decreasing with spatial distance from the center location.

[0110] The fourth step of image conversion includes image morphological operations. The binary output image from the third step is processed by morphological conversion to fill holes in the image. A hole can be defined as a background area (represented by 0) surrounded by a connected border of foreground pixels (represented by 1). Two basic image morphological operations are "erosion" and "dilation". In an erosion operation, a kernel slides (or moves) over a binary image. A pixel (1 or 0) in a binary image is considered to be 1 if all pixels below the kernel are 1. Otherwise, it is eroded (changed to 0). The erosion operation is used to remove isolated 1s in a binary image. However, erosion also shrinks clusters of 1s by eroding edges. The dilation operation is opposite to erosion. In this operation, when the kernel slides over a binary image, if the value of at least one pixel below the kernel is 1, the values ​​of all pixels in the binary image area overlapped by the kernel are changed to 1. If a dilation operation is applied to a binary image after an erosion operation, the effect is to close small holes within clusters of 1 (represented by 0s in the image). The output from this step is provided as input to an intensity extractor component 259 which performs the fifth step of the image conversion technique.

[0111] Strength Extractor

[0112] The intensity extractor 259 divides the segment image into valid areas or segments by filtering out structures at the boundaries of segments and grooves. The intensity extractor can apply different segmentations to divide the segment image from eight to seventeen or more valid areas. Examples of areas in the segment image include four grooves, four corners, four edges between corners, and various vertical and horizontal lines at the boundaries of segments and grooves. The areas corresponding to the known structures that separate the valid areas are then removed from the image. The image portions of the remaining valid areas are processed by the intensity extractor 259. Intensity values ​​are extracted for each valid area of ​​the converted image and the corresponding unconverted image and averaged. For example, if intensity values ​​are extracted from 17 valid areas of the converted image, the intensity extractor also extracts intensity values ​​from the same 17 valid areas of the non-converted image. Therefore, a total of 34 features are extracted from each segment image.

[0113] In the case of a binary image, the average intensity of the valid area can be between 1 and 0. For example, the intensity of a dark pixel is considered to be 0 and the intensity of a bright (or blank) pixel is considered to be 1. If all pixels in the valid area are dark pixels, the average intensity of the valid area will be 0. Similarly, if all pixels in the valid area are bright pixels, the intensity of the area will be 1. The valid area in a healthy image appears blank or bright in the binary image, while the dark pixels represent an unhealthy image. The average intensity of the corresponding valid area in the grayscale image is also extracted. The average intensity of the valid area from the grayscale image and the converted binary image is provided as input to a good and bad classifier. In one specific implementation, the classification confidence score from the classifier is compared with a threshold to classify the image as a healthy (or good or successful) image or an unhealthy (or bad or failed) image. An example of a threshold is 80%. A higher threshold may result in more images being classified as unhealthy.

[0114] Training data generator for convolutional neural networks (CNNs)

[0115] The third image processing technique can use cropping, translation, and reflection to prepare the input image for training a convolutional neural network (CNN). The segment image is rectangular, and CNNs such as ResNet (He et al., CVPR 2016, in <<arxiv.org / abs / 1512.03385> >) and VGG (Simonyan et al., 2015, available at <<arxiv.org / abs / 1409.1556> > available at ) uses a square input image of size 224×224 pixels. The training data generator of CNN 275 includes an image cropper 277, an image transformer 279, and an image reflector 281 to prepare the input image for CNN. Image transformer 279 can also expand the labeled input image to increase the size of the training data.

[0116] Image Cropper

[0117] The image cropper 277 includes logic for cropping the image of the segment of the image generation chip to match the input image size required by the convolutional neural network (CNN). In one specific implementation, the high-resolution image obtained from the genotyping instrument has a pixel size of 3600×1600. As described above, higher resolution images may require more computing resources to process, so the disclosed technology downsamples the image to reduce the size. The size of the image can be reduced to 1 / 4 to 1 / 40 on each side from the pixel size of the original image, resulting in an image with a size range of 964×400 pixels to 90×40 pixels. In one example, the segment image is downsampled from the 3600×1600 pixel size of the original image to 1 / 20 on each side, resulting in a size of 180×80 pixels (J×K image). The downsampled image can be provided to the image cropper to prepare the input as needed by the convolutional neural network (CNN).

[0118] Image cropping can crop different parts of a rectangular segment image to prepare a square input image for a CNN. For example, the center portion of a rectangular segment image of an image generation chip can be cropped and placed in an analysis frame of 224×224 pixels. If the input image of the segment is larger than 224×224 pixels, such as 504×224 pixels, the image cropper can crop 224×224 pixels from the image to fill the M×N analysis frame. The image cropper can crop a portion smaller than 224×224 pixels for placement in an M×N analysis frame.

[0119] Image Converter

[0120] The image transformer 279 includes logic to position the labeled input image of the segment of the image generation chip in multiple locations in the analysis frame. "Translation" in our case can mean moving the shape or segment image without rotating or flipping. Translation can also mean sliding the segment image in the analysis frame. After translation, the segment images look the same and have the same size, but are positioned at different locations in the analysis frame. The image transformer includes logic to move or slide the segment images at different locations horizontally, vertically or diagonally in the analysis frame.

[0121] The analysis frame can be larger than the size of the labeled input image. In one case, the analysis frame is square in shape with a size of 224×224 pixels. Analysis frames of other sizes and shapes can be used. The labeled input image of the segment can be positioned at multiple locations in the analysis frame. When the image size is 180×80 pixels and the analysis frame is 224×224 pixels, the image can be translated horizontally and also vertically. In another example, when the image size is 224×100 pixels, the image can only be translated horizontally in an analysis frame of 224×224 pixels. In another example, when the image size is 200×90 pixels, it can be translated horizontally and vertically in an analysis frame of 224×224 pixels.

[0122] Pixels of the input image surrounding the marker in the analysis frame may be zero-padded. In this case, pixels of the input image surrounding the marker are assigned a value of "0" of corresponding intensity. Translation of one input marker image may produce multiple analysis frames containing the input image at different locations in the frame. This process may augment the training data, thereby increasing the number of training examples.

[0123] Image reflector

[0124] The image reflector 281 includes logic to reflect horizontally and vertically a smaller size input marker image located in a larger size analysis frame. Reflection refers to the image of an object or in our case a segment image as seen in a reflector. The smaller size marker input image can be placed in the center of the larger size analysis frame. The image reflector 281 includes logic to reflect the segment image horizontally and vertically along the edges to fill the surrounding pixels in a larger size analysis frame (such as 224×224 pixels). Reflection padding can increase the probability of detecting a failed process image because the image or part of the image is replicated in multiple locations in the analysis frame.

[0125] Process cycle images

[0126] Examples of successful and unsuccessful production images of segments on an image generating chip are now presented. Figure 4400 is an illustration of a production image of 24 segments on an image generating chip. The segments are arranged in twelve rows and two columns. Each segment has four slots. Illustration 400 shows a segment image of a successful production cycle. Other configurations with segments can also be used, such as image generating chips including 48, 96 or more segments. In the following figures, we present examples of segment images of unsuccessful production cycles. The production process is susceptible to operational and chemical processing errors. Operational defects can be caused by mechanical or sample handling problems. Chemical processing errors can be caused by problems in the sample or by chemical processing of the sample. The disclosed technology attempts to classify poorly processed image cycles that occur due to operational and chemical processing errors.

[0127] Figure 5A An example 510 of a section image from an unsuccessful production cycle is shown. Figure 5A The image 512 of the segment in the second column and seventh row of the image generation chip in is dark in the lower half and slightly lighter in the top portion. The cause of this failure is related to the hybridization process. Therefore, the failed image of the segment is marked as a "hybrid (Hyb)" failure. Hybridization failure may also occur due to failure of the robot that handles the sample during the sample preparation process on the image generation chip. The call rate for this segment is "97.545", which is below the 98% pass threshold. In some cases, the call rate of the segment from the genotyping instrument may be higher than the pass threshold, but even so, the segment image may fail due to hybridization errors.

[0128] It may be noted that in illustration 510, the image of segment 514 at row 11 and column 2 has a dark area on the right wall. This may also indicate a processing problem, however, the overall call rate for this image is above the passing threshold and it has not been marked as a failed image. There is sufficient sample redundancy on the segments so that small area segments with obvious failures can be ignored and do not cause errors in the results. For example, in one instance, the scanner reads fluorescence from approximately 700K probes on a segment with a redundancy of 10. Therefore, the call rate is based on readings from approximately 7 million probes. Figure 5B Another example of a hybridization failure is presented in the illustration 515 of . The four segments in the dashed box on the image generation chip show poor production images of the segments due to hybridization failure. Note that the call rate values ​​for these four segments are above the pass threshold, but the images of these segments are marked as failed due to hybridization errors.

[0129] Figure 5CAn illustration 520 of nine segment images showing unsuccessful processing due to failed spacer shifting is presented. When samples are prepared on the segments on the image generation chip, dark markers are placed around the segments. The spacers separate the samples in each segment from other samples in adjacent segments. If the markers are not placed correctly, part of the image signal may be blocked. Figure 5C As shown in the figure, offset errors can occur on multiple adjacent segments. The top portion of the nine segments in this figure appears dark. When moving from left to right, the dark portion of the top portion of the segment increases. The spatial offset problem is an operator error because it is caused by the laboratory technician inaccurately placing the marker during sample preparation on the image generation chip. Figure 5D Three more examples of failed images of segments due to spacer shift failure are presented. Box 525 shows five segment images with spacer shift failure, as the top portion of the segment image is dark with increasing width from the upper right side to the upper left side. Box 527 shows two segment images indicating a failed process due to a spacer shift problem at the bottom portion of the segment. Similarly, box 529 shows images of two segments that failed due to a spacer offset problem.

[0130] Figure 5E An example of a failed image of a segment resulting from an unsuccessful process caused by an offset failure is shown. In an offset failure, the image of the segment on the image generation chip is offset to one side. For example, in illustration 530, all segment images on the image generation chip are shifted to the left, so the dark outer border at the left edge of the image generation chip is cut off from the image. An offset failure can be caused by a scanning error, such as a misalignment of the scanner or incorrect placement of the image generation chip on the chip carrier.

[0131] Fig. 5F An example of a failed segment image due to surface abrasion failure is shown. Surface abrasion is caused by scratches on the segment surface in the image generation chip during the manufacturing process or during sample preparation on the segment. As shown in diagram 535, the scratches appear as lines on the image of the segment. Note that although the call rate values ​​of the three segments in the left dashed box are above the pass threshold, the image is marked as failed due to surface abrasion failure.

[0132] Figure 5GIt is an illustration 540 of a failed segment image caused by reagent flow failure. Due to reagent flow failure, ten segment images in frame 542 are marked as failed images. Due to unsuccessful processes caused by improper reagent flow, segment images fail. During the genotyping process, reagents are introduced into the image generation chip from one side. Reagents flow from one end of the image generation chip to the opposite end and completely cover all segments. Sometimes, there are problems with the flow of reagents, and they cannot be evenly spread to all segments. In this case, when a sufficient amount of reagents does not cover the segments, the reagents may dry up. Improper reagent flow can reduce the emission signal intensity from some segments, which may affect image quality because fluorescent dyes may not be evenly distributed on all segments. Compared with the segment images representing successful process cycles, the colors of failed images caused by reagent flow failure may look darker. Figure 5H Another example of a failed segment image due to reagent flow failure is shown in illustration 545. Figure 5G and Figure 5H As shown, reagent flow failure can affect multiple adjacent segments in an area of ​​the image generation chip.

[0133] Fig.5I An example of a failed image due to an unknown cause is presented. The failed segment image is marked as "unhealthy". Failed images in the unhealthy failure category may be caused by mixed or unidentified causes and weak signals. The illustration 550 of the segment image also shows an example of a spacer failure of the segment in the upper left corner of the image generation chip. The image segment in the upper left position (row 1 and column 2) is marked as a spacer failure. It can be seen that the top portion of the failed segment image is dark. The portion of the dark area at the top increases from the right corner to the left corner of the segment image.

[0134] Feature generation based on principal component analysis

[0135] We now present an example of an eigenimage, which in the field of facial recognition is called an eigenface. From tens of thousands of labeled images, a linear basis of 40 to 100 or more image components is identified. Fig. 6A An example of 96 eigenimages obtained by applying principal component analysis (PCA) is presented. The 96 eigenimages were selected based on the ranking of the components according to the measure of explainable variability as described above. Figure 6B Shown from Fig. 6A The top 40 eigenimages of the 96 eigenimages in FIG. 40 . In one embodiment, it was observed that 40 components explained most of the variability. The selected additional components appear to reflect noise patterns or natural variability in sample processing.

[0136] The dimensionality reduction and creation of a basis of eigenimages using PCA is now described.The first step is to reduce the resolution of the image of the segment and prepare the reduced image for input to PCA. Fig. 7A A diagram 710 showing dimensionality reduction is presented. A segment image of 3600×1600 pixel resolution is downsampled to 1 / 20 on each side, resulting in a segment image of reduced size of 180×80 pixels. The downsampled segment image is flattened. The resulting flattened segment image is a one-dimensional array, i.e. 14,400×1 pixels per segment image.

[0137] Other dimensionality reductions can be used as an alternative to the 1 / 20 size on each edge. The principle is that much less information and pixel density is required to assess the overall health of the flow cell than to call individual clusters or balls in the flow cell. Therefore, reductions in the range of 1 / 2 to 1 / 50 or in the range of 1 / 4 to 1 / 40 can be used, with more extreme reductions in resolution expected as the initial resolution of the segment image increases. It is desirable to choose a resolution reduction that adapts the captured segment to the input aperture of the deep learning framework, especially when transfer learning can be applied to pre-training with the deep learning framework. For an input aperture of 224×224 pixels, downsampling to 180×80 pixels, 1 / 20 (with horizontal and vertical reflections) has proven to be a good choice. Other reductions will be apparent for different segment images and different deep learning input apertures.

[0138] In an alternative embodiment, PCA is applied to the downsampled images as described in the previous application. As described above, the flattened segment images are normalized to obtain Figure 7B The normalized flattened rescaled segment image shown in the diagram 740 of is given as input to PCA. PCA thus produces 14,400 principal components or eigenimages. Each input image is a vector in a 14,400-dimensional space. The principal components or eigenimages are then sorted using explained variance and a basis is created, for example a basis of 40 to 100 components. These components form a basis of a linear space.

[0139] Feature generation based on image segmentation

[0140] A second image processing technique for generating features from images of segments involves thresholding the image regions or segments. Fig. 8AAn example 810 of segmentation-based feature generation applied to a segment image is shown. Illustration 812 in example 810 is a production image of a segment of an image generation chip. Image transformation is applied to transform the grayscale production image 812 to generate a corresponding binary image 814. In one specific implementation, some or all of the five steps presented above with reference to the image transformer 257 may be performed to convert the grayscale image to a binary image. Dark pixels in the binary image 814 indicate unhealthy or bad image pixels, while bright pixels indicate healthy or good image pixels.

[0141] exist Fig. 8A , the illustration 816 on the right is an exemplary schematic diagram of the segment indicating the various areas of the segment and the boundaries or lines surrounding these areas. The area from which the fluorescence signal intensity is recorded is also referred to as the effective area or area of ​​interest. For example, the cross-sectional schematic diagram 816 indicates the effective area of ​​four grooves extending parallel to each other from top to bottom. Areas of the segment image that are not effective areas are filtered out from the image. For example, the boundary areas of the grooves are separated from each other by vertical lines indicating the boundaries or boundaries of the grooves. Similarly, the boundaries on the four sides of the segment image can be filtered out. Segmentation techniques can divide the segment image into 4 to 20 or more segments or effective areas. The thresholding determines how many effective areas are producing the desired signal intensity.

[0142] The number of valid regions determines the number of features generated for each image. For example, if the segment image is segmented into eight valid regions, the image intensity of the eight valid regions of the converted image and the image intensity values ​​of the same eight valid regions of the original segment image before conversion are provided as input to the classifier. Therefore, in this example, a total of 16 features of each segment image will be provided to the classifier. The average intensity of the signal intensity from the valid regions can be used as the input of the classifier. For example, if the segment image is segmented into eight valid regions, the average intensity of the eight valid regions is calculated for the grayscale image and the binary image. These sixteen intensity values ​​are provided as input to the classifier to classify the segment image as good or bad. Other segmentation schemes can be used, which divide the image into fewer or more fragments, such as 4, 12, 17 or more fragments per image. If provided as input to a random forest classifier, a subset of features is randomly selected for each decision tree. The decision tree votes the image as healthy (or successful) or unhealthy (or failed). The majority vote in the random forest is used to classify the image. In one implementation, the value of the number of trees in the random forest classifier ranges from 200 to 500, and the value of the depth of the model ranges from 5 to 40. The failure modes between regions and sections of the image generating chip can be further evaluated for root cause classification.

[0143] Figure 8BAn illustration 820 of three pairs of segment images 822, 824, and 826 is presented. The left image in each pair is a pre-converted grayscale segment image, and the right image in each pair is a pre-converted grayscale segment image after applying the grayscale image as in reference 826. Figure 2 The binary images after the image conversion are processed. The first image pair 822 is a production image of a successful genotyping process. The second image pair 824 is a production image of a failed production image due to hybridization (or mixing) failure. The third image pair 826 has a failed image due to surface abrasion problems.

[0144] One-vs-Many (OvR) Classification

[0145] Fig. 9 A graphical illustration 900 of running a one-to-many classifier is presented. The figure shows an example of running a one-to-many (OvR) classifier on a data set consisting of samples belonging to three categories (square, circle and triangle) as shown in the left figure 931. The first hyperplane 916 shown in the upper right figure 915 shows a hyperplane determined as a reference truth value for the square category. Hyperplane 916 separates the data points in the square category from the remaining data points (circle and triangle). Similarly, Figure 936 and Figure 955 separate the data points in the circle category and the triangle category in the data from other categories in the data via hyperplane 937 and hyperplane 956 respectively. The position of the hyperplane is determined by the weight vector. The training algorithm attempts to maximize the boundary of the hyperplane and the reference truth category for generalization, but this can lead to incorrect classification of one or more data points. We apply OvR classification to distinguish segment images from process cycles belonging to good categories from images belonging to multiple bad (or failed) categories.

[0146] Random Forest Classifier

[0147] The disclosed technology can apply various classifiers to distinguish images from good or healthy images and bad or unhealthy images belonging to multiple failure categories. The applied classifiers include random forest, K nearest neighbor, multinomial logistic regression and support vector machine. The specific implementation of the disclosed technology is presented by taking random forest classifier as an example.

[0148] Random forest classifier (also known as random decision forest) is an ensemble machine learning technique. Ensemble techniques or algorithms combine more than one technique of the same or different types for classifying objects. Random forest classifiers consist of multiple decision trees that operate as an ensemble. Each individual decision tree in a random forest acts as a base classifier and outputs a class prediction. The class with the most votes becomes the prediction of the random forest model. The basic concept behind random forests is that a large number of relatively unrelated models (decision trees) running as a group (committee) will outperform any of the constituent models individually.

[0149] The disclosed technology applies random forest classifiers in a two-stage classification process. The first trained random forest classifier performs the task of separating successful production images from unsuccessful production images. The second trained random forest classifier performs the task of root cause analysis of unsuccessful production images by predicting the failure class of the unsuccessful images. This two-stage classification was chosen due to the dominance of successful production runs, but one-stage classification can also be used. Another reason for choosing a two-stage approach is that the method allows control of the sensitivity threshold in order to classify images as healthy or successful production images versus unhealthy or failed production images. The threshold can be increased in the first stage classification so that the classifier classifies more production images as failed images. These failed images are then processed by the second stage classifier for root cause analysis by identifying the failure class.

[0150] Training of Random Forest Classifier

[0151] Fig. 10A The training of two random forest classifiers shown in Figure 1000 is described. Figure 1 As shown, the training data includes input features of labeled process cycle images stored in the training database 138. In one exemplary training of the classifier, 20,000 labeled production images of a segment are used. The labeled images include good images from successful production cycles and failed images from unsuccessful production cycles. As more labeled production images are received from laboratories performing genotyping processes, the size of the training database 138 will increase.

[0152] In one implementation, a random forest classifier was trained using 96 weights of the components of the labeled production images. The random forest classifier had 200 decision trees and a depth of 20. It should be understood that a random forest classifier with a range of 200 to 500 decision trees and a depth range of 10 to 40 is expected to provide good results for this implementation. Hyperparameters were tuned using random search cross validation. The search range for depth was from 5 to 150, and the search range for the number of trees was from 100 to 500. Increasing the number of trees can improve the performance of the model, however, it can also increase the time required for training. A training database 1001 comprising features of 20,000 production cycle images is used to train a binary classifier labeled good and bad classifier 151. The same training database can be used to train a root cause classifier 171 to predict the failure class. As Fig. 10A As shown, the root cause classifier 171 is trained on a training database 1021 consisting only of bad or failed production images.

[0153] Decision trees tend to overfit. To overcome this problem, bagging techniques are used to train decision trees in random forests. Bagging is a combination of bootstrapping and aggregation techniques. In the bootstrapping process, during training, a sample of rows is taken from the training database and used to train each decision tree in the random forest. For example, a subset of the features of the selected rows can be used to train decision tree 1. Therefore, the training data for decision tree 1 can be referred to as row sample 1 and column sample 1 or RS1+CS1. Columns or features can be selected randomly. Decision tree 2 and subsequent decision trees in the random forest are trained in a similar manner by using a subset of the training data. It should be noted that the training data for the decision tree is generated with replacement, that is, the same row data can be used to train multiple decision trees.

[0154] The second part of the bagging technique is the aggregation part that is applied during production. Each decision tree outputs a classification for each class. In case of binary classification, it can be either 1 or 0. The output of the random forest is the aggregation of the outputs of the decision trees in the random forest, where the majority vote is selected as the output of the random forest. By using votes from multiple decision trees, the random forest reduces the high variance in the decision tree results, resulting in good prediction results. By using row and column sampling to train the individual decision trees, each decision tree becomes an expert on the training records with the selected features.

[0155] During training, the output of the random forest is compared to the ground truth labels and the prediction error is calculated. During back propagation, the weights of the 96 components (or eigenimages) are adjusted so that the prediction error is reduced. The number of components or eigenimages depends on the number of components selected from the output of the principal component analysis (PCA) using the explained variance measure. During binary classification, the good and bad classifier uses image description features from the training data and applies a one-to-many (OvR) classification of the good class (or healthy labeled images) and multiple bad classes (images labeled with one of the six failure classes). The parameters of the trained random forest classifier (such as the weights of the components) are stored for good and bad classification of production cycle images during inference.

[0156] Training of the root cause classifier 171 is performed in a similar manner. The training database 1021 includes features from labeled process cycle images from bad process cycles belonging to multiple failure categories. The image description features are used to train the random forest classifier 171 for one-vs-all (OvR) classification of each failure category with the rest of the labeled training examples.

[0157] Classification using Random Forest Classifier

[0158] Classification of production images using the trained classifiers 151 and 171 is now described. Fig. 10B A two-stage classification 1080 of production images using a good and bad classifier 151 in a first stage and a root cause classifier 171 in a second stage is presented. The process is presented using a sequence of process flow steps labeled 1 to 9. The process begins at step 1, accessing a trained random forest classifier labeled good and bad classifier 151. Input features of production images stored in database 1030 are provided as input to classifier 151. The classifier distinguishes good images belonging to successful process cycles from bad images belonging to failed process cycles. The bad images belong to multiple failure categories, for example, each image may belong to one of the six failure categories described above. The trained classifier accesses a basis of intrinsic images used to analyze the production images. The trained classifier creates image description features for the production images based on linear combinations of the intrinsic images. As described above, the weights of the intrinsic images are learned during the training of the classifier.

[0159] When one-to-many classification is applied, all decision trees in the random forest classifier predict an output for each class, i.e., whether the image belongs to one of seven classes (one good class and six failed classes). Therefore, each decision tree in the random forest will output seven probability values, i.e., one value for each class. The results from the decision trees are aggregated, and a majority vote is used to predict whether the image is good or bad. For example, if more than 50% of the decision trees in the random forest classify the image as good, the image is classified as a good image belonging to a successful production cycle. For example, the sensitivity of the classifier can be adjusted by setting the threshold higher, which will result in more images being classified as bad. In processing step 2, the output of the classifier 151 is checked. If the image is classified as a good image (step 3), the process ends (step 4). Otherwise, if the image is classified as a bad image indicating a failed process cycle (step 5), the system calls the root cause classifier 171 (step 6).

[0160] The root cause classifier is applied in the second stage of the two-stage process to determine the failure class of the bad image. The process continues in the second stage (step 8) by accessing the production image input features of the bad image (step 7) and providing the input features to the trained root cause classifier 171. Each decision tree in the root cause classifier 171 votes on the input image features by applying a one-to-many classification. In this case, the classification determines whether the image belongs to one of the six failure classes or the remaining classes of the five failure classes. Each decision tree provides a classification for each class. The majority vote from the decision trees determines the failure class of the image (step 9).

[0161] Other classifiers can be used to classify good segment images and bad segment images and perform root cause analysis. For example, the disclosed technology can apply a K nearest neighbor (k-NN or KNN) algorithm to classify segment images. The k-NN algorithm assumes that similar examples (or segment images in our specific implementation) are very close. The k-NN algorithm captures the concept of similarity (also called proximity or closeness) by calculating the distance between data points or images. Straight-line distance (or Euclidean distance) is typically used for this purpose. In k-NN classification, the output is a class membership, for example, a good image class or a bad image class. An image is classified by multiple votes of its neighboring images, where the object is assigned to the class that is most common among its k nearest neighbors. The value of k is a positive integer.

[0162] In order to choose the right value of k for the data, the k-NN algorithm is run multiple times with different values ​​of k, and a value of k is chosen that reduces the number of errors encountered while maintaining the algorithm's ability to accurately predict when given data it has not encountered before. Suppose the value of k is set to 1. This can lead to incorrect predictions. Consider having two clusters of data points: good images and bad images. If there is a query example that is surrounded by many good image data points, but is close to a bad image data point that is also in the cluster of good image data points. When k=1, k-NN incorrectly predicts the query example as a bad image. When the value of k is increased, the predictions of the k-NN algorithm become more stable due to majority voting (in classification) and averaging (in regression). Therefore, the algorithm is more likely to make more accurate predictions until a certain value of k is reached. As the value of k increases, an increased number of errors begins to be observed. It is expected that k values ​​in the range of 6 to 50 will work.

[0163] Examples of other classifiers that can be trained and applied by the disclosed techniques include multinomial logistic regression, support vector machines (SVM), gradient boosted trees, naive Bayes, etc. Three criteria are used to evaluate the performance of the classifiers: training time, accuracy, and interpretability of the results. The performance of the random forest classifier is better than the other classifiers. Other classifiers are briefly introduced in the text below.

[0164] The support vector machine classifier also performs as well as the random forest classifier. The SVM classifier locates a hyperplane between the feature vectors of the good class and the feature vectors of multiple bad classes. The disclosed technology may include training a multinomial logistic regression. A multinomial regression model can be trained to predict the probabilities of different possible results (multi-class classification). The model is used when the output is a classification. Therefore, the model can be trained to predict whether an image belongs to a good class or one of multiple bad classes. The performance of the logistic regression classifier is lower than that of the random forest and SVM classifiers. The disclosed technology may include training a gradient boosting model, which is an integration of prediction models such as decision trees. The model attempts to optimize the cost function on the function space by iteratively selecting functions that point in the direction of the negative gradient. For example, the model can be trained to minimize the mean squared error on the training data set. The gradient boosting model requires more training time than other classifiers. The disclosed technology may include training a naive Bayes classifier, which assumes that the value of a particular feature is independent of the value of any other feature. The naive Bayes classifier assumes that each of the features independently contributes to the probability of an example belonging to a category. A Naive Bayes classifier can be trained to classify images into a good class and multiple bad classes.

[0165] Example of feature engineering for deep learning-based classification

[0166] We present examples of feature engineering for images input to a Convolutional Neural Network (CNN). Fig.11A The feature engineering techniques shown in , such as cropping, zero padding, and reflection operations, are used to generate analysis frames 1105, 1107, and 1109 having a size of M×N pixels. The size of the analysis frame is designed to match the input image size used by a specific convolutional neural network. In one instance, the size of the analysis frame is 224×224 pixels, which matches the size of the input image required by the ResNet-18 and VGG-16 models. The dimensions used in this example are 3600×1600 pixels for the segment image. This example illustrates how a 3600×1600 pixel segment image can be adapted to a square frame of size 224×224 pixels for input to a convolutional neural network (CNN). The first step of the process is to upsample the 3600×1600 pixel segment image by 1 / 20 on each side to obtain a reduced size image of 180×80 pixels. The second step is to place the 180×80 pixel image into a frame of 224×224 pixels. The placed image is then reflected horizontally to fill a 224x224 pixel square frame on the left and right sides of the 180x80 pixel placed image. The 180x80 pixel image is then reflected vertically to fill the top and bottom portions of the 224x224 square frame.

[0167] Fig.11AAn illustration 1100 includes cropping, zero padding, and reflection operations on a segment image. An exemplary segment image 1103 is shown. The segment image 1103 is in the shape of a rectangle. In cropping, the center portion or any other portion of the segment image is cropped. Fig.11A A cropped portion of a segment image 1103 placed in an analysis frame 1105 is illustrated. The cropped portion of the segment image may have square dimensions, i.e., the cropped portion has the same number of pixels along the height and width of the cropped portion. For example, the cropped image placed in the analysis frame 1105 may be an 80×80 pixel cropped portion of the J×K image 1103. The cropped portion may be positioned in the center of the analysis frame. In other implementations, the cropped portion may be positioned closer to one side of the analysis frame. In another implementation, the cropped portion may be resampled so that the size of the cropped segment is equal to the size of the analysis frame, e.g., 224×224 pixels. This will result in the cropped segment completely filling the analysis frame.

[0168] Fig.11A The zero padding operation is shown after the segment image 1103 is placed in the analysis frame 1107. The segment image may be positioned in the center of the analysis frame, as shown in illustration 1107. The segment image may also be positioned at any other location in the analysis frame. The surrounding pixels in the analysis frame may be given a "0" image intensity value.

[0169] Fig.11A Also shown is the application of reflections to fill pixels in the analysis frame 1109. The segment image can be positioned at the center of the analysis frame 1107 and then reflected horizontally to fill the analysis frame along the left and right edges of the analysis frame. It can also be reflected vertically along the top and bottom edges, filling the top and bottom portions of the analysis frame 1109. Evaluation of various strategies described in detail below shows that reflections to fill the frame work well in training and production without requiring any changes to the input aperture of the image processing framework being evaluated. Surprisingly, reflections significantly outperform zero padding.

[0170] Fig. 11B The example illustrates augmenting training data by applying translation to a training example segment image 1103. Applying translation for data augmentation includes placing the segment image at multiple locations in the analysis frame. For example, the segment image is located at the center of the analysis frame, as shown in analysis frame 1111. Horizontal and vertical translation of the segment image can produce multiple analysis frames, as shown in illustrations 1113, 1115, 1117, etc. The surrounding area may initially be zero-filled, depicted by a dark color (or black), as shown in FIG. Fig. 11B However, the surrounding area may not always be black. The disclosed technique can apply horizontal and vertical reflection padding, such as Fig.11A As shown, the surrounding area is filled with the reflection segment image.

[0171] Transfer learning using fine-tuning

[0172] In many real-world applications, the entire convolutional neural network (CNN) is not trained from scratch with random initialization. This is because in most cases the training dataset is small. Typically, it is small enough to be trained on a dataset such as ImageNet, which contains about 14 million images with 1000 categories (available in <<image-net.org> >, and then use the pre-trained CNN as the initialization or fixed feature extractor for the task of interest. This process is called transfer learning, which transfers the knowledge learned from the source dataset to the target dataset. A commonly used transfer learning technique is called fine-tuning.<d2l.ai / chapter_computer-vision / fine-tuning.html> > Reproduction Fig. 12A Fine-tuning (1200) is shown. There are four steps to fine-tune the deep learning model.

[0173] Step 1: Pre-train a neural network model, i.e., the source model, on a source dataset (e.g., ImageNet dataset).

[0174] Step 2 : The second step is to create a new neural network model, the target model. This copies all the model design and its parameters on the source model, except for the output layer. We assume that these model parameters contain the knowledge learned from the source dataset and that this knowledge will also apply to the target dataset. We also assume that the output layers of the source model are closely related to the labeling of the source dataset and are therefore not used in the target model.

[0175] Step 3 : The third step is to add an output layer to the target model whose output size is the number of target dataset classes, and randomly initialize the model parameters of this layer. Therefore, for the detection task, the output layer of our model may have two classes (normal and failed). For the classification task, the output layer of our model may have five classes corresponding to the five defect classes. The model may have additional classes to classify images with unknown failure types that are not classified in the existing known failure classes.

[0176] Step 4 : The fourth step is to train the target model on the target dataset. In our case, the training dataset consists of about 75,000 labeled segment images. We train the output layer from scratch, while the parameters of all remaining layers are fine-tuned based on the parameters of the source model.

[0177] The network architecture of the VGG-16 model

[0178] In recent years, the VGG architecture (Simonyan et al., 2015, in<arxiv.org / abs / 1409.1556> >) is widely used in computer vision. It consists of stacked convolutional and max pooling layers. We have used a smaller and therefore faster 16-layer architecture, which is called VGG-16. The architecture (1210) is presented in Fig. 12B The model consists of five convolutional layers conv1(1220), conv2(1222), conv3(1224), conv4(1226), and conv5(1228). One way to analyze the output of the convolution is through a fully connected (FC) network. Therefore, the output of the convolutional layer is given to the fully connected (FC) layer. There are three fully connected layers fc6(1240), fc7(1242), and fc8(1244). The model uses "max pooling" between the convolutional layers and between the last convolutional layer and the fully connected layers. There are five "max pooling" layers 1230, 1232, 1234, 1236, and 1238. We extract features from the "conv5" layer (1228) with a stride of 16 pixels.

[0179] The model takes as input an image size of 224×224 pixels. The input image can be an RGB image. The image is passed through a stack of convolutional (conv) layers, where filters are used with very small receptive fields: 3×3 (to capture the concept of left / right, top / bottom, center). In one configuration, it utilizes 1×1 convolutional filters, which can be viewed as a linear transformation of the input channels (followed by a nonlinearity). The convolution stride is fixed to 1 pixel; the spatial padding of the convolutional layer input is such that the spatial resolution is preserved after the convolution, i.e., for a 3×3 convolutional layer, the padding is 1 pixel.

[0180] Spatial pooling is performed by five "max pooling" layers 1230, 1232, 1234, 1236 and 1238, as shown in Fig. 12BAs shown, they are followed by some convolutional layers. The output of the convolution is also called a feature map. This output is given as input to a max pooling layer. The goal of the pooling layer is to reduce the dimensionality of the feature map. For this reason, it is also called "downsampling". The factor by which it will be downsampled is called the "stride" or "downsampling factor". The pooling stride is represented by "s". In one type of pooling called "max pooling", the maximum value is selected for each stride. For example, consider applying max pooling with s=2 to a 12-dimensional vector x=[1,10,8,2,3,6,7,0,5,4,9,2]. A max pooling vector x with stride s=2 means that we select the maximum value from every two values ​​starting from index 0, resulting in a vector [10,8,6,7,5,9]. Therefore, a max pooling vector x with stride s=2 produces a 6-dimensional vector. In the example, the VGG-16 architecture, max pooling is performed on a 2×2 pixel window with a stride of 2.

[0181] Fig. 12B The image size is reduced after passing through each of the five convolutional layers and corresponding maximum pooling layers in the VGG model. The size of the input image is 224×224 pixels. After passing through the first convolutional layer and the maximum pooling layer (1230) labeled conv1 (1220), the image size is reduced to 112×112 pixels. After passing through the second convolutional layer and the maximum pooling layer (1232) labeled conv2 (1222), the image size is reduced to 56×56 pixels. The third convolutional layer and the maximum pooling layer (1234) labeled conv3 (1224) reduce the image size to 28×28 pixels. The fourth convolutional layer and the maximum pooling layer (1236) labeled conv4 (1226) reduce the image size to 14×14 pixels. Finally, the fifth convolutional layer and the maximum pooling layer (1238) labeled conv5 (1228) reduce the image size to 7×7 pixels.

[0182] Three fully connected (FC) layers 1240, 1242, and 1244 follow the stack of convolutional layers. The first two FC layers 1240 and 1242 each have 4096 channels, and the third FC layer 1244 performs 1000-way classification and therefore contains 1000 channels (one channel for each class). The final layer is a softmax layer. The depth of the convolutional layers can vary in different architectures of the VGG model. The configuration of the fully connected layers can be the same in different architectures of the VGG model.

[0183] Fig. 12C The VGG-16 architecture 1270 applied by the disclosed technology for the detection task is illustrated. The model architecture illustrates the parameter values ​​of five convolutional layers conv1 to conv5 and three fully connected layers fc6 to fc8. The output from the final FC layer 1244 produces two outputs that classify the segment image as normal or failed. Fig. 12C The labels in correspond to Fig. 12B The corresponding elements in the architecture diagram in .

[0184] The network architecture of the RESNET-18 model

[0185] ResNet architecture (He et al., CVPR 2016, in <<arxiv.org / abs / 1512.03385> > is designed to avoid the problems of very deep neural networks. Mainly, the use of residual connections helps to overcome the vanishing gradient problem. We use the ResNet-18 architecture with 18 trainable layers. Fig.12D and Fig.12E An exemplary ResNet-18 architecture 1280 is illustrated. The architecture of ResNet-18 is organized into four layers (e.g. Fig.12D 1260, 1262 and as shown Fig.12E 1264, 1268 shown). Each layer includes two blocks labeled "0" and "1". Each block includes two convolutional layers, labeled "conv1" and "conv2". There is a convolutional layer 1259 before the layer 1 block (1260). Therefore, there are a total of 17 convolutional layers separated by batch normalization and ReLU. A fully connected (FC) layer (1270) at the end produces two outputs. This architecture is used for detection tasks, where the network classifies segment images into normal (or good) and failed (or bad) images.

[0186] Training deep learning models with dozens of convolutional layers is challenging because the distribution of inputs to layers deep in the network may change after each mini-batch as the weights are updated. This slows down the convergence of the model. Batch Normalization (Ioffe and Szegedy 2015, in<arxiv.org / abs / 1502.03167> >) technique can overcome this problem. Batch normalization normalizes the input to the layer for each mini-batch and reduces the number of epochs required to train the deep learning model.

[0187] Performance Comparison - Feature Engineering and Deep Learning Models

[0188] We compare the performance of two models, ResNet-18 and VGG-16, using augmented input data generated by applying the above feature engineering techniques. Fig.13 The diagram 1300 in Figure 1 presents the results of the performance comparison using the macro F1 score. The macro F1 performance score can be considered to reflect a combination of precision and recall values. For example, the F1 score can combine the precision (p) and recall (r) values ​​with equal weight, as shown in equation (1) below. The macro F1 score can be calculated as the average of the F1 scores of all failure categories.

[0189]

[0190] Return to reference Fig.13 In the results, the top two bars 1305 and 1310 represent the scores of the VGG-16 and ResNet-18 models, respectively. When trained using the fine-tuning method with transfer learning, both models produce the same macro F1 (or F1) performance score of 96.2. The VGG-16 and ResNet-18 models are trained on the entire segment image of the image generation chip. As described above, reflection padding is used to fill the larger analysis frame. The training data is expanded by copying images. When there are fewer images belonging to one category (such as a failure category) than images belonging to one or more other categories, the system can expand the training data by making multiple copies of labeled images in that category to balance the data set. Alternatively, image copies can be used multiple times during training to balance the training data.

[0191] Fig.13 The chart in also shows the performance of the ResNet-18 model with different feature engineering techniques, i.e., the third to fifth bars of the performance comparison chart. The third bar 1315 shows that when the ResNet-18 model is trained using only the cropping technique for feature engineering, a score of 77.2 is achieved. When only cropping is used to generate training data, the results are not good because many defects do not appear in the central part of the segment image. Therefore, such defects may be missed by the model during training. The fourth bar 1320 shows that when the ResNet-18 model is trained using training data generated using zero padding, the performance score is 92.3, which is used to fill the area in the analysis frame around the segment image. Since there are unnatural (dark) areas in the analysis frame around the segment image location, zero padding does not perform well. The fifth bar marked 1325 from the top shows that when the ResNet-18 model is trained using training data augmented by making copies of the labeled image, the performance score is 94.9. In this case, translation is not used to create additional training data changes.

[0192] The bar 1335 at the bottom of the chart 1300 represents the performance of the base model. The base model is a ResNet-18 CNN, which is used as a feature extractor without fine-tuning. No feature engineering techniques such as reflection filling are used. No data augmentation is used either. The score of the base model is 74.3.

[0193] Fig.14 A breakdown of the performance scores of the failure categories of the best performing model disclosed is presented. The best model achieved an overall macro F1 score of 97% and an accuracy of 96%. Fig.14 Table 1401 in presents the F1 scores for different failure categories, examples of which are given above. Figure 5A-Figure 5I Presented in.

[0194] Confusion matrix 1420 shows the model performance of predicted labels and true labels for five failure categories. As shown by the values ​​on the diagonal (marked as 1425), most defect categories are correctly predicted. The vertical bar on the right represents the number of samples with non-failure categories. The number of samples with a specific failure type can be calculated by summing the number in the row of that failure type (marked as 1430). For example, the number of samples containing offset and spacer type failures is 38 and 26, respectively. The correct prediction of the failure category can be determined by looking at the values ​​in the diagonal. For example, the model correctly predicted 36 samples with offset failure and 24 samples with spatial offset failure. Similarly, of the 129 samples with hybridization or "hyb" failure, 123 samples were correctly predicted. Of the total 170 samples with reagent flow failure, the model correctly predicted 163 samples with reagent flow failure. The highest number of samples contains surface abrasion failure. Of the total 437 samples with abrasion failure, a total of 428 samples were correctly predicted.

[0195] Misclassified samples

[0196] Fig.15 Presented from Fig.14 Analysis of the 26 misclassified six images in the confusion matrix presented in Figure 2. For each segment image, the manually annotated failure labels are listed in the top row, and the predicted labels from the model are listed in the bottom row. Some images were misclassified by the deep learning model, while others were misclassified by the humans who labeled the training set.

[0197] The four sample images from the left 1503, 1505, 1507, and 1509 have multiple defects belonging to different failure categories. The manual annotation represents only one defect out of the multiple defects. For example, the first image on the left (labeled 1503) was labeled by the human annotator as having surface abrasion. The surface abrasion is present in the top left portion of the image as indicated in the bounding box. The model predicts the image as having hybrid or hyb failure. It can be seen that the image has hyb failures in two locations, in the top right portion of the image and near the bottom of the image, as pointed by the arrows.

[0198] The last two images 1511 and 1513 were incorrectly labeled by the human annotator, and the deep learning model predicted these images correctly. For example, the fifth image on the left (labeled 1511) was labeled as having a spacer shift failure. However, the model predicted the image as having a surface abrasion failure. Since the dark portion of the image is closely positioned to the bottom edge of the image, the human annotator may have incorrectly identified the failure class. Similarly, the sixth image on the left (labeled 1513) was labeled by the human annotator as a hybridization or hyb failure. The model predicted the failure class as a reagent flow failure, which is the correct failure class for the image. Therefore, the performance of machine learning is even better than the F1 score indicates.

[0199] Performance improvements through the use of deep learning

[0200] We compare the performance of our deep learning based approach (deepIBEX) with an earlier solution using a random forest model (IBEX) on the same split of the dataset. The results show that deepIBEX outperforms IBEX as measured by macro F1 score and accuracy in both the tasks of anomaly detection (separating good from bad images) and classification (identifying the root cause of bad images).

[0201] method

[0202] The dataset of segment images is split into training, validation, and test sets with sample ratios of 70%, 15%, and 15%. For both models compared here (deepIBEX and IBEX), we use the same dataset split for parameter tuning, training, and evaluation. We use random grid search to tune model hyperparameters on the training and validation sets. We finally evaluate the model performance on the test set.

[0203] Test parameters

[0204] Ten hyperparameter sets for each model were examined via random grid search. In IBEX, the main hyperparameters we tuned were the final dimension of PCA and the tree depth in random forest. In deepIBEX, we tuned the learning rate, batch size, momentum.

[0205] Macro F1 score and accuracy measurement

[0206] The deepIBEX model performs better than the IBEX model, as indicated by the macro F1 score and accuracy. The F1 score for each class can be defined as the harmonic mean of the recall and precision for that class. The F1 score can also be calculated using equal weights of precision and recall as shown in equation (1) above. The macro F1 score can be calculated as the average of the F1 scores of all failed classes. The macro F1 score and accuracy measures of model performance on the test set are presented in the table below (Table 1). A higher macro F1 score or higher accuracy means that the model performs better on the test data. Accuracy is defined as the proportion of correctly labeled samples in all test samples. For detection and classification, deepIBEX exceeds IBEX.

[0207] Table 1: Comparison of DeepIBEX and IBEX models

[0208]

[0209] Process flow for training and applying good and bad classifiers

[0210] Fig.16 A flow chart 1601 is presented that presents process steps for training a good and bad classifier and applying the trained classifier to classify production images of segments of an image generating chip.

[0211] The processing steps presented in flowchart 1601 can be implemented using a processor programmed using a computer program stored in a memory accessible by a computer system and executed by the processor, by dedicated logic hardware including field programmable integrated circuits, and by a combination of dedicated logic hardware and computer programs. Figure 1 Similarly, it should be understood that many steps can be combined, performed in parallel, or performed in a different order without affecting the functions implemented. In addition, it should be understood that the flowcharts herein only show steps relevant to understanding the technology, and it should be understood that many additional steps for implementing other functions can be performed before, after, and between the steps shown.

[0212] The process begins at step 1602. The creation training data process step 1610 may include a plurality of operations (1612, 1614, and 1616) that may be performed to create training data including a labeled image of a segment of an image generation chip. The training data creation may include generating a J×K labeled image (step 1612). This step may include cutting out a portion from a larger image to generate a J×K labeled image. The J×K labeled image may be positioned at a plurality of locations in an M×N analysis frame (step 1614). The size of the M×N analysis frame is designed to match the input image size required by a convolutional neural network (CNN). The J×K image may not completely fill the M×N analysis frame because the analysis frames may be smaller in size than the M×N analysis frame. In addition, the M×N analysis frame may be square in shape, and the J×K image may be rectangular in shape. At step 1616, a portion of the J×K labeled image positioned in the M×N analysis frame may be used to fill in around the edge of the J×K labeled image. Horizontal reflections may be used to fill in the M×N analysis frame along the left and right edges of the analysis frame. Vertical reflections can be used to fill the M×N analysis frame along its top and bottom edges.The above steps can be repeated to generate many training examples by changing the position of the same J×K labeled image in the M×N analysis frame.

[0213] At step 1620, a convolutional neural network (CNN) can be trained using the training data generated by performing the above process steps. The system can train a pre-trained CNN, such as a VGG-16 or ResNet-18 model. The trained CNN model can then be applied to the production images of the segment to classify the image as good or bad (step 1624). Feature engineering techniques such as reflection filling can be applied to the production images of the segment of the image generation chip to fill the M×N input frames into the CNN model. Images classified as good or normal by the classifier can indicate a successfully completed process. Images classified as bad or failed can indicate a process failure (step 1626). The process can continue at step 1628 to further classify bad process cycle images to determine the root cause of the failure. If the production image is classified as good, the process can end at step 1630.

[0214] Specific implementation

[0215] The disclosed technology applies image classification to the evaluation and root cause analysis of the genotyping process. The classifier performs two tasks: separating successful and unsuccessful production images, and then performing root cause analysis on the unsuccessful images.

[0216] Training and inference of good and bad classifiers

[0217] Classification of successful and unsuccessful production images is first presented. In one specific implementation of the disclosed technology, a method for training a convolutional neural network (CNN) to identify and classify images of sections of an image-generating chip from bad or failed or unsuccessful processes that cause process cycle failures is described. The method includes extracting image features using a pre-trained convolutional neural network (CNN). The pre-trained CNN can accept images of size M×N. Examples of image sizes for input to the CNN may include 224×224 pixels, 227×227 pixels, 299×299 pixels. Alternatively, the size of the input image to the CNN may be in the range of 200 pixels to 300 pixels on one side, or in the range of 75 pixels to 550 pixels on one side. The input image may be square-shaped or rectangular-shaped. The method includes creating a training data set using a labeled image of size J×K, which is smaller than M×N, and the labeled image is normal and depicts a process failure. Exemplary sizes of J×K sized marker images are 180×80 pixels, 224×100 pixels, 200×90 pixels, 120×120 pixels, 224×224 pixels, 504×224 pixels, etc. The method includes creating a training data set using a marker image of size J×K that is smaller than M×N, the marker image being normal and the delineation process failing. The image is from a segment of an image generation chip. The method includes positioning the J×K marker image at multiple locations in an M×N (224×224) frame. The method includes using at least a portion of a particular J×K marker image to fill in around the edge of the particular J×K marker image, thereby filling in the M×N frame.

[0218] The method includes further training the pre-trained CNN using the training data set to produce a segment classifier. The method includes storing coefficients of the trained classifier to recognize and classify images of segments of an image generating chip from a production process cycle. The trained classifier can receive images of segments of an image generating chip and classify the images as normal or depicting a process failure.

[0219] In a production implementation, the trained CNN can be applied to identify bad process cycle images of a segment of an image generating chip. We now present a method for identifying bad process cycle images of a segment of an image generating chip that causes a failure of a process cycle. The method includes creating an input to a trained classifier.

[0220] The input can fill the M×N input aperture of the image processing framework, or it can have a smaller size J×K and be reflected to fill the M×N analysis frame. With the latter approach, creation of the input includes accessing an image of a segment of size J×K. The method includes positioning the J×K image in the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned in the M×N analysis frame. Depending on the relative size of J×K to M×N, some zero padding can be applied, for example, to fill narrow strips along the top and bottom of the analysis frame, but reflections have been found to perform better. The method includes inputting the M×N analysis frame into a trained classifier. The method includes using the trained classifier to classify an image of a segment of an image generating chip as normal or a failure of the delineation process. The method includes outputting the resulting classification of a segment of an image generating chip.

[0221] The disclosed method implementations and other methods optionally include one or more of the following features. The method may also include features described in conjunction with the above method. For the sake of brevity, alternative combinations of method features are not enumerated separately. Features applicable to methods, systems, and articles are not repeated for each set of essential features of the statutory classification. The reader will understand how the features identified in this section can be easily combined with essential features in other statutory classifications.

[0222] In one specific implementation, the method further includes positioning the J×K image at the center of the M×N analysis frame.

[0223] In one specific implementation, the method includes applying a horizontal reflection to at least a portion of the particular JxK marker image to fill in around an edge of the particular JxK marker image in the MxN analysis frame.

[0224] In one specific implementation, the method includes applying a vertical reflection to at least a portion of the specific JxK marker image to fill in around an edge of the specific JxK marker image in the MxN analysis frame.

[0225] In one implementation, the J×K labeled image is generated by cropping a portion from a larger image and placing the J×K cropped portion in an M×N frame. Examples of larger image sizes include images of size 504×224 pixels or even larger.

[0226] A marker image of size J×K can be obtained by downsampling the high resolution image from the scanner so that the resolution of the high resolution image is reduced by 1 / 2 to 1 / 50 on each side of the original resolution in pixels. A reduction to 1 / 25 on each side reduces the pixel count to 1 / 625 of the original pixel count. In one specific implementation, a high resolution image of a segment obtained from a scanner or genotyping instrument has a size of 3600×1600 pixels.

[0227] The computer-implemented methods described above may be practiced in a system comprising computer hardware. The computer-implemented system may practice one or more of the methods described above. The computer-implemented system may incorporate any features of the methods described immediately above or throughout this application that are applicable to the methods implemented by the system. For the sake of brevity, alternative combinations of system features are not enumerated separately. Features applicable to systems, methods, and articles of manufacture are not repeated for each set of essential features of the statutory classification. The reader will understand how the features identified in this section can be easily combined with essential features in other statutory classifications.

[0228] As an article of manufacture rather than a method, a non-transitory computer readable medium (CRM) may be loaded with program instructions executable by a processor. When the program instructions are executed, one or more of the above-described computer-implemented methods are implemented. Alternatively, the program instructions may be loaded on a non-transitory CRM and, when combined with appropriate hardware, become a component of one or more of a computer-implemented system for practicing the methods disclosed in the present invention.

[0229] Each of the features discussed in this particular implementation section of the method implementation is equally applicable to the CRM and system implementations. As indicated above, for the sake of brevity, all method features are not repeated here and should be considered repeated by reference.

[0230] Training and inference root cause analysis

[0231] In one specific implementation of the disclosed technology, a method for training a convolutional neural network (CNN) to classify images of a segment of an image generating chip by the root cause of a process failure is described. The method includes extracting image features using a pre-trained CNN. The pre-trained CNN can accept images of size M×N. Examples of image sizes include 224×224 pixels. The method includes creating a training data set using a labeled image of size J×K, which belongs to at least one of multiple failure categories that cause process failure. Exemplary sizes of labeled images of size J×K are 180X80 pixels, 200X90 pixels, etc. The image is from a segment of an image generating chip. The method includes positioning the J×K labeled image at multiple locations in an M×N (224×224) frame. The method includes filling the M×N frame with at least a portion of a specific J×K labeled image to fill in around the edge of the specific J×K labeled image. The method includes further training the pre-trained CNN using the training data set to produce a segment classifier. The method includes storing coefficients of a trained classifier to recognize and classify images of a segment of an image-generating chip from a production process cycle. The trained classifier may receive an image of a segment of an image-generating chip and classify the image by root cause of a process failure in a plurality of failure categories.

[0232] In a production implementation, the trained CNN can be applied to classify bad process cycle images of a segment. We now present a method for identifying and classifying bad process cycle images of a segment of an image generating chip that caused a failure of a process cycle. The method includes creating an input to a trained classifier.

[0233] The input can fill the M×N input aperture of the image processing framework, or it can have a smaller size J×K and be reflected to fill the M×N analysis frame. With the latter approach, creating the input includes accessing an image of a segment of size J×K. The method includes positioning the J×K image in the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned in the M×N analysis frame. Depending on the relative size of J×K to M×N, some zero padding can be applied, for example, to fill narrow strips along the top and bottom of the analysis frame, but reflections have been found to perform better. The method includes inputting the M×N analysis frame to a trained classifier. The method includes using the trained classifier to classify the image of the segment of the image generating chip by the root cause of the process failure in multiple failure categories. The method includes outputting the resulting classification of the segment of the image generating chip.

[0234] The disclosed method implementations and other methods optionally include one or more of the following features. The method may also include features described in conjunction with the above method. For the sake of brevity, alternative combinations of method features are not enumerated separately. Features applicable to methods, systems, and articles are not repeated for each set of essential features of the statutory classification. The reader will understand how the features identified in this section can be easily combined with essential features in other statutory classifications.

[0235] In one specific implementation, the method further includes positioning the J×K image at the center of the M×N analysis frame.

[0236] In one specific implementation, the method includes applying a horizontal reflection to at least a portion of the particular JxK marker image to fill in around an edge of the particular JxK marker image in the MxN analysis frame.

[0237] In one specific implementation, the method includes applying a vertical reflection to at least a portion of the specific JxK marker image to fill in around an edge of the specific JxK marker image in the MxN analysis frame.

[0238] In one implementation, the J×K labeled image is generated by cropping a portion from a larger image and placing the J×K cropped portion in an M×N frame. Examples of larger image sizes include images of size 504×224 pixels or even larger.

[0239] The marker image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 of the original resolution. In one specific implementation, the high-resolution image of the segment obtained from the scanner or genotyping instrument has a size of 3600×1600 pixels.

[0240] The plurality of failure categories may include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0241] The plurality of failure categories may include a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0242] The computer-implemented methods described above may be practiced in a system comprising computer hardware. The computer-implemented system may practice one or more of the methods described above. The computer-implemented system may incorporate any features of the methods described immediately above or throughout this application that are applicable to the methods implemented by the system. For the sake of brevity, alternative combinations of system features are not enumerated separately. Features applicable to systems, methods, and articles of manufacture are not repeated for each set of essential features of the statutory classification. The reader will understand how the features identified in this section can be easily combined with essential features in other statutory classifications.

[0243] As an article of manufacture rather than a method, a non-transitory computer readable medium (CRM) may be loaded with program instructions executable by a processor. When the program instructions are executed, one or more of the above-described computer-implemented methods are implemented. Alternatively, the program instructions may be loaded on a non-transitory CRM and, when combined with appropriate hardware, become a component of one or more of a computer-implemented system for practicing the methods disclosed in the present invention.

[0244] Each of the features discussed in this particular implementation section of the method implementation is equally applicable to the CRM and system implementations. As indicated above, for the sake of brevity, all method features are not repeated here and should be considered repeated by reference.

[0245] Combination of process failures, single-pass good and bad detection and root causes

[0246] In one specific implementation of the disclosed technology, a method for training a convolutional neural network (CNN) to identify and classify images of a segment of an image generating chip from a bad process cycle that causes a process cycle failure is described. The method includes using a pre-trained CNN to extract image features. The pre-trained CNN can accept images of size M×N. The method includes creating a training data set using a labeled image of size J×K, which is normal and belongs to at least one failure category of multiple failure categories that cause process failure. The image is from a segment of an image generating chip. The method includes positioning the J×K labeled image at multiple locations in an M×N (224×224) frame. The method includes using at least a portion of a specific J×K labeled image to fill in around the edge of a specific J×K labeled image, thereby filling in M×N. The method includes further training the pre-trained CNN using the training data set to produce a segment classifier. The method includes storing coefficients of the trained classifier to identify and classify images of a segment of an image generating chip from a production process cycle. The trained classifier may receive an image of a section of the image generating chip and classify the image as normal or belonging to at least one failure category of a plurality of failure categories that lead to process failure.

[0247] In a production implementation, the trained CNN can be applied to classify bad process cycle images of a segment. We propose a method for identifying bad process cycle images of a segment of an image generating chip that caused a failure of a process cycle. The method includes creating an input to a trained classifier.

[0248] The input can fill the M×N input aperture of the image processing framework, or it can have a smaller size J×K and be reflected to fill the M×N analysis frame. With the latter approach, creation of the input includes accessing an image of a segment of size J×K. The method includes positioning the J×K image in the M×N analysis frame. The method includes filling the M×N analysis frame using horizontal and / or vertical reflections along the edges of the J×K image positioned in the M×N analysis frame. Depending on the relative size of J×K to M×N, some zero padding can be applied, for example, to fill narrow strips along the top and bottom of the analysis frame, but reflections have been found to perform better. The method includes inputting the M×N analysis frame into a trained classifier. The method includes using the trained classifier to classify an image of a segment of an image generating chip as normal or belonging to at least one of a plurality of failure categories. The method includes outputting the resulting classification of a segment of an image generating chip.

[0249] The method may also include features described in conjunction with the above method. For the sake of brevity, alternative combinations of method features are not enumerated separately. Features applicable to methods, systems, and articles of manufacture are not repeated for each set of essential features of the statutory classification. The reader will understand how the features identified in this section can be easily combined with essential features in other statutory classifications.

[0250] The computer-implemented methods described above may be practiced in a system comprising computer hardware. The computer-implemented system may practice one or more of the methods described above. The computer-implemented system may incorporate any features of the methods described immediately above or throughout this application that are applicable to the methods implemented by the system. For the sake of brevity, alternative combinations of system features are not enumerated separately. Features applicable to systems, methods, and articles of manufacture are not repeated for each set of essential features of the statutory classification. The reader will understand how the features identified in this section can be easily combined with essential features in other statutory classifications.

[0251] As an article of manufacture rather than a method, a non-transitory computer readable medium (CRM) may be loaded with program instructions executable by a processor. When the program instructions are executed, one or more of the above-described computer-implemented methods are implemented. Alternatively, the program instructions may be loaded on a non-transitory CRM and, when combined with appropriate hardware, become a component of one or more of a computer-implemented system for practicing the methods disclosed in the present invention.

[0252] Each of the features discussed in this particular implementation section of the method implementation is equally applicable to the CRM and system implementations. As indicated above, for the sake of brevity, all method features are not repeated here and should be considered repeated by reference.

[0253] Terms

[0254] The following clauses describe various aspects related to the technology described herein.

[0255] Detect good and bad separately and then classify the root cause

[0256] Good and bad training

[0257] Clause 1. A method of training a convolutional neural network to identify and classify images of sections of an image generating chip that caused a process failure, comprising:

[0258] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0259] Create a training dataset using labeled images of size J×K smaller than M×N that depict success and failure of the process.

[0260] The labeled image comes from the segment of the image generation chip,

[0261] Position the J×K marker images at multiple locations in the M×N frame,

[0262] Filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image;

[0263] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier; and

[0264] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0265] The classifier thus trained can receive images of a section of the image generating chip and classify the images as depicting a successful or failed process.

[0266] Clause 2. The method of clause 1, wherein filling in around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image further comprises:

[0267] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0268] Clause 3. The method of clause 1, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image further comprises:

[0269] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0270] Clause 4. The method of clause 1, wherein the training dataset is created using labeled images of size J×K, further comprising:

[0271] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0272] Clause 5. The method of clause 1, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0273] Clause 6. The method of clause 5, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0274] Clause 7. A non-transitory computer readable storage medium having printed thereon computer program instructions for identifying and classifying images of a section of an image generating chip that caused a process failure, the instructions when executed on a processor implementing a method comprising:

[0275] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0276] Create a training dataset using labeled images of size J×K smaller than M×N that depict success and failure of the process.

[0277] The labeled image comes from the segment of the image generation chip,

[0278] Position the J×K marker images at multiple locations in the M×N frame,

[0279] Filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image;

[0280] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier; and

[0281] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0282] The classifier thus trained can receive an image of a segment of the image generating chip and classify the image as normal or a failure of the delineation process.

[0283] Clause 8. The non-transitory computer-readable storage medium of clause 7, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image, implementing the method further comprises:

[0284] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0285] Clause 9. The non-transitory computer-readable storage medium of clause 7, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image, implementing the method further comprises:

[0286] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0287] Clause 10. The non-transitory computer-readable storage medium of clause 7, wherein the training dataset is created using labeled images of size J×K, and implementing the method further comprises:

[0288] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0289] Clause 11. A non-transitory computer-readable storage medium according to clause 7, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0290] Clause 12. The non-transitory computer-readable storage medium of Clause 11, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0291] Clause 13. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to identify and classify images of sections of an image generating chip that caused a process failure, the instructions when executed on the processors performing actions comprising:

[0292] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0293] Create a training dataset using labeled images of size J×K smaller than M×N that depict success and failure of the process.

[0294] The labeled image comes from the segment of the image generation chip,

[0295] Position the J×K marker images at multiple locations in the M×N frame,

[0296] Filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image;

[0297] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier; and

[0298] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0299] The classifier thus trained can receive an image of a segment of the image generating chip and classify the image as normal or a failure of the delineation process.

[0300] Clause 14. The system of clause 13, wherein filling around an edge of the particular J×K tagged image using at least a portion of the particular J×K tagged image further implements actions comprising:

[0301] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0302] Clause 15. The system of clause 13, wherein filling around an edge of the particular J×K tagged image using at least a portion of the particular J×K tagged image further implements actions comprising:

[0303] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0304] Clause 16. The system of clause 13, wherein the training dataset is created using labeled images of size J×K, further implementing actions comprising:

[0305] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0306] Clause 17. The system of clause 13, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0307] Clause 18. The system of clause 17, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0308] Training Root Cause Analysis

[0309] Clause 19. A method of training a convolutional neural network to classify images of a segment of an image generating chip by root cause of a process failure, comprising:

[0310] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0311] A training dataset is created using labeled images of size J×K, the labeled images belonging to at least one failure category among a plurality of failure categories that cause the process to fail, where

[0312] The labeled image comes from the segment of the image generation chip,

[0313] Position the J×K marker images at multiple locations in the M×N frame,

[0314] filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame,

[0315] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier; and

[0316] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0317] The classifier thus trained may receive images of a section of an image generating chip and classify the images by root cause of a process failure in a plurality of failure categories.

[0318] Clause 20. The method of clause 19, wherein filling in around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image further comprises:

[0319] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0320] Clause 21. The method of clause 19, wherein filling in around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image further comprises:

[0321] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0322] Clause 22. The method of clause 19, wherein the training dataset is created using labeled images of size J×K, further comprising:

[0323] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0324] Clause 23. The method of clause 19, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0325] Clause 24. The method of clause 23, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0326] Clause 25. The method of clause 19, wherein the plurality of failure categories comprises at least hybridization failure, spatial displacement failure, offset failure, surface abrasion failure, and reagent flow failure.

[0327] Clause 26. The method of clause 19, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0328] Clause 27. A non-transitory computer readable storage medium having printed thereon computer program instructions for training a convolutional neural network to classify images of a segment of an image generating chip by root cause of a process failure, the instructions when executed on a processor implementing a method comprising:

[0329] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0330] A training dataset is created using labeled images of size J×K, the labeled images belonging to at least one failure category among a plurality of failure categories that cause the process to fail, where

[0331] The labeled image comes from the segment of the image generation chip,

[0332] Position the J×K marker images at multiple locations in the M×N frame,

[0333] filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame,

[0334] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier; and

[0335] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0336] The classifier thus trained may receive images of a section of an image generating chip and classify the images by root cause of a process failure in a plurality of failure categories.

[0337] Clause 28. The non-transitory computer-readable storage medium of clause 27, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image, implementing the method further comprises:

[0338] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0339] Clause 29. The non-transitory computer-readable storage medium of clause 27, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image, implementing the method further comprises:

[0340] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0341] Clause 30. The non-transitory computer-readable storage medium of clause 27, wherein the training dataset is created using labeled images of size J×K, and implementing the method further comprises:

[0342] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0343] Clause 31. A non-transitory computer-readable storage medium according to clause 27, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0344] Clause 32. The non-transitory computer-readable storage medium of Clause 31, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0345] Clause 33. The non-transitory computer-readable storage medium of clause 27, wherein the plurality of failure categories comprises at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0346] Clause 34. The non-transitory computer-readable storage medium of Clause 27, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0347] Clause 35. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to train a convolutional neural network to classify images of a segment of an image generating chip by a root cause of a process failure, the instructions when executed on the processors performing actions comprising:

[0348] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0349] A training dataset is created using labeled images of size J×K, the labeled images belonging to at least one failure category among a plurality of failure categories that cause the process to fail, where

[0350] The labeled image comes from the segment of the image generation chip,

[0351] Position the J×K marker images at multiple locations in the M×N frame,

[0352] filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame,

[0353] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier; and

[0354] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0355] The classifier thus trained may receive images of a section of an image generating chip and classify the images by root cause of a process failure in a plurality of failure categories.

[0356] Clause 36. The system of clause 35, wherein filling around an edge of the particular J×K tagged image using at least a portion of the particular J×K tagged image further implements actions comprising:

[0357] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0358] Clause 37. The system of clause 35, wherein filling around an edge of the particular J×K tagged image using at least a portion of the particular J×K tagged image further implements actions comprising:

[0359] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0360] Clause 38. The system of clause 35, wherein the training dataset is created using labeled images of size J×K, further implementing actions comprising:

[0361] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0362] Clause 39. A system according to clause 35, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0363] Clause 40. The system of clause 39, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0364] Clause 41. A system according to clause 35, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0365] Clause 42. The system of clause 35, wherein the plurality of failure categories comprises a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0366] Inferring good and bad

[0367] Clause 43. A method of generating a process cycle image of a segment of an image generating chip that identifies a failure of a process cycle, comprising:

[0368] Create the input to the trained classifier, including:

[0369] Access an image in segments of size J × K,

[0370] Position the J×K image in the M×N analysis frame,

[0371] filling the M×N analysis frame using at least a portion of the particular J×K marker image to fill in around edges of the particular J×K marker image,

[0372] Input the M×N analysis frames to the trained classifier;

[0373] classifying the image of the segment of the image generating chip as normal or a failure of the delineation process using the trained classifier; and

[0374] The output image generates the resulting classification of the segments of the chip.

[0375] Clause 44. The method of clause 43, further comprising positioning the JxK image at the center of the MxN analysis frame.

[0376] Clause 45. The method of clause 43, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further comprises:

[0377] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0378] Clause 46. The method of clause 43, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further comprises:

[0379] A vertical reflection is applied to at least a portion of the particular JxK image.

[0380] Clause 47. The method of clause 43, wherein creating an input to a trained classifier further comprises:

[0381] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0382] Clause 48. The method of clause 43, wherein the J×K image is obtained by downsampling a high resolution image from a scanner such that the resolution of the high resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0383] Clause 49. The method of clause 48, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0384] Clause 50. The method according to clause 43, further comprising:

[0385] accessing a second convolutional neural network upon determining that an image depiction process of a segment of the image generating chip has failed, wherein the second convolutional neural network is trained to classify the image in a plurality of failure categories by root cause of the process failure; and

[0386] A second convolutional neural network is applied to the images of the segment for classification among a plurality of failure categories to select as a possible root cause of the failed process cycle among the plurality of failure categories.

[0387] Clause 51. The method of clause 50, wherein the plurality of failure categories comprises at least hybridization failure, spatial displacement failure, offset failure, surface abrasion failure, and reagent flow failure.

[0388] Clause 52. The method of clause 50, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0389] Clause 53. A non-transitory computer readable storage medium having printed thereon computer program instructions for identifying a process cycle image of a segment of an image generating chip that caused a failure of a process cycle, the instructions when executed on a processor implementing a method comprising:

[0390] Create the input to the trained classifier, including:

[0391] Access an image in segments of size J × K,

[0392] Position the J×K image in the M×N analysis frame,

[0393] filling the M×N analysis frame using at least a portion of the particular J×K marker image to fill in around edges of the particular J×K marker image,

[0394] Input the M×N analysis frames to the trained classifier;

[0395] classifying the image of the segment of the image generating chip as normal or a failure of the delineation process using the trained classifier; and

[0396] The output image generates the resulting classification of the segments of the chip.

[0397] Clause 54. The non-transitory computer-readable storage medium of Clause 53, implementing the method further comprising positioning the JxK image at the center of the MxN analysis frame.

[0398] Clause 55. The non-transitory computer-readable storage medium of clause 53, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image, implementing the method further comprises:

[0399] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0400] Clause 56. The non-transitory computer-readable storage medium of clause 53, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image, implementing the method further comprises:

[0401] A vertical reflection is applied to at least a portion of the particular JxK image.

[0402] Clause 57. The non-transitory computer-readable storage medium of clause 53, wherein creating an input to a trained classifier, implementing the method further comprises:

[0403] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0404] Clause 58. The non-transitory computer-readable storage medium of clause 53, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0405] Clause 59. The non-transitory computer-readable storage medium of Clause 58, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0406] Clause 60. The non-transitory computer-readable storage medium of clause 53, wherein the method further comprises:

[0407] accessing a second convolutional neural network upon determining that an image depiction process of a segment of the image generating chip has failed, wherein the second convolutional neural network is trained to classify the image in a plurality of failure categories by root cause of the process failure; and

[0408] A second convolutional neural network is applied to the images of the segment for classification among a plurality of failure categories to select as a possible root cause of the failed process cycle among the plurality of failure categories.

[0409] Clause 61. The non-transitory computer-readable storage medium of clause 60, wherein the plurality of failure categories comprises at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0410] Clause 62. The non-transitory computer-readable storage medium of Clause 60, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0411] Clause 63. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to identify a process cycle image of a segment of an image generating chip that caused a failure of a process cycle, the instructions when executed on the processors implementing a method comprising:

[0412] Create the input to the trained classifier, including:

[0413] Access an image in segments of size J × K,

[0414] Position the J×K image in the M×N analysis frame,

[0415] filling the M×N analysis frame using at least a portion of the particular J×K marker image to fill in around edges of the particular J×K marker image,

[0416] Input the M×N analysis frames to the trained classifier;

[0417] classifying the image of the segment of the image generating chip as normal or a failure of the delineation process using the trained classifier; and

[0418] The output image generates the resulting classification of the segments of the chip.

[0419] Clause 64. The system of clause 63, further implementing actions comprising: positioning the JxK image at the center of the MxN analysis frame.

[0420] Clause 65. The system of clause 63, wherein filling around an edge of the particular JxK image using at least a portion of the particular JxK image further implements actions comprising:

[0421] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0422] Clause 66. The system of clause 63, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further implements actions comprising:

[0423] A vertical reflection is applied to at least a portion of the particular JxK image.

[0424] Clause 67. The system of clause 63, wherein creating an input to the trained classifier further performs actions comprising:

[0425] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0426] Clause 68. The system of clause 63, wherein the J×K image is obtained by downsampling a high resolution image from a scanner such that the resolution of the high resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0427] Clause 69. The system of clause 68, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0428] Clause 70. The system of clause 63, further implementing actions comprising:

[0429] accessing a second convolutional neural network upon determining that an image depiction process of a segment of the image generating chip has failed, wherein the second convolutional neural network is trained to classify the image in a plurality of failure categories by root cause of the process failure; and

[0430] A second convolutional neural network is applied to the images of the segment for classification among a plurality of failure categories to select as a possible root cause of the failed process cycle among the plurality of failure categories.

[0431] Clause 71. A system according to clause 70, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0432] Clause 72. The system of clause 70, wherein the plurality of failure categories comprises a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0433] Inferring the root cause

[0434] Clause 73. A method of identifying and classifying a process cycle image of a segment of an image generating chip that caused a failure of the process cycle, comprising:

[0435] Create the input to the trained classifier, including:

[0436] Access an image in segments of size J × K,

[0437] Position the J×K image in the M×N analysis frame,

[0438] filling in around edges of the particular J×K labeled image using at least a portion of the particular J×K labeled image to fill in the M×N analysis,

[0439] Input the M×N analysis frames to the trained classifier;

[0440] classifying an image of a segment of the image generating chip by root cause of a process failure in a plurality of failure categories using a trained classifier; and

[0441] The output image generates the resulting classification of the segments of the chip.

[0442] Clause 74. The method of clause 73, further comprising positioning the J×K image at the center of the M×N analysis frame.

[0443] Clause 75. The method of clause 73, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further comprises:

[0444] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0445] Clause 76. The method of clause 73, wherein filling in around an edge of the particular J×K image using at least a portion of the particular J×K image further comprises:

[0446] A vertical reflection is applied to at least a portion of the particular JxK image.

[0447] Clause 77. The method of clause 73, wherein creating an input to the trained classifier further comprises:

[0448] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0449] Clause 78. The method of clause 73, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0450] Clause 79. The method of clause 78, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0451] Clause 80. The method of clause 73, wherein the plurality of failure categories comprises at least hybridization failure, spatial displacement failure, offset failure, surface abrasion failure, and reagent flow failure.

[0452] Clause 81. The method of clause 73, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0453] Clause 82. A non-transitory computer readable storage medium having printed thereon computer program instructions for identifying and classifying process cycle images of segments of an image generating chip that caused a failure of a process cycle, the instructions when executed on a processor implementing a method comprising:

[0454] Create the input to the trained classifier, including:

[0455] Access an image in segments of size J × K,

[0456] Position the J×K image in the M×N analysis frame,

[0457] filling in around edges of the particular J×K labeled image using at least a portion of the particular J×K labeled image to fill in the M×N analysis,

[0458] Input the M×N analysis frames to the trained classifier;

[0459] classifying an image of a segment of the image generating chip by root cause of a process failure in a plurality of failure categories using a trained classifier; and

[0460] The output image generates the resulting classification of the segments of the chip.

[0461] Clause 83. The non-transitory computer-readable storage medium of clause 82, wherein the method further comprises:

[0462] Position the J×K image at the center of the M×N analysis frame.

[0463] Clause 84. The non-transitory computer-readable storage medium of clause 82, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image, implementing the method further comprises:

[0464] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0465] Clause 85. The non-transitory computer-readable storage medium of clause 82, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image, implementing the method further comprises:

[0466] A vertical reflection is applied to at least a portion of the particular JxK image.

[0467] Clause 86. The non-transitory computer-readable storage medium of clause 82, wherein creating an input to a trained classifier, implementing the method further comprises:

[0468] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0469] Clause 87. A non-transitory computer-readable storage medium as described in Clause 82, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0470] Clause 88. The non-transitory computer-readable storage medium of Clause 87, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0471] Clause 89. The non-transitory computer-readable storage medium of clause 82, wherein the plurality of failure categories comprises at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0472] Clause 90. The non-transitory computer-readable storage medium of clause 82, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0473] Clause 91. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to identify and classify process cycle images of segments of an image generating chip that caused a failure of a process cycle, the instructions when executed on the processors implementing a method comprising:

[0474] Create the input to the trained classifier, including:

[0475] Access an image in segments of size J × K,

[0476] Position the J×K image in the M×N analysis frame,

[0477] filling in around edges of the particular J×K labeled image using at least a portion of the particular J×K labeled image to fill in the M×N analysis,

[0478] Input the M×N analysis frames to the trained classifier;

[0479] classifying an image of a segment of the image generating chip by root cause of a process failure in a plurality of failure categories using a trained classifier; and

[0480] The output image generates the resulting classification of the segments of the chip.

[0481] Clause 92. The system according to clause 91, further implementing actions including:

[0482] Position the J×K image at the center of the M×N analysis frame.

[0483] Clause 93. The system of clause 91, wherein filling around an edge of the particular JxK image using at least a portion of the particular JxK image further implements actions comprising:

[0484] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0485] Clause 94. The system of clause 91, wherein filling around an edge of the particular JxK image using at least a portion of the particular JxK image further implements actions comprising:

[0486] A vertical reflection is applied to at least a portion of the particular JxK image.

[0487] Clause 95. The system of clause 91, wherein creating an input to the trained classifier further performs actions comprising:

[0488] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0489] Clause 96. A system according to clause 91, wherein the J×K image is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0490] Clause 97. A system according to clause 96, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0491] Clause 98. A system according to clause 91, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0492] Clause 99. The system of clause 91, wherein the plurality of failure categories comprises a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0493] Combination of process failures, single-pass good and bad detection and root causes

[0494] Combined training

[0495] Clause 100. A method of training a convolutional neural network to identify and classify images of sections of an image generating chip that caused a process cycle failure, comprising:

[0496] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0497] A training dataset is created using labeled images of size J×K that are normal and belong to at least one of a plurality of failure categories that cause the process to fail, where

[0498] The labeled image comes from the segment of the image generation chip,

[0499] Position the J×K marker images at multiple locations in the M×N frame,

[0500] filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame,

[0501] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier, and

[0502] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0503] The classifier thus trained may receive an image of a section of the image generating chip and classify the image as normal or as belonging to at least one of a plurality of failure classes that lead to process failure.

[0504] Clause 101. The method of clause 100, wherein filling in around an edge of the particular JxK marker image using at least a portion of the particular JxK marker image further comprises:

[0505] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0506] Clause 102. The method of clause 100, wherein filling in around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image further comprises:

[0507] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0508] Clause 103. The method of clause 100, wherein the training dataset is created using labeled images of size J×K, further comprising:

[0509] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0510] Item 104. A method according to item 100, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0511] Clause 105. The method of clause 104, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0512] Clause 106. The method of clause 100, wherein the plurality of failure categories comprises at least hybridization failure, spatial displacement failure, offset failure, surface abrasion failure, and reagent flow failure.

[0513] Clause 107. The method of clause 100, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0514] Clause 108. A non-transitory computer readable storage medium having printed thereon computer program instructions for training a convolutional neural network to identify and classify images of sections of an image generating chip that cause a process cycle failure, the instructions when executed on a processor implementing a method comprising:

[0515] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0516] A training dataset is created using labeled images of size J×K that are normal and belong to at least one of a plurality of failure categories that cause the process to fail, where

[0517] The labeled image comes from the segment of the image generation chip,

[0518] Position the J×K marker images at multiple locations in the M×N frame,

[0519] filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame,

[0520] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier, and

[0521] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0522] The classifier thus trained may receive an image of a section of the image generating chip and classify the image as normal or as belonging to at least one of a plurality of failure classes that lead to process failure.

[0523] Clause 109. The non-transitory computer-readable storage medium of clause 108, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image, implementing the method further comprises:

[0524] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0525] Clause 110. The non-transitory computer-readable storage medium of clause 108, wherein filling around an edge of the particular J×K marker image using at least a portion of the particular J×K marker image, implementing the method further comprises:

[0526] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0527] Clause 111. The non-transitory computer-readable storage medium of clause 108, wherein the training dataset is created using labeled images of size J×K, and implementing the method further comprises:

[0528] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0529] Clause 112. A non-transitory computer-readable storage medium as described in clause 108, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0530] Clause 113. The non-transitory computer-readable storage medium of Clause 112, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0531] Clause 114. The non-transitory computer-readable storage medium of clause 108, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0532] Clause 115. The non-transitory computer-readable storage medium of clause 108, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0533] Clause 116. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to train a convolutional neural network to identify and classify images of sections of an image generating chip that caused a process cycle failure, the instructions when executed on the processors performing actions comprising:

[0534] Use a pre-trained convolutional neural network to extract image features, where the pre-trained convolutional neural network accepts images of size M×N;

[0535] A training dataset is created using labeled images of size J×K that are normal and belong to at least one of a plurality of failure categories that cause the process to fail, where

[0536] The labeled image comes from the segment of the image generation chip,

[0537] Position the J×K marker images at multiple locations in the M×N frame,

[0538] filling the M×N frame using at least a portion of the specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame,

[0539] further training the pre-trained convolutional neural network using the training dataset to produce a segment classifier, and

[0540] storing coefficients of a classifier trained to recognize and classify images of segments of an image generating chip from a production process cycle,

[0541] The classifier thus trained may receive an image of a section of the image generating chip and classify the image as normal or as belonging to at least one of a plurality of failure classes that lead to process failure.

[0542] Clause 117. The system of clause 116, wherein filling around an edge of the particular J×K tagged image using at least a portion of the particular J×K tagged image further implements actions comprising:

[0543] A horizontal reflection is applied to at least a portion of a particular J×K marked image.

[0544] Clause 118. The system of clause 116, wherein filling around an edge of the particular J×K tagged image using at least a portion of the particular J×K tagged image further implements actions comprising:

[0545] A vertical reflection is applied to at least a portion of a particular JxK marker image.

[0546] Clause 119. The system of clause 116, wherein the training dataset is created using labeled images of size J×K, further implementing actions comprising:

[0547] A J×K labeled image is produced by cropping out portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0548] Clause 120. A system according to clause 116, wherein the marked image of size J×K is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0549] Clause 121. The system of clause 120, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0550] Clause 122. A system according to clause 116, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0551] Clause 123. The system of clause 116, wherein the plurality of failure categories comprises a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0552] Combination Inference

[0553] Clause 124. A method of generating a process cycle image of a segment of an image generating chip that identifies a failure of a process cycle, comprising:

[0554] Create the input to the trained classifier, including:

[0555] Access an image in segments of size J × K,

[0556] Position the J×K image in the M×N analysis frame,

[0557] filling in around edges of the particular J×K labeled image using at least a portion of the particular J×K labeled image to fill in the M×N analysis,

[0558] Input the M×N analysis frame to the trained classifier,

[0559] classifying the image of the segment of the image generating chip as normal or belonging to at least one of a plurality of failure categories using a trained classifier, and

[0560] The output image generates the resulting classification of the segments of the chip.

[0561] Clause 125. The method of clause 124, further comprising positioning the JxK image at the center of the MxN analysis frame.

[0562] Clause 126. The method of clause 124, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further comprises:

[0563] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0564] Clause 127. The method of clause 124, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further comprises:

[0565] A vertical reflection is applied to at least a portion of the particular JxK image.

[0566] Clause 128. The method of clause 124, wherein creating an input to a trained classifier further comprises:

[0567] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0568] Clause 129. The method of clause 124, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0569] Clause 130. The method of clause 129, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0570] Clause 131. The method of clause 124, wherein the plurality of failure categories comprises at least hybridization failure, spatial displacement failure, offset failure, surface abrasion failure, and reagent flow failure.

[0571] Clause 132. The method of clause 124, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0572] Clause 133. A non-transitory computer readable storage medium having printed thereon computer program instructions for identifying a process cycle image of a segment of an image generating chip that caused a failure of a process cycle, the instructions when executed on a processor implementing a method comprising:

[0573] Create the input to the trained classifier, including:

[0574] Access an image in segments of size J × K,

[0575] Position the J×K image in the M×N analysis frame,

[0576] filling in around edges of the particular J×K labeled image using at least a portion of the particular J×K labeled image to fill in the M×N analysis,

[0577] Input the M×N analysis frame to the trained classifier,

[0578] classifying the image of the segment of the image generating chip as normal or belonging to at least one of a plurality of failure categories using a trained classifier, and

[0579] The output image generates the resulting classification of the segments of the chip.

[0580] Clause 134. The non-transitory computer-readable storage medium of clause 133, further comprising positioning the JxK image at the center of the MxN analysis frame.

[0581] Clause 135. The non-transitory computer-readable storage medium of clause 133, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image, implementing the method further comprises:

[0582] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0583] Clause 136. The non-transitory computer-readable storage medium of clause 133, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image, implementing the method further comprises:

[0584] A vertical reflection is applied to at least a portion of the particular JxK image.

[0585] Clause 137. The non-transitory computer-readable storage medium of clause 133, wherein creating an input to a trained classifier, implementing the method further comprises:

[0586] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0587] Clause 138. A non-transitory computer-readable storage medium as described in clause 133, wherein the J×K image is obtained by downsampling a high-resolution image from a scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0588] Clause 139. The non-transitory computer-readable storage medium of clause 138, wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

[0589] Clause 140. A non-transitory computer-readable storage medium as described in clause 133, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0590] Clause 141. The non-transitory computer-readable storage medium of clause 133, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

[0591] Clause 142. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to identify a process cycle image of a segment of an image generating chip that caused a failure of a process cycle, the instructions when executed on the processors implementing a method comprising:

[0592] Create the input to the trained classifier, including:

[0593] Access an image in segments of size J × K,

[0594] Position the J×K image in the M×N analysis frame,

[0595] filling in around edges of the particular J×K labeled image using at least a portion of the particular J×K labeled image to fill in the M×N analysis,

[0596] Input the M×N analysis frame to the trained classifier,

[0597] classifying the image of the segment of the image generating chip as normal or belonging to at least one of a plurality of failure categories using a trained classifier, and

[0598] The output image generates the resulting classification of the segments of the chip.

[0599] Clause 143. The system of clause 142, further comprising positioning the J×K image at the center of the M×N analysis frame.

[0600] Clause 144. The system of clause 142, wherein filling around an edge of the particular JxK image using at least a portion of the particular JxK image further implements actions comprising:

[0601] A horizontal reflection is applied to at least a portion of the particular J×K image.

[0602] Clause 145. The system of clause 142, wherein filling around an edge of the particular J×K image using at least a portion of the particular J×K image further implements actions comprising:

[0603] A vertical reflection is applied to at least a portion of the particular JxK image.

[0604] Clause 146. The system of clause 142, wherein creating an input to the trained classifier further performs actions comprising:

[0605] A J×K image is produced by cropping portions from a larger image and placing the J×K cropped portions in an M×N frame.

[0606] Clause 147. A system according to clause 142, wherein the J×K image is obtained by downsampling the high-resolution image from the scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

[0607] Clause 148. The system of clause 147, wherein the high resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

[0608] Clause 149. A system according to clause 142, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

[0609] Clause 150. The system of clause 142, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

[0610] Computer Systems

[0611] Fig.1717 is a simplified block diagram of a computer system 1700 that can be used to implement the technology disclosed in the present invention. The computer system typically includes at least one processor 1772 that communicates with multiple peripheral devices via a bus subsystem 1755. These peripheral devices may include a storage subsystem 1710, which includes, for example, a memory subsystem 1722 and a file storage subsystem 1736, a user interface input device 1738, a user interface output device 1776, and a network interface subsystem 1774. The input devices and output devices allow a user to interact with the computer system. The network interface subsystem provides an interface to an external network, including an interface to corresponding interface devices in other computer systems.

[0612] In one implementation, a good and bad classifier 151 that classifies bad images is communicatively linked to the storage subsystem and the user interface input device.

[0613] User interface input devices 1738 may include keyboards; pointing devices, such as a mouse, trackball, touch pad, or graphics tablet; scanners; touch screens incorporated into displays; audio input devices, such as voice recognition systems and microphones; and other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into a computer system.

[0614] User interface output devices 1776 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from a computer system to a user or to another machine or computer system.

[0615] Storage subsystem 1710 stores programming and data structures that provide some or all of the modules and methods described herein. These software modules are typically executed by a processor alone or in combination with other processors.

[0616] The memory used in the storage subsystem may include multiple memories, including a main random access memory (RAM) 1732 for storing instructions and data during program execution and a read-only memory (ROM) 1734 in which fixed instructions are stored. The file storage subsystem 1736 may provide persistent storage for program files and data files, and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media tape disk. Modules that implement certain specific implementation functions may be stored in the storage subsystem by the file storage subsystem, or in other machines accessible to the processor.

[0617] The bus subsystem 1755 provides a mechanism for the various components and subsystems of the computer system to communicate with each other as intended. Although the bus subsystem is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0618] The computer system itself can be of different types, including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a mainframe, a server farm, a widely distributed group of loosely networked computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the Fig.17 The description of the computer system depicted in the is intended only as a specific example for illustrating the disclosed technology. Many other configurations of the computer system are possible, with more Fig.17 More or fewer components of the computer system depicted in FIG.

[0619] The computer system 1700 includes a GPU or FPGA 1778. It may also include a machine learning processor hosted by a machine learning cloud platform such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's tensor processing unit (TPU), rack-mounted solutions (such as GX4 rack-mounted series, GX8 rack-mounted series), NVIDIA DGX-1, Microsoft's Stratix V FPGA, Graphcore's intelligent processor unit (IPU), Qualcomm's Zeroth Platform with Snapdragon processor, NVIDIA's Volta, NVIDIA's DRIVE PX, NVIDIA's JETSON TX1 / TX2 MODULE, Intel's Nirvana, Movidius VPU, Fujitsu DPI, ARM's DynamicIQ, IBM TrueNorth, etc.

Claims

1. A method of training a convolutional neural network to recognize and classify a genotyping process captured by an image of a segment of an image generating chip as successful or failed, comprising: Extracting image features using the pre-trained convolutional neural network, wherein the pre-trained convolutional neural network accepts an image of size M×N; Create a training dataset using labeled images of size J×K smaller than M×N, the labeled images depicting success and failure of the genotyping process; The labeled image comes from a segment of the image generating chip, positioning the J×K marker image at a plurality of locations in the M×N frame, Filling the M×N frame using at least a portion of a specific J×K marker image to fill around an edge of the specific J×K marker image; further training the pre-trained convolutional neural network using the training dataset to generate a segment classifier; storing coefficients of the trained classifier to recognize and classify images of segments of the image generating chip from a production genotyping process cycle, Thus, the trained classifier is capable of accepting an image of a segment of the image generating chip and classifying the genotyping process as a success or a failure, independent of the corresponding genotyping call rate of the segment of the image generating chip; as well as The image generating chip is processed as part of the genotyping process based on whether the genotyping process is classified as a success or a failure by the trained classifier.

2. The method according to claim 1, wherein filling around an edge of a specific J×K marker image using at least a portion of the specific J×K marker image further comprises: A horizontal reflection is applied to the at least a portion of the particular JxK mark image.

3. The method according to claim 1, wherein filling around an edge of a specific J×K marker image using at least a portion of the specific J×K marker image further comprises: A vertical reflection is applied to the at least a portion of the particular JxK mark image.

4. The method of claim 1, wherein creating the training dataset using labeled images of size J×K further comprises: The JxK labeled image is generated by cropping out portions from a larger image and placing the JxK cropped portions in the MxN frame.

5. The method of claim 1, wherein the marked image of size J×K is obtained by downsampling a high-resolution image from a scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side. The method according to claim 5 , wherein the high-resolution image obtained from the scanner has a size of 3600×1600 pixels.

7. A non-transitory computer readable storage medium having printed thereon computer program instructions for identifying and classifying a genotyping process captured by an image of a segment of an image generating chip as successful or failed, the instructions when executed on a processor implementing a method comprising: Extracting image features using a pre-trained convolutional neural network, wherein the pre-trained convolutional neural network accepts an image of size M×N; Create a training dataset using labeled images of size J×K smaller than M×N, the labeled images depicting success and failure of the genotyping process; The labeled image comes from a segment of the image generating chip, positioning the J×K marker image at a plurality of locations in the M×N frame, Filling the M×N frame using at least a portion of a specific J×K marker image to fill around an edge of the specific J×K marker image; further training the pre-trained convolutional neural network using the training dataset to generate a segment classifier; storing coefficients of the trained classifier to recognize and classify images of segments of the image generating chip from a production genotyping process cycle, Thus, the trained classifier is capable of accepting an image of a segment of the image generating chip and classifying the genotyping process as a success or failure, independent of the corresponding genotyping call rate of the segment of the image generating chip; as well as The image generating chip is processed as part of the genotyping process based on whether the genotyping process is classified as a success or a failure by the trained classifier.

8. The non-transitory computer-readable storage medium of claim 7, wherein using at least a portion of a specific J×K marker image to fill in around an edge of the specific J×K marker image, implementing the method further comprises: A horizontal reflection is applied to the at least a portion of the particular JxK mark image.

9. The non-transitory computer-readable storage medium of claim 7, wherein using at least a portion of a specific J×K marker image to fill in around an edge of the specific J×K marker image, implementing the method further comprises: A vertical reflection is applied to the at least a portion of the particular JxK mark image.

10. The non-transitory computer-readable storage medium of claim 7, wherein the training dataset is created using labeled images of size J×K, and implementing the method further comprises: The JxK labeled image is generated by cropping out portions from a larger image and placing the JxK cropped portions in the MxN frame.

11. The non-transitory computer-readable storage medium of claim 7, wherein the marked image of size J×K is obtained by downsampling a high-resolution image from a scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side. 12 . The non-transitory computer-readable storage medium of claim 11 , wherein the high-resolution image obtained from a scanner has a size of 3600×1600 pixels.

13. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to identify and classify a genotyping process captured by an image of a segment of an image generating chip as successful or failed, the instructions when executed on the processors performing actions comprising: Extracting image features using a pre-trained convolutional neural network, wherein the pre-trained convolutional neural network accepts an image of size M×N; Create a training dataset using labeled images of size J×K smaller than M×N, the labeled images depicting success and failure of the genotyping process; The labeled image comes from a segment of the image generating chip, positioning the J×K marker image at a plurality of locations in the M×N frame, Filling the M×N frame using at least a portion of a specific J×K marker image to fill around an edge of the specific J×K marker image; further training the pre-trained convolutional neural network using the training dataset to generate a segment classifier; storing coefficients of the trained classifier to recognize and classify images of segments of the image generating chip from a production genotyping process cycle, Thus, the trained classifier is capable of accepting an image of a segment of the image generating chip and classifying the genotyping process as a success or failure, independent of the corresponding genotyping call rate of the segment of the image generating chip; as well as The image generating chip is processed as part of the genotyping process based on whether the genotyping process is classified as a success or a failure by the trained classifier.

14. The system of claim 13, wherein using at least a portion of a specific J×K marker image to fill in around an edge of the specific J×K marker image further implements actions including: A horizontal reflection is applied to the at least a portion of the particular JxK mark image.

15. The system of claim 13, wherein using at least a portion of a specific J×K marker image to fill in around an edge of the specific J×K marker image further implements actions including: A vertical reflection is applied to the at least a portion of the particular JxK mark image.

16. The system of claim 13, wherein the training dataset is created using labeled images of size J×K, further implementing actions comprising: The JxK labeled image is generated by cropping out portions from a larger image and placing the JxK cropped portions in the MxN frame.

17. The system of claim 13, wherein the marked image of size J×K is obtained by downsampling a high-resolution image from a scanner such that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

18. The system of claim 17, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

19. A method of training a convolutional neural network to classify a genotyping process captured by an image of a segment of an image generating chip by a root cause of a genotyping process failure, comprising: Extracting image features using the pre-trained convolutional neural network, wherein the pre-trained convolutional neural network accepts an image of size M×N; A training dataset is created using labeled images of size J×K, the labeled images belonging to at least one failure category among a plurality of failure categories that cause the genotyping process to fail, wherein The labeled image comes from a segment of the image generating chip, positioning the J×K marker image at a plurality of locations in the M×N frame, filling the M×N frame using at least a portion of a specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame, further training the pre-trained convolutional neural network using the training dataset to generate a segment classifier; storing coefficients of the trained classifier to recognize and classify images of segments of the image generating chip from a production genotyping process cycle, whereby, independent of the corresponding genotyping call rates of the segments of the image generating chip, the trained classifier is capable of accepting an image of the segments of the image generating chip and classifying the genotyping process by the root cause of the failure of the genotyping process in the plurality of failure categories; as well as processing an image of a segment of the image generating chip as part of the genotyping process based on whether the genotyping process was classified as a success or a failure by the trained classifier, The processing thus further includes determining a root cause of the failure when the genotyping process is classified as a failure.

20. The method of claim 19, wherein filling around an edge of a specific J×K marker image using at least a portion of the specific J×K marker image further comprises: A horizontal reflection is applied to the at least a portion of the particular JxK mark image.

21. The method of claim 19, wherein filling around an edge of a specific J×K marker image using at least a portion of the specific J×K marker image further comprises: A vertical reflection is applied to the at least a portion of the particular JxK mark image.

22. The method of claim 19, wherein creating the training dataset using labeled images of size J×K further comprises: The JxK labeled image is generated by cropping out portions from a larger image and placing the JxK cropped portions in the MxN frame.

23. The method of claim 19, wherein the marked image of size J x K is obtained by downsampling a high-resolution image from a scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

24. The method of claim 23, wherein the high-resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

25. The method of claim 19, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

26. The method of claim 19, wherein the plurality of failure categories includes a remaining failure category indicating unhealthy patterns on the image due to unidentified failure causes.

27. A non-transitory computer readable storage medium having printed thereon computer program instructions for training a convolutional neural network to classify genotyping processes captured by images of segments of an image generating chip by root cause of genotyping process failure, the instructions when executed on a processor implementing a method comprising: Extracting image features using the pre-trained convolutional neural network, wherein the pre-trained convolutional neural network accepts an image of size M×N; A training dataset is created using labeled images of size J×K, the labeled images belonging to at least one failure category among a plurality of failure categories that cause the genotyping process to fail, wherein The labeled image comes from a segment of the image generating chip, positioning the J×K marker image at a plurality of locations in the M×N frame, filling the M×N frame using at least a portion of a specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame, further training the pre-trained convolutional neural network using the training dataset to generate a segment classifier; storing coefficients of the trained classifier to recognize and classify images of segments of the image generating chip from a production genotyping process cycle, Whereby, independent of the corresponding genotyping call rate of the segment of the image generating chip, the trained classifier is capable of accepting an image of the segment of the image generating chip and classifying the genotyping process by the root cause of the failure of the genotyping process in the plurality of failure categories; as well as processing an image of a segment of the image generating chip as part of the genotyping process based on whether the genotyping process was classified as a success or a failure by the trained classifier, The processing thus further includes determining a root cause of the failure when the genotyping process is classified as a failure.

28. The non-transitory computer-readable storage medium of claim 27, wherein using at least a portion of a specific J×K marker image to fill in around an edge of the specific J×K marker image, implementing the method further comprises: A horizontal reflection is applied to the at least a portion of the particular JxK mark image.

29. The non-transitory computer-readable storage medium of claim 27, wherein using at least a portion of a specific J×K marker image to fill in around an edge of the specific J×K marker image, implementing the method further comprises: A vertical reflection is applied to the at least a portion of the particular JxK mark image.

30. The non-transitory computer-readable storage medium of claim 27, wherein the training dataset is created using labeled images of size J×K, and implementing the method further comprises: The JxK labeled image is generated by cropping out portions from a larger image and placing the JxK cropped portions in the MxN frame.

31. The non-transitory computer-readable storage medium of claim 27, wherein the marked image of size J×K is obtained by downsampling a high-resolution image from a scanner so that the resolution of the high-resolution image is reduced by 1 / 2 to 1 / 50 on each side.

32. The non-transitory computer-readable storage medium of claim 31, wherein the high-resolution image obtained from a scanner has a size of 3600 x 1600 pixels.

33. The non-transitory computer readable storage medium of claim 27, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

34. The non-transitory computer-readable storage medium of claim 27, wherein the plurality of failure categories includes a remaining failure category that indicates an unhealthy pattern on the image due to an unidentified failure cause.

35. A system comprising one or more processors coupled to a memory, the memory loaded with computer instructions to train a convolutional neural network to classify genotyping processes captured by images of segments of an image generating chip by root cause of genotyping process failure, the instructions when executed on the processors performing actions comprising: Extracting image features using the pre-trained convolutional neural network, wherein the pre-trained convolutional neural network accepts an image of size M×N; A training dataset is created using labeled images of size J×K, the labeled images belonging to at least one failure category among a plurality of failure categories that cause the genotyping process to fail, wherein The labeled image comes from a segment of the image generating chip, positioning the J×K marker image at a plurality of locations in the M×N frame, filling the M×N frame using at least a portion of a specific J×K marker image to fill around edges of the specific J×K marker image, thereby filling the M×N frame, further training the pre-trained convolutional neural network using the training dataset to generate a segment classifier; storing coefficients of the trained classifier to recognize and classify images of segments of the image generating chip from a production genotyping process cycle, Whereby, independent of the corresponding genotyping call rate of the segment of the image generating chip, the trained classifier is capable of accepting an image of the segment of the image generating chip and classifying the genotyping process by the root cause of the failure of the genotyping process in the plurality of failure categories; as well as processing an image of a segment of the image generating chip as part of the genotyping process based on whether the genotyping process was classified as a success or a failure by the trained classifier, The processing thus further includes determining a root cause of the failure when the genotyping process is classified as a failure.

36. The system of claim 35, wherein filling around an edge of a particular JxK marker image using at least a portion of the particular JxK marker image further implements actions comprising: A horizontal reflection is applied to the at least a portion of the particular JxK mark image.

37. The system of claim 35, wherein filling around an edge of a particular JxK marker image using at least a portion of the particular JxK marker image further implements actions comprising: A vertical reflection is applied to the at least a portion of the particular JxK mark image.

38. The system of claim 35, wherein the training dataset is created using labeled images of size J×K, further implementing actions comprising: The JxK labeled image is generated by cropping out portions from a larger image and placing the JxK cropped portions in the MxN frame.

39. The system of claim 35, wherein the marked image of size J x K is obtained by downsampling a high resolution image from a scanner such that the resolution of the high resolution image is reduced by 1 / 2 to 1 / 50 on each side.

40. The system of claim 39, wherein the high resolution image obtained from the scanner has a size of 3600 x 1600 pixels.

41. The system of claim 35, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

42. The system of claim 35, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.

43. A method of identifying a genotyping process cycle image of a segment of an image generating chip as depicting a failure of a genotyping process cycle, comprising: Create the input to the trained classifier, including: Access an image in segments of size J × K, Position the J×K image in the M×N analysis frame, filling the M×N analysis frame using at least a portion of a specific J×K marker image to fill around an edge of the specific J×K marker image, thereby filling the M×N analysis frame, Input the M×N analysis frame to the trained classifier, classifying, using the trained classifier, a genotyping process cycle captured by the image of the segment of the image generating chip as normal or belonging to at least one failure category of a plurality of failure categories applicable to the genotyping process cycle, outputting a resulting classification for the segment of the image generating chip independent of the corresponding genotyping call rate for the segment of the image generating chip, and An image of a segment of the image generating chip is processed based on whether a genotyping process cycle captured by the image is classified by the trained classifier as normal and depicting a successful process or belongs to the at least one failure category.

44. The method of claim 43, further comprising positioning the JxK image at the center of the MxN analysis frame.

45. The method of claim 43, wherein filling around an edge of a specific J×K image using at least a portion of the specific J×K image further comprises: A horizontal reflection is applied to the at least a portion of the particular JxK image.

46. ​​The method of claim 43, wherein filling around an edge of a specific J×K image using at least a portion of the specific J×K image further comprises: A vertical reflection is applied to the at least a portion of the particular JxK image.

47. The method of claim 43, wherein creating the input to the trained classifier further comprises: The JxK image is generated by cropping portions from a larger image and placing the JxK cropped portions in the MxN frame.

48. The method of claim 43, wherein the JxK image is obtained by downsampling a high resolution image from a scanner such that the resolution of the high resolution image is reduced by 1 / 2 to 1 / 50 on each side.

49. The method of claim 43, wherein the plurality of failure categories include at least hybridization failure, spatial shift failure, offset failure, surface abrasion failure, and reagent flow failure.

50. The method of claim 43, wherein the plurality of failure categories includes a remaining failure category that indicates unhealthy patterns on the image due to unidentified failure causes.