Quality scoring based on artificial intelligence

Abstract

The disclosure relates to artificial intelligence-based quality scoring that processes a first input through a first neural network and produces a first output. The first input includes first image data derived from an image of an analyte and its surrounding background captured by the sequencing system in a sequencing run. The disclosed techniques process the first output by a post-processor and generate metadata regarding the analyte and its surrounding context. The disclosed techniques process a second input through a second neural network and produce a second output. The second input includes third image data derived by modifying the second image data based on the metadata. The second image data is derived from the image of the analyte and its surrounding background. The second output identifies base detection of one or more of the analytes at one or more sequencing cycles of the sequencing run.

Description

[0001] This application is a divisional application of the application with application number 202080004529.4, application date March 22, 2020, and invention name “Artificial Intelligence-Based Sequencing”.

[0002] Priority application

[0003] This patent application claims priority or the benefit of the following patent applications:

[0004] U.S. Provisional Patent Application No. 62 / 821,602, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-1 / IP-1693-PRV);

[0005] U.S. Provisional Patent Application No. 62 / 821,618, entitled “Artificial Intelligence-Based Generation of Sequencing Metadata,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-3 / IP-1741-PRV);

[0006] U.S. Provisional Patent Application No. 62 / 821,681, entitled “Artificial Intelligence-Based Base Calling,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-4 / IP-1744-PRV);

[0007] U.S. Provisional Patent Application No. 62 / 821,724, entitled “Artificial Intelligence-Based QualityScoring,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-7 / IP-1747-PRV);

[0008] U.S. Provisional Patent Application No. 62 / 821,766, entitled “Artificial Intelligence-Based Sequencing,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-9 / IP-1752-PRV);

[0009] Netherlands Patent Application No. 2023310, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on June 14, 2019 (Attorney Docket No. ILLM1008-11 / IP-1693-NL);

[0010] Netherlands Patent Application No. 2023311, entitled “Artificial Intelligence-Based Generation of Sequencing Metadata,” filed on June 14, 2019 (attorney docket number ILLM 1008-12 / IP-1741-NL);

[0011] Netherlands Patent Application No. 2023312, entitled “Artificial Intelligence-Based Base Calling,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-13 / IP-1744-NL);

[0012] Netherlands Patent Application No. 2023314, entitled “Artificial Intelligence-Based QualityScoring,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-14 / IP-1747-NL); and

[0013] Netherlands Patent Application No. 2023316, entitled “Artificial Intelligence-Based Sequencing,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-15 / IP-1752-NL).

[0014] U.S. nonprovisional patent application No. 16 / 825,987, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-16 / IP-1693-US);

[0015] U.S. nonprovisional patent application No. 16 / 825,991, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-17 / IP-1741-US);

[0016] U.S. nonprovisional patent application No. 16 / 826,126, entitled “Artificial Intelligence-Based Base Calling,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-18 / IP-1744-US);

[0017] U.S. non-provisional patent application No. 16 / 826,134, entitled “Artificial Intelligence-Based QualityScoring,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-19 / IP-1747-US);

[0018] U.S. nonprovisional patent application No. 16 / 826,168, entitled “Artificial Intelligence-Based Sequencing,” filed on March 21, 2020 (Attorney Docket No. ILLM 1008-20 / IP-1752-PRV);

[0019] PCT Patent Application No. PCT________, entitled “Training Data Generation for ArtificialIntelligence-Based Sequencing”, filed concurrently (Attorney Docket No. ILLM1008-21 / IP-1693-PCT), which was subsequently published as PCT Publication No. WO________;

[0020] concurrently filed PCT Patent Application No. PCT________, entitled “Artificial Intelligence-Based Generation of Sequencing Metadata” (Attorney Docket No. ILLM 1008-22 / IP-1741-PCT), which was subsequently published as PCT Publication No. WO________;

[0021] concurrently filed PCT Patent Application No. PCT________, entitled “Artificial Intelligence-Based Base Calling” (Attorney Docket No. ILLM 1008-23 / IP-1744-PCT), which subsequently published as PCT Publication No. WO________; and

[0022] A PCT patent application entitled “Artificial Intelligence-Based Quality Scoring” was filed at the same time as PCT Patent Application No. PCT________ (Agent Docket No. ILLM 1008-24 / IP-1747-PCT), which was subsequently published as PCT Publication No. WO________.

[0023] These priority applications are hereby incorporated by reference as if fully set forth herein for all purposes.

[0024] Literature Incorporation

[0025] The following documents are incorporated by reference as if fully set forth herein for all purposes:

[0026] U.S. Provisional Patent Application No. 62 / 849,091, entitled “Systems and Devices for Characterization and Performance Analysis of Pixel-Based Sequencing,” filed on May 16, 2019 (Attorney Docket No. ILLM 1011-1 / IP-1750-PRV);

[0027] U.S. Provisional Patent Application No. 62 / 849,132, entitled “Base Calling Using Convolutions,” filed on May 16, 2019 (Attorney Docket No. ILLM 1011-2 / IP-1750-PR2);

[0028] U.S. Provisional Patent Application No. 62 / 849,133, entitled “Base Calling Using Compact Convolutions,” filed on May 16, 2019 (Attorney Docket No. ILLM 1011-3 / IP-1750-PR3);

[0029] U.S. Provisional Patent Application No. 62 / 979,384, entitled “Artificial Intelligence-Based Base Calling of Index Sequences,” filed on February 20, 2020 (Attorney Docket No. ILLM 1015-1 / IP-1857-PRV);

[0030] U.S. Provisional Patent Application No. 62 / 979,414, entitled “Artificial Intelligence-Based Many-To-ManyBase Calling,” filed on February 20, 2020 (Attorney Docket No. ILLM 1016-1 / IP-1858-PRV);

[0031] U.S. Provisional Patent Application No. 62 / 979,385, entitled “Knowledge Distillation-Based Compression of Artificial Intelligence-Based Base Caller,” filed on February 20, 2020 (Attorney Docket No. ILLM 1017-1 / IP-1859-PRV);

[0032] U.S. Provisional Patent Application No. 62 / 979,412, filed on February 20, 2020, entitled “Multi-Cycle Cluster Based Real Time Analysis System” (Attorney Docket No. ILLM 1020-1 / IP-1866-PRV);

[0033] U.S. Provisional Patent Application No. 62 / 979,411, entitled “Data Compression for Artificial Intelligence-Based Base Calling,” filed on February 20, 2020 (Attorney Docket No. ILLM 1029-1 / IP-1964-PRV);

[0034] U.S. Provisional Patent Application No. 62 / 979,399, entitled “Squeezing Layer for Artificial Intelligence-Based Base Calling,” filed on February 20, 2020 (Attorney Docket No. ILLM 1030-1 / IP-1982-PRV);

[0035] Liu P, Hemani A, Paul K, Weis C, Jung M, Wehn ​​N, “3D-Stacked Many-Core Architecture for Biological Sequence Analysis Problems,” Int JParallel Prog, 2017, Vol. 45(No. 6): pp. 1420–1460;

[0036] Z. Wu, K. Hammad, R. Mittmann, S. Magierowski, E. Ghafar-Zadeh, and X. Zhong, “FPGA-Based DNA Basecalling Hardware Acceleration,” Proc. IEEE 61st Int. Midwest Symp. Circuits Syst., August 2018, pp. 1098–1101;

[0037] Z. Wu, K. Hammad, E. Ghafar-Zadeh, and S. Magierowski, “FPGA-Accelerated 3rd Generation DNA Sequencing,” IEEE Transactions on Biomedical Circuits and Systems, Vol. 14, No. 1, February 2020, pp. 65–74;

[0038] Prabhakar et al., “Plasticine: A Reconfigurable Architecture for Parallel Patterns,” ISCA 17, June 24-28, 2017, Toronto, ON, Canada;

[0039] M.Lin, Q.Chen and S.Yan, "Network in Network", Proc.of ICLR, 2014;

[0040] L.Sifre, "Rigid-motion Scattering for Image Classification", Ph.D. thesis, 2014;

[0041] L. Sifre and S. Mallat, "Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination", Proc. of CVPR, 2013;

[0042] F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions", Proc. of CVPR, 2017;

[0043] X. Zhang, X. Zhou, M. Lin and J. Sun, "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," arXiv:1707.01083, 2017;

[0044] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition", Proc. of CVPR, 2016;

[0045] S. Xie, R. Girshick, P. Dollár, Z. Tu and K. He, "Aggregated Residual Transformations for Deep Neural Networks", Proc. of CVPR, 2017;

[0046] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, "Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications", arXiv:1704.04861, 2017;

[0047] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “MobileNetV2: InvertedResiduals and Linear Bottlenecks,” arXiv:1801.04381v3, 2018;

[0048] Z. Qin, Z. Zhang, X. Chen, and Y. Peng, “FD-MobileNet: Improved Mobile Net with aFast Downsampling Strategy,” arXiv:1802.03750,2018;

[0049] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam, “Rethinking Atrous Convolution for Semantic Image Segmentation,” CoRR, abs / 1706.05587, 2017;

[0050] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, and S. Guadarrama, “Speed / accuracy trade-offs for modern convolutionalobject detectors,” arXiv preprint arXiv:1611.10012,2016;

[0051] S.Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WAVENET:A GENERATIVE MODEL FOR RAW AUDIO”, arXiv:1609.03499,2016;

[0052] S. Arik, M., Chrzanowski, A., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., Sengupta, S., & Shoeybi, M. “DEEP VOICE: REAL-TIME NEURAL TEXT-TO-SPEECH”. arXiv:1702.07825, 2017;

[0053] Yu, F., & Koltun, V. “MULTI-SCALE CONTEXT AGGREGATION BY DILATED CONVOLUTIONS”. arXiv:1511.07122, 2016;

[0054] He, K., Zhang, X., Ren, S., & Sun, J. “DEEP RESIDUAL LEARNING FOR IMAGE RECOGNITION”. arXiv:1512.03385, 2015;

[0055] Srivastava, R. K., Greff, K., & Schmidhuber, J. “HIGHWAY NETWORKS”. arXiv:1505.00387, 2015;

[0056] Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. Q. “DENSELY CONNECTED CONVOLUTIONAL NETWORKS”. arXiv:1608.06993, 2017;

[0057] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. “GOING DEEPER WITH CONVOLUTIONS”. arXiv:1409.4842, 2014;

[0058] Ioffe, S., & Szegedy, C. “BATCH NORMALIZATION: ACCELERATING DEEP NETWORK TRAINING BY REDUCING INTERNAL COVARIATE SHIFT”. arXiv:1502.03167, 2015;

[0059] JMWolterink, T.Leiner, MAViergever and I. "DILATED CONVOLUTIONALNEURAL NETWORKS FOR CARDIOVASCULAR MR SEGMENTATION IN CONGENITAL HEARTDISEASE", arXiv:1704.03669, 2017;

[0060] L.C. Piqueras, “Automatic Model Based on a Deep Convolutional Neural Network for Audio Generation,” Tampere University of Technology, 2016;

[0061] J.Wu, “Introduction to Convolutional Neural Networks”, Nanjing University, 2017;

[0062] "Illumina CMOS Chip and One-Channel SBS Chemistry", Illumina, Inc, 2018, page 2;

[0063] “skikit-image / peak.py at master”, GitHub, page 5, [retrieved on November 16, 2018], retrieved from the Internet <URL: https: / / github.com / scikit-image / scikit-image / blob / master / skimage / feature / peak.py#L25 >

[0064] "3.3.9.11. Watershed and random walker for segmentation", Scipy lecturenotes, page 2, [retrieved on November 13, 2018], retrieved from the Internet<URL:http: / / scipy-lectures.org / packages / scikit-image / auto_examples / plot_segmentations.html> ;

[0065] Mordvintsev, Alexander and Revision, Abid K., "Image Segmentation with Watershed Algorithm", version 43532856, 2013, page 6, [retrieved on November 13, 2018], retrieved from the Internet <URL:https: / / opencv-python-tutroals.readthedocs.io / en / latest / py_tutorials / py_imgproc / py_watershed / py_water shed.html> ;

[0066] Mzur, “Watershed.py”, October 25, 2017, page 3, [retrieved November 13, 2018], retrieved from the Internet<URL:https: / / github.com / mzur / watershed / blob / master / Watershed.py> ;

[0067] Thakur, Pratibha, et al., “A Survey of Image Segmentation Techniques,” International Journal of Research in Computer Applications and Robotics, Vol. 2, No. 4, April 2014, pp. 158-165;

[0068] Long, Jonathan et al., “Fully Convolutional Networks for Semantic Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, vol. 4, April 1, 2017, p. 10.

[0069] Ronneberger, Olaf, et al., “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Medical image computing and computer-assisted intervention, May 18, 2015, page 8;

[0070] Xie, W. et al., “Microscopy cell counting and detection with fully convolutional regression networks”, Computer methods in biomechanics and biomedical engineering: Imaging & Visualization, Issue 6 (Volume 3), pp. 283-292, 2018;

[0071] Xie, Yuanpu et al., "Beyond classification: structured regression for robust cell detection using convolutional neural network", International Conference on Medical Image Computing and Computer-Assisted Intervention, October 12, 2015, page 12;

[0072] Snuverink, IAF, “Deep Learning for Pixelwise Classification of Hyperspectral Images,” Master of Science Thesis, TU Delft, November 23, 2017, page 19;

[0073] Shevchenko, A., "Keras weighted categorical_crossentropy", Page 1, [Retrieved on January 15, 2019], retrieved from the Internet<URL:https: / / gist.github.com / skeeet / cad06d584548fb45eece1d4e28cfa98b> ;

[0074] van den Assem, DCF, “Predicting periodic and chaotic signals using Wavenets,” Master of Science thesis, Delft University of Technology, August 18, 2017, pp. 3-38;

[0075] I.J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “CONVOLUTIONAL NETWORKS,” Deep Learning, MIT Press, 2016; and

[0076] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, and G. Wang, “RECENT ADVANCES IN CONVOLUTIONAL NEURAL NETWORKS,” arXiv:1512.07108, 2017. Technical Field

[0077] The technology disclosed in the present invention relates to artificial intelligence type computers and digital data processing systems and corresponding data processing methods and products for simulating intelligence (i.e., knowledge-based systems, inference systems, and knowledge acquisition systems); and includes systems for uncertainty inference (e.g., fuzzy logic systems), adaptive systems, machine learning systems, and artificial neural networks. Specifically, the technology disclosed relates to the use of deep neural networks such as deep convolutional neural networks for analyzing data. Background Art

[0078] The subject matter discussed in this section should not be considered to be prior art simply because it is mentioned in this section. Similarly, problems mentioned in this section or associated with the subject matter provided as background technology should not be considered to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which themselves may also correspond to specific implementations of the technology protected by the claims.

[0079] A deep neural network is an artificial neural network that uses multiple nonlinear and complex transformation layers to continuously model high-level features. Deep neural networks provide feedback via backpropagation, which carries the difference between observed and predicted outputs to adjust parameters. Deep neural networks have evolved with the availability of large training data sets, the power of parallel and distributed computing, and sophisticated training algorithms. Deep neural networks have led to significant advances in many fields such as computer vision, speech recognition, and natural language processing.

[0080] Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are components of deep neural networks. Convolutional neural networks are particularly successful in image recognition with an architecture that includes convolutional layers, nonlinear layers, and pooling layers. Recurrent neural networks are designed to exploit the sequential information of input data and have cyclic connections between building blocks such as perceptrons, long short-term memory units, and gated recurrent units. In addition, many other emerging deep neural networks for limited contexts have been proposed, such as deep spatiotemporal neural networks, multidimensional recurrent neural networks, and convolutional autoencoders.

[0081] The goal of training a deep neural network is to optimize the weight parameters in each layer, which gradually combines simpler features into complex features so that the most appropriate hierarchical representation can be learned from the data. A single cycle of the optimization process is performed as follows. First, given a training data set, the forward pass sequentially calculates the output in each layer and propagates the function signal forward through the network. In the final output layer, the target loss function measures the error between the inferred output and the given label. In order to minimize the training error, the backward pass uses the chain rule to backpropagate the error signal and calculate the gradient relative to all weights in the entire neural network. Finally, the weight parameters are updated using an optimization algorithm based on stochastic gradient descent. While batch gradient descent performs parameter updates for each complete data set, stochastic gradient descent provides a stochastic approximation by performing updates for each small data example set. Several optimization algorithms are derived from stochastic gradient descent. For example, the Adagrad and Adam training algorithms perform stochastic gradient descent while adaptively modifying the learning rate based on the update frequency and momentum of the gradient of each parameter, respectively.

[0082] Another core element in deep neural network training is regularization, which refers to strategies designed to avoid overfitting and thus achieve good generalization performance. For example, weight decay adds a penalty factor to the objective loss function so that the weight parameters converge to a small absolute value. Dropout randomly removes hidden units from a neural network during training and can be thought of as an ensemble of possible subnetworks. To enhance the capabilities of dropout, new activation functions, maxout, and a dropout variant of recurrent neural networks (called rnnDrop) have been proposed. In addition, batch normalization provides a new regularization method by normalizing the scalar features of each activation within a mini-batch and learning each mean and variance as parameters.

[0083] Given that sequence data is multidimensional and high-dimensional, deep neural networks hold great promise in bioinformatics research due to their broad applicability and enhanced predictive power. Convolutional neural networks have been used to solve sequence-based problems in genomics, such as motif discovery, pathogenic variant identification, and gene expression inference. Convolutional neural networks use a weight-sharing strategy that is particularly useful for studying DNA because it can capture sequence motifs, which are short, recurring local patterns in DNA that are assumed to have significant biological functions. The hallmark of convolutional neural networks is the use of convolutional filters.

[0084] Unlike traditional classification methods based on carefully designed features and handcrafted features, convolutional filters perform adaptive learning of features, similar to the process of mapping raw input data to information representations of knowledge. In this sense, convolutional filters act as a series of motif scanners, because a group of such filters are able to recognize relevant patterns in the input and update themselves during the training process. Recurrent neural networks can capture long-range dependencies in sequence data with different lengths, such as protein or DNA sequences.

[0085] Therefore, there is an opportunity to use a principled deep learning based framework for template generation and base calling.

[0086] In the era of high-throughput technologies, accumulating the most interpretable data at the lowest cost per work remains a major challenge. Cluster-based nucleic acid sequencing methods, such as those that use bridge amplification to form clusters, have made important contributions to the goal of increasing nucleic acid sequencing throughput. These cluster-based methods rely on sequencing dense populations of nucleic acids immobilized on solid supports and typically involve the use of image analysis software to deconvolute the optical signals generated during the simultaneous sequencing of multiple clusters located at different locations on the solid support.

[0087] However, this type of sequencing technology based on solid phase nucleic acid clusters is still faced with considerable obstacles, which limit achievable throughput. For example, in the sequencing method based on clusters, obstacles may be present in determining that the nucleic acid sequences of two or more clusters that are too close to each other physically cannot be spatially resolved or actually physically overlapped on a solid support. For example, current image analysis software may require valuable time and computing resources to determine which cluster in two overlapping clusters has emitted a light signal. Therefore, for a variety of detection platforms, the compromise about the quantity and / or quality of obtainable nucleic acid sequence information is inevitable.

[0088] Genomic methods based on high-density nucleic acid clusters have also expanded to other areas of genome analysis. For example, nucleic acid cluster-based genomics can be used for sequencing applications, diagnostics and screening, gene expression analysis, epigenetic analysis, polymorphic genetic analysis, etc. Each of these nucleic acid cluster-based genomic technologies is also limited when data generated by nucleic acid clusters that are in close proximity or spatially overlapping cannot be parsed.

[0089] Clearly, there remains a need to increase the quality and quantity of nucleic acid sequencing data that can be rapidly and cost-effectively obtained for a wide variety of uses, including genomics (e.g., for genomic characterization of any and all animal, plant, microbial or other biological species or populations), pharmacogenetics, transcriptomics, diagnostics, prognosis, biomedical risk assessment, clinical and research genetics, personalized medicine, drug efficacy and drug interaction assessment, veterinary medicine, agriculture, evolutionary and biodiversity studies, aquaculture, forestry, oceanography, ecological and environmental management, and other purposes.

[0090] The disclosed technology provides neural network-based methods and systems that address these and similar needs, including increasing throughput levels in high-throughput nucleic acid sequencing technologies, and provide other related advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] This patent or patent application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. Color drawings are also available in pairs via the Supplemental Content tab.

[0092] In the drawings, similar reference symbols generally refer to similar parts throughout the different views. In addition, the drawings are not necessarily drawn to scale, but rather focus on illustrating the principles of the disclosed technology. In the following description, various specific implementations of the disclosed technology are described with reference to the following drawings, in which:

[0093] Figure 1 One specific implementation of a processing pipeline that uses sub-pixel base calling to determine cluster metadata is shown.

[0094] Figure 2 One specific implementation of a flow cell containing clusters within its blocks is depicted.

[0095] Figure 3 An example of an Illumina GA-IIx flow cell with eight channels is shown.

[0096] Figure 4Depicted is an image set of sequencing images for four-channel chemistry, ie, the image set has four sequencing images captured using four different wavelength bands (images / imaging channels) in the pixel domain.

[0097] Figure 5 It is a specific implementation of dividing the sequencing image into sub-pixels (or sub-pixel regions).

[0098] Figure 6 The initial center coordinates of the clusters identified by the base caller during sub-pixel base calling are shown.

[0099] Figure 7 Depicted is one specific implementation of merging sub-pixel base calls generated over multiple sequencing cycles to generate a so-called "cluster map" containing cluster metadata.

[0100] Figure 8a An example of a cluster map generated by merging sub-pixel base calls is shown.

[0101] Figure 8b Depicted is a specific implementation of sub-pixel base calling.

[0102] Fig. 9 Another example of a cluster map identifying cluster metadata is shown.

[0103] Fig.10 It shows how to calculate the centroid (COM) of disjoint regions in a cluster map.

[0104] Fig.11 One specific implementation of calculating the weighted attenuation factor based on the Euclidean distance of sub-pixels in a disjoint region from the COM of the disjoint region is depicted.

[0105] Fig.12 One specific implementation of an exemplary ground truth decay map derived from an exemplary cluster map produced by sub-pixel base calling is shown.

[0106] Fig.13 A specific implementation of deriving a ternary map from a cluster map is shown.

[0107] Fig.14 A specific implementation of deriving a binary map from a cluster map is shown.

[0108] Fig.15 is a block diagram illustrating one specific implementation of generating training data for training a neural network-based template generator and a neural network-based base caller.

[0109] Fig.16Features of the disclosed training examples used to train a neural network-based template generator and a neural network-based base caller are shown.

[0110] Fig.17 A specific implementation of processing input image data through a disclosed neural network-based template generator and generating output values ​​for each cell in an array is shown. In one specific implementation, the array is an attenuation map. In another specific implementation, the array is a ternary map. In yet another specific implementation, the array is a binary map.

[0111] Fig.18 One specific implementation of a post-processing technique is shown, which is applied to an attenuation map, a ternary map, or a binary map produced by a neural network-based template generator to derive cluster metadata, including cluster center, cluster shape, cluster size, cluster background, and / or cluster border.

[0112] Fig.19 One specific implementation of extracting cluster intensities in the pixel domain is depicted.

[0113] Fig. 20 One specific implementation of extracting cluster intensities in the sub-pixel domain is shown.

[0114] Fig.21a Three different implementations of a neural network based template generator are shown.

[0115] Figure 21b Depicted is one specific implementation of input image data fed as input to a neural network-based template generator 1512. The input image data includes a series of image sets having sequencing images generated during a certain number of initial sequencing cycles of a sequencing run.

[0116] Fig. 22 Shows from Figure 21b A specific implementation of extracting patches from a series of image sets in order to produce a series of "reduced-size" image sets forming input image data.

[0117] Fig.23 Describes the Figure 21b A specific implementation of upsampling a series of image sets in to produce a series of "upsampled" image sets forming the input image data.

[0118] Fig.24 Shows from Fig.23 A specific implementation of extracting patches from a series of upsampled image sets in to produce a series of "upsampled and downsized" image sets forming the input image data.

[0119] Fig.25One specific implementation of an overall exemplary process for generating ground truth data for training a neural network-based template generator is shown.

[0120] Fig.26 A specific implementation of the regression model is shown.

[0121] Fig. 27 A specific implementation of generating a ground truth decay map from a cluster map is described. The ground truth decay map is used as ground truth data for training a regression model.

[0122] Fig.28 It is a specific implementation of using back-propagation based gradient update technique to train regression models.

[0123] Fig.29 is a specific implementation of generating templates through a regression model during inference.

[0124] Fig.30 One specific implementation of subjecting the attenuation map to post-processing to identify cluster metadata is shown.

[0125] Fig.31 Depicted is a specific implementation of a watershed segmentation technique for identifying non-overlapping adjacent clusters / groups of intra-cluster sub-pixels that characterize clusters.

[0126] Fig.32 is a table showing an exemplary U-Net architecture for a regression model.

[0127] Fig.33 Different methods of extracting cluster intensities using cluster shape information identified in a template image are shown.

[0128] Fig.34 Different methods of using the output of a regression model for base calling are shown.

[0129] Fig.35 The difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center is shown. The results show that using the COM improves base calling.

[0130] Fig.36 An exemplary decay map produced by the regression model is shown on the left. Fig.36 Also shown on the right is an exemplary ground truth decay map that the regression model approximates during training.

[0131] Fig.37 One specific implementation of a peak locator for identifying cluster centers in an attenuation map by detecting peaks is depicted.

[0132] Fig.38Peaks detected by the peak locator in the attenuation map produced by the regression model are compared with corresponding peaks in the ground truth attenuation map.

[0133] Fig.39 The performance of the regression model using precision and recall statistics is shown.

[0134] Fig.40 The performance of the regression model and the RTA base caller were compared for a library concentration of 20 pM (normal operation).

[0135] Fig.41 The performance of the regression model and the RTA base caller were compared for a 30 pM library concentration (dense run).

[0136] Fig.42 The number of non-duplicate correct read pairs detected by the regression model (ie, the number of paired reads where neither read aligned inward within a reasonable distance) was compared to the number of non-duplicate correct read pairs detected by the RTA base caller.

[0137] Fig.43 A first attenuation map resulting from the regression model is shown on the right. Fig.43 A second decay map resulting from the regression model is shown on the left.

[0138] Fig.44 The performance of the regression model and the RTA base caller were compared for a 40 pM library concentration (a highly dense run).

[0139] Fig.45 A first attenuation map resulting from the regression model is shown on the left. Fig.45 The results of the thresholding, peak location and watershed segmentation techniques applied to the first attenuation map are shown on the right.

[0140] Fig.46 A specific implementation of a binary classification model is shown.

[0141] Fig.47 It is a specific implementation for training a binary classification model using a back-propagation based gradient update technique involving the softmax score.

[0142] Fig.48 is another implementation for training a binary classification model using a backpropagation-based gradient update technique involving a sigmoid score.

[0143] Fig.49 Another specific implementation of input image data fed to a binary classification model and corresponding class labels for training the binary classification model is shown.

[0144] Fig.50 is a specific implementation of generating templates by a binary classification model during inference.

[0145] Fig.51 One specific implementation of subjecting the binary map to peak detection to identify cluster centers is shown.

[0146] Fig.52a An exemplary binary map produced by a binary classification model is shown on the left. Fig.52a Also shown on the right is an exemplary ground truth binary map that the binary classification model approaches during training.

[0147] Fig.52b The performance of a binary classification model using the precision statistic is shown.

[0148] Fig.53 is a table showing an example architecture for a binary classification model.

[0149] Fig.54 A specific implementation of the ternary classification model is shown.

[0150] Fig.55 It is a specific implementation of training a ternary classification model using a back-propagation based gradient update technique.

[0151] Fig.56 Another specific implementation of input image data fed to a ternary classification model and corresponding class labels for training the ternary classification model is shown.

[0152] Fig.57 is a table showing an example architecture for a ternary classification model.

[0153] Fig.58 is a specific implementation of generating templates through a ternary classification model during inference.

[0154] Fig.59 The ternary map produced by the ternary classification model is shown.

[0155] Fig.60 Depicted is the cell array produced by the ternary classification model 5400, along with the cell-by-cell output values.

[0156] Fig.61 One specific implementation of post-processing the ternary map to identify cluster centers, cluster backgrounds, and cluster interiors is shown.

[0157] Fig.62a Exemplary predictions for a three-class classification model are shown.

[0158] Figure 62b Other exemplary predictions for the ternary classification model are shown.

[0159] Figure 62c Other exemplary predictions for the ternary classification model are also shown.

[0160] Fig.63 Describes from Fig.62a A specific implementation of deriving cluster centers and cluster shapes from the output of the ternary classification model in .

[0161] Fig.64 The base calling performance of binary classification models, regression models, and the RTA base caller were compared.

[0162] Fig.65 The performance of the ternary classification model was compared with that of the RTA base caller under three contexts, five sequencing metrics, and two run densities.

[0163] Fig.66 The performance of the regression model was compared with that of the RTA base caller. Fig.65 The comparisons were performed under the three contexts discussed in [ 14 ], five sequencing metrics, and two run densities.

[0164] Fig.67 Focus on the penultimate layer of the neural network-based template generator.

[0165] Fig.68 Visualizes what the penultimate layer of the neural network-based template generator has learned due to back-propagation-based gradient update training. The example implementation visualizes Fig.67 24 of the 32 trained convolutional filters of the penultimate layer depicted in .

[0166] Fig.69 The cluster center predictions of the binary classification model (blue) are superimposed on those of the RTA base caller (pink).

[0167] Fig.70 Cluster center predictions (pink) made by the RTA base caller are superimposed on a visualization of the trained convolutional filters of the penultimate layer of the binary classification model.

[0168] Fig.71 One specific implementation of training data for training a neural network based template generator is shown.

[0169] Fig.72 It is a specific implementation of image registration using beads based on cluster center prediction of a neural network-based template generator.

[0170] Fig.73 One specific implementation of cluster statistics for clusters identified by a neural network-based template generator is shown.

[0171] Fig.74 It shows how the ability of the neural network-based template generator to distinguish adjacent clusters improves when the number of initial sequencing cycles using input image data increases from five to seven.

[0172] Fig.75 Shown is the difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center.

[0173] Fig.76 The performance of the neural network-based template generator for additional detected clusters is plotted.

[0174] Fig.77 Different datasets used to train a neural network based template generator are shown.

[0175] Fig.78 The processing stages used by an RTA base caller for base calling are shown according to one specific implementation.

[0176] Fig.79 One specific implementation of base calling using the disclosed neural network-based base caller is illustrated.

[0177] Fig.80 It is a specific implementation of transforming the location / position information of cluster centers identified from the output of a neural network-based template generator from the sub-pixel domain to the pixel domain.

[0178] Fig.81 is a specific implementation that uses a cycle-specific transformation and an image channel-specific transformation to derive so-called "transformed cluster centers" from reference cluster centers.

[0179] Fig.82 An image patch is illustrated which is part of the input data fed to a neural network based base caller.

[0180] Fig.83 Depicted is one specific implementation of determining distance values ​​for a distance channel when a single target cluster is being base called by a neural network based base caller.

[0181] Fig.84 A specific implementation of encoding the distance value between the calculated pixel and the target cluster pixel by pixel is shown.

[0182] Fig.85a Depicted is one specific implementation of determining distance values ​​for a distance channel when multiple target clusters are simultaneously base called by a neural network based base caller.

[0183] Figure 85b It is shown that for each target cluster in the target clusters, some nearest pixels are determined based on the distance from the pixel center to the nearest cluster center.

[0184] Fig.86 A specific implementation of pixel-by-pixel encoding of the calculated minimum distance value between a pixel and the nearest cluster in a cluster is shown.

[0185] Fig.87 One specific implementation of pixel-to-cluster classification / attribution / categorization using what is referred to herein as "cluster shape data" is illustrated.

[0186] Fig.88 A specific implementation of calculating distance values ​​using cluster shape data is shown.

[0187] Fig.89 One specific implementation of pixel-by-pixel encoding of the calculated distance values ​​between pixels and assigned clusters is shown.

[0188] Fig.90 One specific implementation of a specialized architecture for a neural network-based base caller that isolates the processing of data from different sequencing cycles is illustrated.

[0189] Fig.91 Depicts a specific implementation of isolated convolution.

[0190] Fig.92a Depicts a concrete implementation of combined convolution.

[0191] Figure 92b Depicts another implementation of combined convolution.

[0192] Fig.93 One specific implementation of the convolutional layers of a neural network based base caller is shown, where each convolutional layer has a set of convolutional filters.

[0193] Fig.94 Depicted are two configurations of scaling channels that complement image channels.

[0194] Fig.95a One specific implementation of input data for a single sequencing cycle that produces a red image and a green image is illustrated.

[0195] Fig.95b One specific implementation of a range channel that provides an additive bias that is incorporated into the feature maps generated from the image channels is illustrated.

[0196] Fig.96a , Fig.96b and Fig.96c Depicted is a specific implementation of base calling for a single target cluster.

[0197] Fig.97 One specific implementation of simultaneous base calling for multiple target clusters is shown.

[0198] Fig.98 A specific implementation of simultaneously performing base calling on multiple target clusters at multiple subsequent sequencing cycles to simultaneously generate a base called sequence for each of the multiple target clusters is shown.

[0199] Fig.99 A dimensional diagram for a single cluster base calling implementation is illustrated.

[0200] Fig.100 A dimensional diagram illustrating a multiple cluster, single sequencing cycle base calling implementation.

[0201] Fig.101 A dimensional diagram illustrating a specific implementation of base calling for multiple clusters and multiple sequencing cycles is shown.

[0202] Fig.102a An exemplary array input configuration for multi-cycle input data is depicted.

[0203] Fig.102b An exemplary stacked input configuration for multi-cycle input data is shown.

[0204] Fig.103a One specific implementation in which pixels of an image patch are reconstructed so that the center of a base called target cluster is centered in a center pixel is depicted.

[0205] Fig.103b Another exemplary reconstructed / shifted image patch is depicted, where (i) the center of the central pixel coincides with the center of the target cluster, and (ii) the non-central pixels are equidistant from the center of the target cluster.

[0206] Fig.104 One specific implementation of base calling for a single target cluster at the current sequencing cycle using a standard convolutional neural network and reconstructed input is shown.

[0207] Fig.105 One specific implementation of base calling for multiple target clusters at the current sequencing cycle using a standard convolutional neural network and aligned inputs is shown.

[0208] Fig.106 One specific implementation of base calling for multiple target clusters at multiple sequencing cycles using a standard convolutional neural network and aligned inputs is shown.

[0209] Fig.107 One specific implementation of training a neural network based base caller is shown.

[0210] Fig.108a One specific implementation of a hybrid neural network for use as a neural network-based base caller is depicted.

[0211] Fig.108b A specific implementation of 3D convolution used by the recurrent module of a hybrid neural network to produce a representation of the current hidden state is shown.

[0212] Fig.109 One specific implementation of processing per-cycle input data for a single sequencing cycle in the series of t sequencing cycles to be base called by a cascade of convolutional layers of convolutional modules is shown.

[0213] Fig.110 Depicted is one specific implementation of mixing input data for each cycle of a single sequencing cycle with a corresponding convolutional representation of the single sequencing cycle produced by a cascade of convolutional layers of a convolutional module.

[0214] Fig.111 One specific implementation of arranging the flattened mixed representations of subsequent sequencing cycles into a stack is shown.

[0215] Fig.112a Shows that Fig.111 The stack of undergoes recursive application of 3D convolutions in the forward and backward directions and produces one specific implementation of base calls for each of the clusters at each sequencing cycle of the series of t sequencing cycles.

[0216] Fig.112b A specific implementation of processing a 3D input volume x(t) comprising a flattened set of hybrid representations by applying the input gate, activation gate, forget gate, and output gate of a long short-term memory (LSTM) network of 3D convolution is shown. The LSTM network is part of a recurrent module of a hybrid neural network.

[0217] Fig.113 One specific implementation of balancing trinucleotides (3-mers) in training data for training a neural network-based base caller is shown.

[0218] Fig.114 The base calling accuracy of the RTA base caller was compared to the base calling accuracy of the neural network based base caller.

[0219] Fig.115 The block-to-block generalization of the RTA base caller is compared to the block-to-block generalization of a neural network-based base caller on the same block.

[0220] Fig.116The block-to-block generalization of the RTA base caller is compared to the block-to-block generalization of a neural network-based base caller on the same block and on different blocks.

[0221] Fig.117 The block-to-block generalization of the RTA base caller is also compared with that of a neural network-based base caller on different blocks.

[0222] Fig.118 Shown is how different sizes of image patches fed as input to a neural network-based base caller affect base calling accuracy.

[0223] Fig.119 , Fig.120 , Fig.121 and Fig.122 The neural network-based base caller is shown to generalize slot-to-slot on training data from Acinetobacter baumannii and Escherichia coli.

[0224] Fig.123 Shows the above relative to Fig.119 , Fig.120 , Fig.121 and Fig.122 The error distribution of the described slot-to-slot generalization.

[0225] Fig.124 Will be Fig.123 Error distribution of the detected error sources are attributed to the low cluster intensity in the green channel.

[0226] Fig.125 The error distributions of the RTA base caller and the neural network-based base caller for two sequencing runs (Read 1 and Read 2) were compared.

[0227] Fig.126a Shown is the run-to-run generalization of the neural network-based base caller on four different instruments.

[0228] Fig.126b Shown is the run-to-run generalization of the neural network-based base caller in four different runs performed on the same instrument.

[0229] Fig.127 Shown are the genomic statistics of the training data used to train a neural network-based base caller.

[0230] Fig.128 The genomic context of the training data used to train a neural network-based base caller is shown.

[0231] Fig.129The base calling accuracy of the neural network based base caller in base called long reads (eg, 2×250) is shown.

[0232] Fig.130 A specific implementation of how a neural network-based base caller focuses on a central cluster pixel and its neighboring pixels on an image patch is shown.

[0233] Fig.131 Various hardware components and configurations for training and running a neural network-based base caller according to one implementation are shown. In other implementations, different hardware components and configurations are used.

[0234] Fig.132 Various sequencing tasks that can be performed using a neural network-based base caller are shown.

[0235] Fig.133 is a scatter plot visualized by t-distributed stochastic neighbor embedding (t-SNE) and depicts the base calling results of the neural network based base caller.

[0236] Fig.134 One specific implementation of selecting base call confidence probabilities produced by a neural network-based base caller for use in quality scoring is illustrated.

[0237] Fig.135 A specific implementation of neural network based quality scoring is shown.

[0238] Figures 136a to 136b Depicted is one specific implementation of the correspondence between quality scores and base call confidence predictions made by a neural network-based base caller.

[0239] Fig.137 One specific implementation of inferring quality scores based on base call confidence predictions made by a neural network-based base caller during inference is shown.

[0240] Fig.138 One specific implementation of training a neural network-based quality scorer to process input data derived from sequencing images and directly generate quality indications is shown.

[0241] Fig.139 One specific implementation is shown that directly generates a quality indication during inference as an output of a neural network-based quality scorer.

[0242] Fig.140 Depicted is one specific implementation of using a lossless transform to generate transformed data that can be fed as input to a neural network-based template generator, a neural network-based base caller, and a neural network-based quality scorer.

[0243] Fig.141 One specific implementation of integrating a neural network-based template generator with a neural network-based base caller using area-weighted factorization is illustrated.

[0244] Fig.142 Another specific implementation of integrating a neural network-based template generator with a neural network-based base caller using upsampling and background masking is illustrated.

[0245] Fig.143 An example of area-weighted factorization 14300 of the contribution from only a single cluster per pixel is depicted.

[0246] Fig.144 Depicted is an example of area-weighted factorization of contributions from multiple clusters per pixel.

[0247] Fig.145 Depicts an example of upsampling and background masking using interpolation.

[0248] Fig.146 Depicted is an example of upsampling and background masking using sub-pixel count weighting.

[0249] Fig.147A and Fig.147B A specific implementation of a sequencing system is depicted. The sequencing system includes a configurable processor.

[0250] Fig.147C is a simplified block diagram of a system for analyzing sensor data (such as base calling sensor output) from a sequencing system.

[0251] Fig.148A is a simplified diagram showing aspects of a base calling operation including the functionality of a runtime program executed by a host processor.

[0252] Fig.148B Is a configurable processor such as Fig.147C A simplified diagram depicting the configuration of a configurable processor.

[0253] Fig.149 It can be Fig.147A The sequencing system is a computer system used to implement the technology disclosed in this article.

[0254] Fig.150 Different implementations of data preprocessing are shown, which may include data normalization and data augmentation.

[0255] Fig.151 shows that when a neural network-based base caller is trained on bacterial data and tested on human data, Fig.150 The data normalization technique (DeepRTA(Normalization)) and data augmentation technique (DeepRTA(Enhancement)) of our method reduced the base calling error percentage, where the bacterial data and human data shared the same assays (e.g., both contained intron data).

[0256] Fig.152 It is shown that when a neural network-based base caller is trained on non-exonic data (e.g., intronic data) and tested on exonic data, Fig.151 The data normalization technology (DeepRTA(Normalization)) and data enhancement technology (DeepRTA(Enhancement)) reduce the percentage of base calling errors. DETAILED DESCRIPTION

[0257] The following discussion is presented to enable any person skilled in the art to make and use the disclosed technology, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the specific implementations shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0258] Introduction

[0259] Base calling from digital images is massively parallel and computationally intensive. This presents many technical challenges that we recognized before the introduction of new technologies.

[0260] As the base classification cycle progresses, the signal from the image set being evaluated becomes weaker and weaker, especially as the base chains become longer and longer. As the base classification proceeds along the length of the chain, the signal-to-noise ratio decreases, and therefore the reliability decreases. When the estimated reliability of the base classification changes, an updated reliability estimate is expected.

[0261] Digital images are captured from clusters of amplified sample strands. The sample is amplified by replicating the strands using a variety of physical structures and chemicals. During sequencing by synthesis, tags are chemically attached cyclically and stimulated to emit light. A digital sensor collects photons from the tags that are read out of the pixel to produce an image.

[0262] Interpreting digital images to classify bases requires resolving positional uncertainty, which is hampered by limited image resolution. At resolutions greater than those collected during base calling, it is apparent that the imaged clusters have irregular shapes and uncertain center positions. Because cluster positions are not mechanically regulated, cluster centers are not aligned with pixel centers. The pixel center can be an integer coordinate assigned to a pixel. In other embodiments, the pixel center can be the upper left corner of the pixel. In other embodiments, the pixel center can be the center of mass or centroid of the pixel. Amplification does not produce uniform cluster shapes. Therefore, the distribution of cluster signals in digital images is a statistical distribution rather than a regular pattern. We refer to this as positional uncertainty.

[0263] One of the signal classes may produce no detectable signal and be classified at a particular location based on a "dark" signal. Therefore, a template is necessary for classification during dark cycles. Generating a template uses multiple imaging cycles to resolve initial position uncertainty to avoid missing dark signals.

[0264] Tradeoffs in image sensor size, magnification, and stepped design result in relatively large pixel sizes that are too large to consider cluster centers coincident with sensor pixel centers. The present disclosure uses pixels in two senses. A physical sensor pixel is the area where an optical sensor reports detected photons. A logical pixel (referred to simply as a pixel) is data that corresponds to at least one physical pixel (i.e., data read from a sensor pixel). A pixel can be subdivided or "upsampled" into sub-pixels, such as 4×4 sub-pixels. To account for the likelihood that all photons hit one side of a physical pixel rather than the opposite side, values ​​can be assigned to the sub-pixels by interpolation, such as bilinear interpolation or area weighting. Interpolation or bilinear interpolation is also applied when pixels are reframed by applying an affine transformation to the data from the physical pixels.

[0265] Larger physical pixels are more sensitive to weak signals than smaller pixels. While digital sensors have improved over time, physical limitations on collector surface area are unavoidable. Given the design tradeoffs, conventional systems have been designed to collect and analyze image data from a 3×3 patch of the sensor, with the cluster center located somewhere in the center pixel of the patch.

[0266] The high resolution sensor captures only a portion of the imaging medium at a time. The sensor is stepped across the imaging medium to cover the entire field. Thousands of digital images can be collected during one processing cycle.

[0267] The sensor and lighting design are combined to distinguish at least four illumination response values ​​for classifying bases. If a traditional RGB camera with a Bayer color filter array is used, four sensor pixels will be combined into a single RGB value. This will reduce the effective sensor resolution to a quarter of the original. Alternatively, multiple images can be collected at a single location using different illumination wavelengths and / or different filters rotated into the position between the imaging medium and the sensor. The number of images required to distinguish the four base classifications varies between systems. Some systems use an image with four intensity levels for bases of different categories. Other systems use two images with different illumination wavelengths (e.g., red and green) and / or filters with a truth table to classify bases. The system can also use four images with different illumination wavelengths and / or filters tuned to specific base categories.

[0268] In practice, massively parallel processing of digital images is required to align and assemble relatively short chains (approximately 30 to 2000 base pairs) into longer sequences, which may be millions or even billions of bases in length. Redundant samples are desirable on the imaging medium so that a portion of the sequence can be covered by dozens of sample reads. Millions or at least hundreds of thousands of sample clusters are imaged from a single imaging medium. Massively parallel processing of so many clusters increases sequencing capacity while reducing costs.

[0269] Sequencing capacity is growing at a rate comparable to Moore’s Law. Although the first sequencing cost billions of dollars, in 2018 service providers such as Illumina TM The price of delivering sequencing results is in the hundreds of dollars. As sequencing becomes mainstream and unit prices drop, less computing power is available for classification, increasing the challenge of near real-time classification. With these technical challenges in mind, we turn to the disclosed technology.

[0270] The disclosed technology improves the processing during the generation of templates to resolve positional uncertainty and during base classification of clusters at resolved positions. Applying the disclosed technology, cheaper hardware can be used to reduce the cost of the machine. Near real-time analysis can become cost-effective, thereby reducing the lag between image collection and base classification.

[0271] The disclosed technology can use an upsampled image that is generated by interpolating sensor pixels into sub-pixels and then generating a template that resolves positional uncertainty. The resulting sub-pixels are submitted to a base caller for classification, which treats the sub-pixels as being located at cluster centers. Clusters are determined from groups of adjacent sub-pixels that repeatedly receive the same base classification. This aspect of the technology utilizes existing base calling technology to determine the shape of the clusters and super-locate the cluster centers with sub-pixel resolution.

[0272] Another aspect of the disclosed technology is to create a benchmark truth training data set that pairs images with confidently determined cluster centers and / or cluster shapes. Deep learning systems and other machine learning methods require a large number of training sets. Human-curated data is expensive to compile. The disclosed technology can be used to generate a large number of confidently classified training data sets using existing classifiers in non-standard operating modes without intervention or labor. In non-standard operating modes, such as in a CNN-based deep learning system, the training data associates the original image with cluster centers and / or cluster shapes that can be obtained from an existing classifier, and then the CNN-based deep learning system can directly process the image sequence. A training image can be rotated and reflected to produce other equally valid examples. The training examples can focus on areas with predetermined sizes within the entire image. The context evaluated during base calling determines the size of the exemplary training area, rather than determining the size of the image from the entire imaging medium or the size of the entire imaging medium.

[0273] The disclosed technology can generate different types of maps, which can be used as training data or templates for base classification, and cluster centers and / or cluster shapes are associated with digital images. First, sub-pixels can be classified as cluster centers, thereby locating cluster centers within physical sensor pixels. Second, cluster centers can be calculated as the mass center of cluster shapes. This position can be reported with a selected digital accuracy. Third, cluster centers with surrounding sub-pixels can be reported in an attenuation map with sub-pixel or pixel resolution. As the separation of the region from the cluster center increases, the attenuation map reduces the weight of the photons detected in the region, thereby weakening the signal from a farther position. Fourth, binary or ternary classification can be applied to sub-pixels or pixels in clusters of adjacent regions. In binary classification, a region is classified as belonging to a cluster center or is classified as background. In ternary classification, the third type is a region that contains the interior of a cluster but does not contain a cluster center. The sub-pixel classification for the cluster center position can replace the real-valued cluster center coordinates within a larger optical pixel.

[0274] Maps of alternative styles may be generated initially as ground truth data sets, or neural networks may be used to generate these maps through training. For example, clusters may be depicted as disjoint regions of adjacent sub-pixels with appropriate classification. If the centers have not yet been determined, the intensity map clusters from the neural network may be post-processed by a peak detector filter to calculate the cluster centers. Adjacent regions may be assigned to separate clusters using a so-called watershed analysis. When generated by a neural network inference engine, the maps may be used as templates for evaluating digital image sequences and classifying bases during the cyclic process of base calling.

[0275] When classifying bases in a sequence of digital images, the neural network processes multiple image channels in the current cycle as well as image channels in past and future cycles together. In a cluster, some of the chains may extend before or after the main process of synthesis, where the dephasing mark is called prephasing or phasing. Given the low rates of prephasing and postphasing observed empirically, almost all of the noise in the signal generated by prephasing and postphasing can be processed by the neural network, which processes digital images in the current, past, and future cycles (in only three cycles).

[0276] In the digital image pipeline in the current cycle, careful registration to align the images within the cycle contributes significantly to accurate base classification. Combinations of wavelength and misaligned illumination sources, as well as other sources of error, produce small, correctable differences in the measured cluster center locations. A general affine transformation with translation, rotation, and scaling capabilities can be used to precisely align the cluster centers across the image map tile. Affine transformations can be used to reconstruct the image data and account for shifts in cluster centers.

[0277] Reconstructing image data means interpolating the image data, usually by applying an affine transformation. The reconstruction may place the center of the cluster of interest in the middle of the center pixel of the pixel patch. Alternatively, the reconstruction may align the image with a template to overcome jitter and other differences during image collection. Reconstruction involves adjusting the intensity values ​​of all pixels in the pixel patch. Bilinear and bicubic interpolation and weighted area adjustment are alternative strategies.

[0278] In some implementations, cluster center coordinates can be fed to the neural network as an additional image channel.

[0279] Distance signals can also contribute to base classification. Several types of distance signals reflect the separation of regions from cluster centers. The strongest light signal is considered to coincide with the cluster center. Light signals along the periphery of the cluster sometimes include stray signals from nearby clusters. It has been observed that when the contribution of the signal component decays according to its separation from the cluster center, the classification is more accurate. The distance signals that work include a single cluster distance channel, multiple cluster distance channels, and multiple cluster shape-based distance channels. A single cluster distance channel is applied to a patch with a cluster center in a central pixel. Therefore, the distance of all regions in the patch is the distance from the cluster center in the central pixel. Pixels that do not belong to the same cluster as the central pixel can be marked as background, rather than given the calculated distance. Multiple cluster distance channels pre-calculate the distance from each region to the nearest cluster center. This has the possibility of connecting the region to the wrong cluster center, but this possibility is low. Multiple cluster shape-based distance channels associate regions (sub-pixels or pixels) to the pixel center that produces the same base classification through adjacent regions. By calculation, the possibility that the measured distance is the distance to the wrong pixel is avoided. Multiple Clusters and Multiple Cluster Shape-Based Distance Signal Methods have the advantage of being subject to pre-computation and working with multiple clusters in an image.

[0280] Neural networks can use shape information to separate signal from noise to improve the signal-to-noise ratio. In the above discussion, several methods for classifying regions and providing distance channel information were identified. In any of these methods, regions can be marked as background and not part of a cluster to define cluster edges. Neural networks can be trained to exploit the resulting information about irregular cluster shapes. Distance information and background classification can be used in combination or separately. As cluster density increases, it will become increasingly important to separate signals from adjacent clusters.

[0281] One direction to increase the scale of parallel processing is to increase the cluster density on the imaging medium. Increasing density has the disadvantage of increasing background noise when reading clusters with adjacent neighbors. For example, using shape data rather than arbitrary patches (e.g., patches of 3×3 pixels) helps keep the signals separated as the cluster density increases.

[0282] In one aspect of the disclosed technology, base classification scores can also be used to predict quality. The disclosed technology includes correlating classification scores directly or through a prediction model with traditional Sanger or Phred quality Q scores. Scores such as Q20, Q30, or Q40 are calculated by Q=-10log 10P is logarithmically related to the probability of base classification error. The correlation of class score and Q score can be performed using multi-output neural network or multivariate regression analysis. During base classification, the advantage of real-time calculation of quality score is that defective sequencing operation can be terminated early. Applicants have found that it is possible to occasionally (rarely) decide to terminate the operation at one-eighth to one-quarter of the way of the entire analysis sequence. The decision to terminate can be made after 50 cycles or after 25 to 75 cycles. In the sequencing process that would originally run 300 to 1000 cycles, early termination leads to significant resource savings.

[0283] Specialized convolutional neural network (CNN) architectures can be used to classify bases within multiple cycles. One specialization involves isolation between digital image channels during processing of the initial layer. The convolution filter stack can be constructed to isolate processing between cycles, thereby preventing crosstalk between digital image sets from different cycles. The motivation for isolating processing between cycles is that images acquired at different cycles have residual registration errors and are therefore misaligned with respect to each other and have random translation offsets. This occurs because of the limited motion accuracy of the motion stage of the sensor and because images acquired in different frequency channels have different optical paths and wavelengths.

[0284] The motivation for using image sets from subsequent cycles is that the contributions to pre-phasing and post-phasing the signal in a particular cycle are second-order contributions. Therefore, it may be helpful for the convolutional neural network to structurally isolate the lower-level convolutions of the digital image sets between image collection cycles.

[0285] The convolutional neural network structure can also be specifically used to process information about clustering. The template of cluster centers and / or shapes provides additional information for the convolutional neural network to combine with the digital image data. The cluster center classification and distance data can be repeatedly applied across cycles.

[0286] The convolutional neural network can be constructed to classify multiple clusters in an image field. When classifying multiple clusters, the distance channel of a pixel or sub-pixel can more compactly contain distance information relative to the nearest cluster center or adjacent cluster center to which the pixel or sub-pixel belongs. Alternatively, a large distance vector can be provided for each pixel or sub-pixel, or at least for each pixel or sub-pixel containing a cluster center, which gives complete distance information from the cluster center to all other pixels that are context for a given pixel.

[0287] Some combinations of template generation and base calling can use area-weighted variations instead of distance channels. We now discuss how to use the output of the template generator directly instead of the distance channel.

[0288] We discuss three considerations that affect the direct application of a template image to modify pixel values: whether the image set is processed in the pixel domain or the sub-pixel domain; in either domain, how the area weights are computed; and in the sub-pixel domain, the application of the template image as a mask to modify the interpolated intensity values.

[0289] Performing base classification in the pixel domain has the advantage of not requiring the increase in computation due to upsampling (e.g., 16x). In the pixel domain, even the top-level convolutions can have sufficient cluster density to justify performing computations that would otherwise be lost, rather than adding logic to cancel computations that are not needed. We begin with an example using template image data directly in the pixel domain without a range channel.

[0290] In some implementations, the classification is focused on a particular cluster. In these cases, pixels on the periphery of the cluster may have different modified intensity values, depending on which adjacent cluster is the focus of the classification. The template image in the sub-pixel domain may indicate that overlapping pixels contribute to the intensity values ​​of two different clusters. When two or more adjacent or contiguous clusters overlap a pixel, we refer to the optical pixel as an "overlapping pixel"; both of these two or more adjacent or contiguous clusters contribute to the intensity reading from the optical pixel. Watershed analysis (named for dividing rainwater flows into different waters at ridges) can be applied to separate even adjacent clusters. When data for cluster-by-cluster classification is received, the template image can be used to modify the intensity data of overlapping pixels along the periphery of the cluster. The overlapping pixels may have different modified intensities, depending on which cluster is the focus of the classification.

[0291] The modified intensity of a pixel may be reduced based on the subpixel contribution in the overlapping pixel to the primary cluster (i.e., the cluster to which the pixel belongs or the cluster whose intensity emission is primarily depicted by the pixel) rather than to the distant cluster (i.e., the non-primary cluster whose intensity emission is depicted by the pixel). Assume that 5 subpixels are part of the primary cluster and 2 subpixels are part of the distant cluster. Then, 7 subpixels contribute to the intensity of either the primary cluster or the distant cluster. During focus on the primary cluster, in one implementation, the intensity of the overlapping pixel is reduced by 7 / 16 because 7 of the 16 subpixels contribute to the intensity of either the primary cluster or the distant cluster. In another implementation, the intensity is reduced by 5 / 16 based on the area of ​​the subpixels contributing to the primary cluster divided by the total number of subpixels. In a third implementation, the intensity is reduced by 5 / 7 based on the area of ​​the subpixels contributing to the primary cluster divided by the total area of ​​the contributing subpixels. When the focus shifts to the distant cluster, the latter two calculations change, resulting in a fraction with a numerator of "2".

[0292] Of course, further reduction in intensity can be applied if the distance channel is considered together with the sub-pixel map of the cluster shape.

[0293] Once the pixel intensities of the clusters that are the focus of classification have been modified using the template image, the modified pixel values ​​are convolved through the layers of the neural network-based classifier to produce modified images. These modified images are used to classify bases in subsequent sequencing cycles.

[0294] Alternatively, classification in the pixel domain may be performed simultaneously for all pixels or all clusters in an image block. In this case, only one modification of the pixel value may be applied to ensure reusability of intermediate calculations. Any of the scores given above may be used to modify the pixel intensity, depending on whether a smaller or larger intensity attenuation is desired.

[0295] Once the pixel intensities of the image patch have been modified using the template image, the pixels and surrounding context can be convolved through the layers of a neural network-based classifier to produce a modified image. Performing convolution on the image patch allows for the reuse of intermediate calculations between pixels that have shared context. These modified images are used to classify bases in subsequent sequencing cycles.

[0296] This description can be used in parallel to apply area weights in the sub-pixel domain. In parallel, it is meant that the weights can be calculated for individual sub-pixels. The weights can be, but need not be, the same for different sub-pixel portions of an optical pixel. Repeating the above scenario with a main cluster and a distant cluster of 5 sub-pixels and 2 sub-pixels in overlapping pixels, respectively, the intensity distribution of the sub-pixels belonging to the main cluster can be 7 / 16, 5 / 16, or 5 / 7 of the pixel intensity. Similarly, if the sub-pixel map of the range channel and the cluster shape is considered together, a further reduction in intensity can be applied.

[0297] Once the pixel intensities of the image patch have been modified using the template image, the sub-pixels and surrounding context can be convolved through the layers of a neural network-based classifier to produce a modified image. Performing convolution on the image patch allows for the reuse of intermediate computations between sub-pixels that have shared context. These modified images are used to classify bases in subsequent sequencing cycles.

[0298] Another alternative is to apply a template image as a binary mask in the sub-pixel domain to the image data interpolated into the sub-pixel domain. The template image can be arranged to require background pixels between clusters or to allow sub-pixels from different clusters to be adjacent. The template image can be used as a mask. If an interpolated pixel is classified as background in the template image, the mask determines whether the interpolated pixel retains the value assigned by the interpolation method or receives a background value (e.g., zero).

[0299] Similarly, once the pixel intensities of an image patch have been masked using a template image, the sub-pixels and surrounding context can be convolved through layers of a neural network-based classifier to produce a modified image. Performing convolutions on image patches allows for the reuse of intermediate computations between sub-pixels that have shared context. These modified images are used to classify bases in subsequent sequencing cycles.

[0300] Features of the disclosed techniques can be combined to classify any number of clusters within a shared context, thereby reusing intermediate computations. In one implementation, at optical pixel resolution, approximately ten percent of the pixels hold cluster centers to be classified. In conventional systems, 3×3 optical pixel groups are used for analysis as potential signal contributors to cluster centers, given the observation of irregularly shaped clusters. Even with a 3×3 filter away from the top convolutional layer, the cluster density may roll up into pixels at the cluster center based on light signals from substantially more than half of the optical pixels. Only at supersampled resolution does the cluster center density at the top convolutional layer drop to less than one percent.

[0301] In some implementations, the shared context is large. For example, a 15×15 optical pixel context can contribute to accurate base classification. The equivalent 4x upsampled context would be 60×60 sub-pixels. This level of context helps the neural network recognize the effects of uneven lighting and background during imaging.

[0302] The disclosed technique uses small filters at lower convolutional layers to combine cluster boundaries in the template input with boundaries detected in the digital image input. The cluster boundaries help the neural network separate the signal from the background conditions and normalize the image processing relative to the background.

[0303] The disclosed technique essentially reuses intermediate computations. Assume that 20 to 25 cluster centers appear within a context region of 15×15 optical pixels. Then, the first layer of convolution is reused 20 to 25 times in a block-by-block convolution roll-up. The reuse factor is reduced layer by layer until the penultimate layer, where the reuse factor at the optical resolution drops to less than 1x for the first time.

[0304] Block-by-block roll-up training and inference from multiple convolutional layers applies subsequent roll-ups to blocks of pixels or sub-pixel blocks. There is an overlap region around the perimeter of the block where data used during the roll-up of a first data block overlaps with and can be reused for a second rolled-up block. Within the block, in the central region surrounded by the overlap region are pixel values ​​and intermediate calculations that can be rolled up and reused. Using the overlap region, the convolution results of gradually reducing the size of the context field (for example, from 15×15 to 13×13 by applying a 3×3 filter) can be written to the same storage block holding the convolved values, thereby saving memory without compromising the reuse of basic calculations within the block. For larger blocks, sharing intermediate calculations in the overlap region requires fewer resources. For smaller blocks, it may be possible to calculate multiple blocks synchronously to share intermediate calculations in the overlap region.

[0305] Larger filters and dilation reduce the number of convolutional layers after the lower convolutional layers react to cluster boundaries in the template and / or digital image data, which speeds up computation without hurting classification.

[0306] The input channels for the template data may be selected so that the template structure is consistent with the classification of multiple cluster centers in the digital image field. The above two alternatives do not meet this consistency criterion: reconstruction and distance mapping in the entire context. Reconstruction places the center of only one cluster at the center of the optical pixel. For classification of multiple clusters, it is better to provide a center offset for pixels classified as holding cluster centers.

[0307] Unless each pixel has its own distance map across the entire context, it is difficult to perform distance mapping across the entire context region (if provided). Simpler distance maps provide useful consistency for classifying multiple clusters from a digital image input patch.

[0308] The neural network can learn from the classification in the template of the pixel or sub-pixel at the cluster boundary, so the distance channel can be replaced by a template that provides binary or ternary classification and is accompanied by a cluster center offset channel. When used, the distance map can give the distance of the pixel from the cluster center to which the pixel (or sub-pixel) belongs. Or the distance map can give the distance from the nearest cluster center. The distance map can encode the binary classification with the label value assigned to the background pixel, or it can be a separate channel for pixel classification. Combined with the cluster center offset, the distance map can encode the ternary classification. In some specific implementations, especially in the specific implementation where the pixel classification is encoded with one or two bits, it may be desirable to use separate channels for pixel classification and distance at least during development.

[0309] The disclosed techniques may include reducing computations to save some computational resources in the upper layers. A cluster center offset channel or a ternary classification map may be used to identify the centers of pixel convolutions that do not contribute to the final classification of the pixel center. In many hardware / software implementations, performing a lookup during inference and a skip convolution roll-up may be more efficient than performing even nine multiplications and eight additions to apply a 3×3 filter. In custom hardware that delivers computations to be performed in parallel, each pixel may be classified within the pipeline. The cluster center map may then be used after the final convolution to obtain results for only the pixels that coincide with the cluster center, since only these pixels require final classification. Similarly, in the optical pixel domain, at the currently observed cluster density, roll-up computations will be obtained for approximately ten percent of the pixels. In a 4x upsampled domain, on some hardware, more layers may benefit from skipped convolutions, since less than one percent of sub-pixel classifications will be obtained in the top layer.

[0310] Template generation based on neural network

[0311] The first step in template generation is to determine cluster metadata. Cluster metadata identifies the spatial distribution of clusters and identifies the center, shape, size, background and / or boundaries of these clusters.

[0312] Determining cluster metadata

[0313] Figure 1 One specific implementation of a processing pipeline that uses sub-pixel base calling to determine cluster metadata is shown.

[0314] Figure 2 A specific implementation of a flow cell containing clusters in its blocks is depicted. The flow cell is divided into channels. These channels are further divided into non-overlapping areas called "blocks". During sequencing, the clusters on the blocks and their surrounding background are imaged.

[0315] Figure 3 An exemplary Illumina GA-IIx with eight channels is shown. TM Circulation pool. Figure 3 A zoomed-in view of one block and its cluster and its surrounding background is also shown.

[0316] Figure 4Depicted is an image set of sequencing images for four-channel chemistry, that is, the image set has four sequencing images captured using four different wavelength bands (images / imaging channels) in the pixel domain. Each image in the image set covers a block of a circulation pool, and depicts the intensity emission captured for a specific imaging channel in a specific sequencing cycle of a plurality of sequencing cycles of a sequencing run performed on the circulation pool by a cluster on the block and its surrounding background. In one specific implementation, each imaging channel corresponds to a filter wavelength band in a plurality of filter wavelength bands. In another specific implementation, each imaging channel corresponds to an imaging event in a plurality of imaging events in a sequencing cycle. In yet another specific implementation, each imaging channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter. The intensity emission of the cluster includes signals detected from the analyte, which can be used to classify the bases associated with the analyte. For example, the intensity emission can be a signal indicating the photons emitted by a label, which are chemically attached to the analyte during the cycle of the stimulation label and can be detected by one or more digital sensors, as described above.

[0317] Figure 5 It is a specific implementation of dividing the sequencing image into sub-pixels (or sub-pixel regions). In the illustrated specific implementation, a quarter (0.25) sub-pixel is used, which results in each pixel in the sequencing image being divided into sixteen sub-pixels. In view of the resolution of the sequencing image shown is 20x20 pixels (i.e., 400 pixels), the division produces 6400 sub-pixels. Each sub-pixel in the sub-pixel is regarded as the regional center of the sub-pixel base call by the base caller. In some specific implementations, the base caller does not use a neural network-based process. In other specific implementations, the base caller is a neural network-based base caller.

[0318] For a given sequencing cycle and a particular sub-pixel, the base caller is configured with logic to generate base calls for a given sequencing cycle and a particular sub-pixel by performing an image processing step and extracting the intensity data of the sub-pixel from the corresponding image set of the sequencing cycle. This is done for each sub-pixel in the sub-pixel and for each sequencing cycle in multiple sequencing cycles. Experiments have also been conducted with a quarter sub-pixel division of a block image of a 1800x1800 pixel resolution of an Illumina MiSeq sequencer. Sub-pixel base calls were performed for fifty sequencing cycles and ten blocks of one channel.

[0319] Figure 6 The initial center coordinates of the clusters identified by the base caller during sub-pixel base calling are shown. Figure 6 Also shown is the "origin sub-pixel" or "center sub-pixel" containing the initial center coordinates.

[0320] Figure 7 An example of merging sub-pixel base calls generated over multiple sequencing cycles to generate a so-called "cluster map" containing cluster metadata is depicted. In the illustrated implementation, a breadth-first search approach is used to merge sub-pixel base calls.

[0321] Figure 8a An example of a cluster map generated by merging sub-pixel base calls is shown. Figure 8b An example of sub-pixel base calling is depicted. Figure 8b Also shown is one specific implementation of analyzing a sub-pixel base call sequence produced by sub-pixel base calling to generate a cluster map.

[0322] Sequencing images

[0323] Cluster metadata determination involves analyzing image data generated by a sequencing instrument 102 (e.g., Illumina's iSeq, HiSeqX, HiSeq3000, HiSeq4000, HiSeq2500, NovaSeq6000, NextSeq, NextSeqDx, MiSeq, and MiSeqDx). According to one specific implementation, the following discussion outlines how image data is generated and what it depicts.

[0324] Base calling is the process in which the raw signal of the sequencing instrument 102 (i.e., the intensity data extracted from the image) is decoded into a DNA sequence and a quality score. In one specific implementation, the Illumina platform employs cyclic reversible termination (CRT) chemistry for base calling. The process relies on growing a nascent DNA strand complementary to a template DNA strand with modified nucleotides while tracking the emission signal of each newly added nucleotide. The modified nucleotides have a 3' removable block that anchors the fluorophore signal of the nucleotide type.

[0325] Sequencing is performed in repeated cycles, each cycle comprising three steps: (a) extending the nascent chain by adding modified nucleotides; (b) exciting the fluorophore using one or more lasers of the optical system 104 and imaging through different filters of the optical system 104, thereby generating a sequencing image 108; and (c) cleaving the fluorophore and removing the 3' block in preparation for the next sequencing cycle. The binding and imaging cycles are repeated until a specified number of sequencing cycles are reached, thereby defining the read length of all clusters. Using this method, each cycle interrogates a new position along the template chain.

[0326] The huge power of Illumina platform comes from its ability to perform and sense millions or even billions of clusters that experience CRT reactions at the same time. The sequencing process occurs in a circulation cell 202, which is a small glass slide that keeps the input DNA fragments during the sequencing process. The circulation cell 202 is connected to a high-throughput optical system 104, which includes microscope imaging, excitation lasers and fluorescence filters. The circulation cell 202 includes a plurality of chambers referred to as grooves 204. The grooves 204 are physically separated from each other, and different labeled sequencing libraries may be included, which may be distinguished when there is no cross contamination of samples. An imaging device 106 (e.g., a solid-state imager, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor) takes snapshots at a plurality of positions along the grooves 204, in a series of non-overlapping regions (referred to as blocks 206).

[0327] For example, each channel in the Illumina Genome Analyzer II has one hundred blocks, and each channel in the Illumina HiSeq2000 has sixty-eight blocks. Block 206 accommodates hundreds of thousands to millions of clusters. An image generated from a block with clusters shown as bright spots is shown at 208. Cluster 302 includes approximately one thousand identical copies of the template molecule, but the size and shape of the clusters are different. Before the sequencing run, clusters are generated from the template molecules by bridge amplification of the input library. The purpose of amplification and cluster growth is to increase the intensity of the emission signal because the imaging device 106 cannot reliably sense a single fluorophore. However, the physical distance of the DNA fragments within the cluster 302 is small, so the imaging device 106 perceives the cluster of fragments as a single point 302.

[0328] The output of the sequencing run is sequencing images 108, each of which depicts the intensity emission of clusters on a block in the pixel domain for a particular combination of lane, block, sequencing cycle, and fluorophore (208A, 208C, 208T, 208G).

[0329] In one specific implementation, the biosensor includes an array of photosensors. The photosensors are configured to sense information from corresponding pixel areas (e.g., reaction sites / wells / nanopores) on the detection surface of the biosensor. An analyte disposed in a pixel area is said to be associated with the pixel area, i.e., an associated analyte. In a sequencing cycle, the photosensors corresponding to the pixel areas are configured to detect / capture / sense emission / photons from associated analytes, and in response, a pixel signal is generated for each imaging channel. In one specific implementation, each imaging channel corresponds to a filter wavelength band in a plurality of filter wavelength bands. In another specific implementation, each imaging channel corresponds to an imaging event in a plurality of imaging events in a sequencing cycle. In yet another specific implementation, each imaging channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter.

[0330] The pixel signals from the photosensors are transmitted to a signal processor coupled to the biosensor (e.g., via a communication port). For each sequencing cycle and each imaging channel, the signal processor generates an image whose pixels respectively depict / contain / indicate / represent / characterize the pixel signals obtained from the corresponding photosensors. Thus, the pixels in the image correspond to: (i) the photosensor of the biosensor, which generates the pixel signal depicted by the pixel, (ii) the associated analyte, whose emissions are detected by the corresponding photosensor and converted into the pixel signal, and (iii) the pixel area on the detection surface of the biosensor that holds the associated analyte.

[0331] For example, consider a sequencing run using two different imaging channels, namely a red channel and a green channel. Then, in each sequencing cycle, the signal processor generates a red image and a green image. Thus, for a series of k sequencing cycles of a sequencing run, a sequence with k pairs of red and green images is generated as output.

[0332] The pixels in the red and green images (i.e., different imaging channels) correspond one-to-one within a sequencing cycle. This means that corresponding pixels in a pair of red and green images depict intensity data for the same associated analyte, albeit in different imaging channels. Similarly, the pixels on a pair of red and green images correspond one-to-one between sequencing cycles. This means that corresponding pixels in different pairs of red and green images depict intensity data for the same associated analyte, albeit for different acquisition events / time steps (sequencing cycles) of a sequencing run.

[0333] Corresponding pixels in the red and green images (i.e., different imaging channels) may be considered as pixels of an "image for each cycle" expressing intensity data in a first red channel and a second green channel. An image for each cycle whose pixels depict the pixel signals of a subset of a pixel region (i.e., an area (tile) of the detection surface of the biosensor) is referred to as a "tile image for each cycle". Patches extracted from the tile image for each cycle are referred to as "image patches for each cycle". In one specific implementation, patch extraction is performed by an input preparer.

[0334] The image data includes a sequence of image patches for each cycle generated for a series of k sequencing cycles of a sequencing run. The pixels in the image patch for each cycle contain intensity data for an associated analyte, and intensity data for one or more imaging channels (e.g., a red channel and a green channel) are obtained by a corresponding light sensor, which is configured to detect emission from the associated analyte. In one specific implementation, when base calling is to be performed on a single target cluster, the image patch for each cycle is centered on a center pixel containing intensity data for the target associated analyte, and non-center pixels in the image patch for each cycle contain intensity data for associated analytes adjacent to the target associated analyte. In one specific implementation, the image data is prepared by an input preparer.

[0335] Sub-pixel base calling

[0336] The disclosed technology accesses a series of image sets generated during a sequencing run. The image set includes a sequencing image 108. Each image set in the series of image sets is captured during a corresponding sequencing cycle of the sequencing run. Each image (or sequencing image) in the series of image sets captures a cluster on a block of a flow cell and its surrounding background.

[0337] In one implementation, the sequencing run utilizes a four-channel chemistry, and each image set has four images. In another implementation, the sequencing run utilizes a two-channel chemistry, and each image set has two images. In yet another implementation, the sequencing run utilizes a single-channel chemistry, and each image set has two images. In other implementations, each image set has only one image.

[0338] The sequencing image 108 in the pixel domain is first converted into the sub-pixel domain by the sub-pixel addresser 110 to generate the sequencing image 112 in the sub-pixel domain. In one embodiment, each pixel in the sequencing image 108 is divided into sixteen sub-pixels 502. Therefore, in one embodiment, the sub-pixel 502 is a quarter sub-pixel. In another embodiment, the sub-pixel 502 is a half sub-pixel. Therefore, each sequencing image 112 in the sub-pixel domain has a plurality of sub-pixels 502.

[0339] The sub-pixels are then fed individually as input to a base caller 114 to obtain a base call from the base caller 114 that classifies each of the sub-pixels as one of the four bases (A, C, T, and G). This produces a sequence of base calls 116 for each of the sub-pixels in multiple sequencing cycles of a sequencing run. In one specific implementation, the sub-pixels 502 are identified by the base caller 114 based on their integer coordinates or non-integer coordinates. By tracking the emission signal from the sub-pixels 502 across the set of images generated during multiple sequencing cycles, the base caller 114 recovers the underlying DNA sequence for each sub-pixel. An example of this is shown in FIG. Figure 8b Shown in.

[0340] In other implementations, the disclosed techniques obtain base calls from base caller 114 that classify each of the sub-pixels as one of five bases (A, C, T, G, and N). In such implementations, an N base call indicates an undecided base call, which is typically caused by a low extracted intensity level.

[0341] Some examples of base callers 114 include non-neural network based Illumina products such as RTA (Real Time Analysis), the Firecrest program of the Genome Analyzer analysis pipeline, the IPAR (Integrated Primary Analysis and Reporting) machine, and the OLB (Offline Base Caller). For example, the base caller 114 generates a base call sequence by interpolating the intensity of sub-pixels, and the intensity extraction methods include: nearest neighbor intensity extraction, Gaussian-based intensity extraction, intensity extraction based on the average value of a 2×2 sub-pixel area, intensity extraction based on the brightest point in a 2×2 sub-pixel area, intensity extraction based on the average value of a 3×3 sub-pixel area, bilinear intensity extraction, bicubic intensity extraction, and / or intensity extraction based on weighted area coverage. These techniques are described in detail in the Appendix entitled "Intensity Extraction Methods".

[0342] In other implementations, the base caller 114 may be a neural network based base caller, such as the neural network based base caller 1514 disclosed herein.

[0343] The sub-pixel-by-sub-pixel base call sequence 116 is then fed as input to a searcher 118. The searcher 118 searches for substantially matching base call sequences of adjacent sub-pixels. Base call sequences of adjacent sub-pixels are "substantially matched" when a predetermined portion of the base calls match in positional order (e.g., there are ≥41 matches in 45 cycles, ≤4 mismatches in 45 cycles, ≤4 mismatches in 50 cycles, or ≤2 mismatches in 34 cycles).

[0344] The searcher 118 then generates a cluster map 802 that identifies clusters as disjoint regions of adjacent sub-pixels that share substantially matching base call sequences, such as 804a-d. "Disjoint" and "non-overlapping" are used interchangeably herein. The search involves base calling sub-pixels that comprise portions of the clusters to allow the called sub-pixels to be linked to adjacent sub-pixels with which they share substantially matching base call sequences. In some implementations, the searcher 118 requires that at least some of the disjoint regions have a predetermined minimum number of sub-pixels (e.g., more than 4, 6, or 10 sub-pixels) to be processed as clusters.

[0345] In some implementations, the base caller 114 also identifies the initial center coordinates of the cluster. The sub-pixel containing the initial center coordinates is referred to as the origin sub-pixel. Figure 6 604a-c) and the corresponding origin sub-pixels (606a-c) identified by the base caller 114 are shown in FIG. However, the origin sub-pixel (the initial center coordinate of the cluster) need not be identified, as described below. In some implementations, the searcher 118 uses a breadth-first search to identify a substantially matching base call sequence of sub-pixels by starting with the origin sub-pixels 606a-c and continuing with consecutive adjacent non-origin sub-pixels 702a-c. This is also optional, as described below.

[0346] Cluster Map

[0347] Figure 8a An example of a cluster map 802 generated by merging sub-pixel base calls is shown. The cluster map identifies multiple non-intersecting regions (in Figure 8a Each disjoint region includes a set of non-overlapping adjacent sub-pixels representing a corresponding cluster on the tile (via sub-pixel base calling from its sequencing image and generation of a cluster map for it). The region between the disjoint regions represents the background on the tile. The sub-pixels in the background region are referred to as "background sub-pixels". The sub-pixels in the disjoint regions are referred to as "cluster sub-pixels" or "cluster interior sub-pixels". In this discussion, the origin sub-pixels are those sub-pixels where the initial center cluster coordinates determined by RTA or another base caller are located.

[0348] The origin sub-pixel contains the initial center cluster coordinates. This means that the area covered by the origin sub-pixel includes coordinate positions that coincide with the initial center cluster coordinate positions. Since the cluster map 802 is an image of logical sub-pixels, the origin sub-pixels are some sub-pixels in the cluster map.

[0349] The search to identify clusters having substantially matching base call sequences for sub-pixels need not begin with identifying the origin sub-pixel (the initial center coordinate of the cluster) because the search can be performed for all sub-pixels and can begin at any sub-pixel (e.g., the 0,0 sub-pixel or any random sub-pixel). Thus, since each sub-pixel is evaluated to determine whether it shares a substantially matching base call sequence with another adjacent sub-pixel, the search is not dependent on the origin sub-pixel; the search can begin at any sub-pixel.

[0350] Regardless of whether the origin sub-pixel is used, certain clusters are identified that do not contain the origin sub-pixel (the initial center coordinate of the cluster) predicted by base caller 114. Some examples of clusters identified by merging sub-pixel base calling and that do not contain the origin sub-pixel are Figure 8a 812a, 812b, 812c, 812d, and 812e in FIG. Thus, the disclosed technique identifies additional or additional clusters whose centers have not yet been identified by base caller 114. Thus, using base caller 114 to identify the origin sub-pixel (initial center coordinates of the cluster) is optional and is not required for searching adjacent sub-pixels for substantially matching base call sequences.

[0351] In one specific implementation, the origin sub-pixel (initial center coordinate of the cluster) identified by the base caller 114 is first used to identify a first set of clusters (by identifying substantially matching base call sequences of adjacent sub-pixels). Then, sub-pixels that are not part of the first set of clusters are used to identify a second set of clusters (by identifying substantially matching base call sequences of adjacent sub-pixels). This allows the disclosed technique to identify additional or additional clusters whose centers have not yet been identified by the base caller 114. Finally, sub-pixels that are not part of the first set of clusters and the second set of clusters are identified as background sub-pixels.

[0352] Figure 8b An example of sub-pixel base calling is depicted. Figure 8b In , each sequencing cycle has an image set with four different images (ie, A, C, T, G images) captured using four different wavelength bands (images / imaging channels) and four different fluorescent dyes (one for each base).

[0353] In this example, the pixels in the image are divided into sixteen sub-pixels. The sub-pixels are then independently base called by base caller 114 in each sequencing cycle. In order to base call a given sub-pixel in a particular sequencing cycle, base caller 114 uses the intensity of a given sub-pixel in each of the A, C, T, G images. For example, the intensity in the image area covered by sub-pixel 1 in each of the A, C, T, G images of cycle 1 is used to base call sub-pixel 1 in cycle 1. For sub-pixel 1, these image areas include the upper left sixteenth of the area of ​​the corresponding upper left pixel in each of the A, C, T, G images of cycle 1. Similarly, the intensity in the image area covered by sub-pixel m in each of the A, C, T, G images of cycle n is used to base call sub-pixel m in cycle n. For sub-pixel m, these image areas include the lower right sixteenth of the area of ​​the corresponding lower right pixel in each of the A, C, T, G images of cycle 1.

[0354] This process generates a subpixel-by-subpixel base call sequence 116 across multiple sequencing cycles. The searcher 118 then evaluates pairs of adjacent subpixels to determine whether they have substantially matching base call sequences. If so, the pair of subpixels is stored in the cluster map 802 as being part of the same cluster in the disjoint region. If not, the pair of subpixels is stored in the cluster map 802 as not being part of the same disjoint region. Thus, the cluster map 802 identifies groups of adjacent subpixels for which the base calls for the subpixels are substantially matched across multiple cycles. The cluster map 802 thus uses information from multiple cycles to provide multiple clusters with high confidence, wherein each of the multiple clusters provides sequence data for a single DNA strand.

[0355] The cluster metadata generator 122 then processes the cluster map 802 to determine cluster metadata, including determining the spatial distribution of clusters based on disjoint regions, and determining the center (810a), shape, size, background, and / or borders ( Fig. 9 ).

[0356] In some implementations, the cluster metadata generator 122 identifies those sub-pixels in the cluster map 802 that do not belong to any of the disjoint regions and therefore do not contribute to any cluster as background. Such sub-pixels are referred to as background sub-pixels 806a-c.

[0357] In some implementations, the cluster map 802 identifies cluster boundary portions 808a-c between two adjacent sub-pixels whose base call sequences do not substantially match.

[0358] The cluster map is stored in a memory (e.g., cluster map data storage 120) to serve as a ground truth for training classifiers such as neural network-based template generator 1512 and neural network-based base caller 1514. Cluster metadata may also be stored in a memory (e.g., cluster metadata storage 124).

[0359] Fig. 9 Another example of a cluster map identifying cluster metadata including a spatial distribution of clusters, as well as cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries is shown.

[0360] Center of mass (COM)

[0361] Fig.10 It shows how to calculate the center of mass (COM) of disjoint regions in a cluster map. The COM can be used as a "modified" or "improved" center of the corresponding cluster in downstream processing.

[0362] In some implementations, the centroid generator 1004 determines the hyperlocalized center coordinates of the clusters cluster by cluster by calculating the centroids of the disjoint regions of the cluster map as the average of the coordinates of the corresponding adjacent sub-pixels that form the disjoint regions 1006. The centroid generator then stores the hyperlocalized center coordinates of the clusters cluster by cluster in a memory to be used as ground truth for training the classifier.

[0363] In some implementations, the sub-pixel classifier identifies the centroid sub-pixel 1008 in the disjoint regions 804a-d of the cluster map 802 on a cluster-by-cluster basis at the hyper-located center coordinates 1006 of the cluster.

[0364] In other specific implementations, the cluster map is upsampled using an interpolation method. The upsampled cluster map is stored in a memory to be used as a ground truth for training a classifier.

[0365] Attenuation Factor and Attenuation Map

[0366] Fig.11A specific implementation of calculating a weighted attenuation factor for a sub-pixel based on the Euclidean distance of the sub-pixel from the centroid (COM) of the disjoint region to which the sub-pixel belongs is depicted. In the illustrated implementation, the weighted attenuation factor provides the highest value to the sub-pixel containing the COM, and the value obtained for the sub-pixel farther from the COM is smaller. The weighted attenuation factor is used to derive a reference truth attenuation map 1204 from a cluster map generated according to the sub-pixel base calling described above. The reference truth attenuation map 1204 includes an array of cells and assigns at least one output value to each cell in the array. In some implementations, the cells are sub-pixels, and an output value is assigned to each sub-pixel based on the weighted attenuation factor. The reference truth attenuation map 1204 is then used as a reference truth for training the disclosed neural network-based template generator 1512. In some implementations, information from the reference truth attenuation map 1204 is also used to prepare inputs for the disclosed neural network-based base caller 1514.

[0367] Fig.12 One implementation of an exemplary ground truth decay map 1204 derived from an exemplary cluster map generated by sub-pixel base calling as described above is shown. In some implementations, in the upsampled cluster map, a value is assigned to each neighboring sub-pixel in a disjoint region on a cluster-by-cluster basis based on a decay factor 1102 that is proportional to the distance 1106 of the neighboring sub-pixel from the centroid sub-pixel 1104 in the disjoint region to which the neighboring sub-pixel belongs.

[0368] Fig.12 A ground truth attenuation map 1204 is depicted. In one implementation, the sub-pixel value is an intensity value normalized between zero and one. In another implementation, in the upsampled cluster map, the same predetermined value is assigned to all sub-pixels identified as background. In some implementations, the predetermined value is a zero intensity value.

[0369] In some specific implementations, a reference truth attenuation map 1204 is generated by a reference truth attenuation map generator 1202 based on the upsampled cluster map, and the reference truth attenuation map expresses adjacent sub-pixels in non-intersecting areas and sub-pixels identified as background based on the assigned values ​​of the adjacent sub-pixels. The reference truth attenuation map 1204 is stored in a memory to be used as a reference truth for training a classifier. In a specific implementation, each sub-pixel in the reference truth attenuation map 1204 has a value normalized between zero and one.

[0370] Ternary (three-category) mapping

[0371] Fig.13A specific implementation of deriving a reference truth ternary map 1304 from a cluster map is shown. The reference truth ternary map 1304 comprises an array of cells and assigns at least one output value to each cell in the array. As the name implies, the ternary map specific implementation of the reference truth ternary map 1304 assigns three output values ​​to each cell in the array, so that for each cell, the first output value corresponds to a classification label or score of the background class, the second output value corresponds to a classification label or score of the cluster center class, and the third output value corresponds to a classification label or score of the cluster / cluster internal class. The reference truth ternary map 1304 is used as reference truth data for training a neural network-based template generator 1512. In some specific implementations, information from the reference truth ternary map 1304 is also used to prepare inputs for a neural network-based base detector 1514.

[0372] Fig.13 An exemplary ground truth ternary map 1304 is depicted. In another implementation, in the upsampled cluster map, the ground truth ternary map generator 1302 classifies adjacent sub-pixels in the disjoint region cluster by cluster as cluster internal sub-pixels belonging to the same cluster, classifies centroid sub-pixels as cluster center sub-pixels, and classifies sub-pixels that do not belong to any cluster as background sub-pixels. In some implementations, the classifications are stored in the ground truth ternary map 1304. These classifications and the ground truth ternary map 1304 are stored in a memory to be used as ground truth for training a classifier.

[0373] In other specific implementations, the coordinates of the cluster interior sub-pixels, cluster center sub-pixels, and background sub-pixels are stored in a memory cluster by cluster to be used as ground truth for training the classifier. The coordinates are then scaled down by a factor used to upsample the cluster map. The scaled down coordinates are then stored in a memory cluster by cluster to be used as ground truth for training the classifier.

[0374] In other embodiments, the base truth ternary map generator 1302 generates ternary base truth data 1304 based on the upsampled cluster map using the cluster map. The ternary base truth data 1304 marks background sub-pixels as belonging to the background class, cluster center sub-pixels as belonging to the cluster center class, and cluster interior sub-pixels as belonging to the cluster interior class. In some visualization embodiments, color coding can be used to depict and distinguish different class labels. The ternary base truth data 1304 is stored in a memory to be used as a base truth for training a classifier.

[0375] Binary (two-class) mapping

[0376] Fig.14A specific implementation of a reference truth binary map 1404 derived from a cluster map is shown. Binary map 1404 comprises an array of cells and assigns at least one output value to each cell in the array. As the name implies, the binary map assigns two output values ​​to each cell in the array so that for each cell, the first output value corresponds to a classification mark or score of the cluster center class, and the second output value corresponds to a classification mark or score of the non-center class. The binary map is used as a reference truth data for training a template generator 1512 based on a neural network. In some specific implementations, information from the binary map is also used to prepare input for a base caller 1514 based on a neural network.

[0377] Fig.14 A ground truth binary map 1404 is depicted. The ground truth binary map generator 1402 uses cluster mapping Fig.120 Binary ground truth data 1404 is generated from the upsampled cluster map. The binary ground truth data 1404 marks the cluster center sub-pixel as belonging to the cluster center class and marks all other sub-pixels as belonging to the non-center class. The binary ground truth data 1404 is stored in a memory to be used as the ground truth for training the classifier.

[0378] In some implementations, the disclosed technology generates a cluster map for multiple blocks of a flow cell. Fig.120 , store these cluster maps in memory, and based on these cluster maps Fig.120 To determine the spatial distribution of clusters in the block, and determine the shape and size of these clusters. Then, the cluster map of the upsampled clusters in the block is Fig.120 In the disclosed technology, sub-pixels are classified into cluster-inside sub-pixels, cluster-center sub-pixels, and background sub-pixels belonging to the same cluster, cluster by cluster. Then, the disclosed technology stores the classification in a memory to be used as a reference truth for training a classifier, and stores the coordinates of the cluster-inside sub-pixels, cluster-center sub-pixels, and background sub-pixels in a memory, cluster by cluster, across blocks, to be used as a reference truth for training a classifier. In addition, the disclosed technology reduces the coordinates by a factor for upsampling a cluster map, and stores the reduced coordinates in a memory, cluster by cluster, across blocks, to be used as a reference truth for training a classifier.

[0379] In some implementations, the flow cell has at least one patterned surface having an array of pores that occupy clusters. In such implementations, the disclosed techniques determine, based on the determined shapes and sizes of the clusters: (1) which of the pores are substantially occupied by at least one cluster, (2) which of the pores are minimally occupied, and (3) which of the pores are co-occupied by multiple clusters. This allows for determination of corresponding metadata for multiple clusters that co-occupy the same pore, i.e., the center, shape, and size of two or more clusters that share the same pore.

[0380] In some implementations, the solid support on which the sample is amplified into clusters includes a patterned surface." Patterned surface" refers to the arrangement of different regions in or on the exposed layer of the solid support. For example, one or more regions in these regions can be features where there are one or more amplification primers. Features can be separated by gap regions where there are no amplification primers. In some implementations, the pattern can be the xy format of features in the form of rows and columns. In some implementations, the pattern can be the repeated arrangement of features and / or gap regions. In some implementations, the pattern can be the random arrangement of features and / or gap regions. Exemplary patterned surfaces that can be used in the methods and compositions described herein are described in U.S. Patent No. 8,778,849, U.S. Patent No. 9,079,148, U.S. Patent No. 8,778,848 and U.S. Patent Publication No. 2014 / 0243224, each of which is incorporated herein by reference.

[0381] In some embodiments, the solid support includes an array of holes or depressions in the surface. This can be manufactured using a variety of techniques as generally known in the art, including but not limited to photolithography, imprinting techniques, molding techniques, and micro-etching techniques. Those skilled in the art will appreciate that the technique used will depend on the composition and shape of the array substrate.

[0382] The features in the patterned surface can be holes (e.g., micropores or nanopores) in a hole array on a glass, silicon, plastic or other suitable solid support with a patterned and covalently attached gel (such as, poly (N- (5-azidoacetamidopentyl) acrylamide-co-acrylamide) (PAZAM, see, e.g., U.S. Patent Publication No. 2013 / 184796, WO 2016 / 066586 and WO 2015-002813, each of which is incorporated herein by reference in its entirety)). The method produces a gel pad for sequencing that can be stable in sequencing runs with a large number of cycles. The covalent attachment of the polymer to the hole helps to maintain the gel as a structured feature during multiple uses and throughout the life of the structured substrate. However, in many specific implementations, the gel need not be covalently attached to the hole. For example, under some conditions, a silane-free acrylamide (SFA, see, e.g., U.S. Pat. No. 8,563,477, which is incorporated herein by reference in its entirety) that is not covalently attached to any portion of the structured substrate can be used as a gel material.

[0383] In a particular implementation, a structured substrate can be made by patterning a solid support material with pores (e.g., micropores or nanopores), coating the patterned support with a gel material (e.g., PAZAM, SFA, or a chemically modified variant thereof, such as an azidated version of SFA (azido-SFA)), and polishing the gel-coated support, for example by chemical or mechanical polishing, so that the gel is retained in the pores, while substantially all of the gel is removed or inactivated from the interstitial regions on the surface of the structured substrate between the pores. Primer nucleic acids can be attached to the gel material. A solution of target nucleic acids (e.g., a fragmented human genome) can then be contacted with the polished substrate so that a single target nucleic acid will be seeded into a single pore by interaction with a primer attached to the gel material; however, the target nucleic acid will not occupy the interstitial region due to the absence of the gel material or the inactivation of the gel material. Amplification of the target nucleic acid will be confined to the pores because the absence of the gel in the interstitial region or the inactivation of the gel will prevent the outward migration of the growing nucleic acid colony. The process can be easily and scalably manufactured and utilizes micro- or nano-fabrication methods.

[0384] As used herein, the term "flow cell" refers to a chamber comprising a solid surface through which one or more fluid reagents can flow. The examples of flow cells and associated fluid systems and detection platforms that can be easily used in the disclosed method are described in, for example, Bentley et al., Nature, 456:53-59 (2008); WO 04 / 018497, US 7,057,026, WO 91 / 06678; WO 07 / 123744; US 7,329,492, US 7,211,414, US 7,315,019, US 7,405,281 and US 2008 / 0108082, each of which is incorporated herein by reference.

[0385] Throughout this disclosure, when referring to amplification primers, the terms "P5" and "P7" are used. It should be understood that any suitable amplification primer can be used in the method presented herein, and the use of P5 and P7 is only an exemplary embodiment. The use of amplification primers such as P5 and P7 on a circulation cell is known in the art, as illustrated by the disclosures of WO 2007 / 010251, WO 2006 / 064199, WO2005 / 065814, WO 2015 / 106941, WO 1998 / 044151, and WO 2000 / 018957, each of which is incorporated herein by reference in its entirety. For example, any suitable forward amplification primer, whether immobilized or in solution, can be used in the method presented herein for hybridization with a complementary sequence and amplification of the sequence. Similarly, any suitable reverse amplification primer, whether immobilized or in solution, can be used in the methods presented herein for hybridizing with complementary sequences and amplifying sequences. Those skilled in the art will understand how to design and use primer sequences suitable for capturing and amplifying nucleic acids presented herein.

[0386] In some implementations, the flow cell has at least one non-patterned surface, and the clusters are non-uniformly dispersed on the non-patterned surface.

[0387] In some implementations, the density of the clusters is about 100,000 clusters / mm 2 Up to about 1,000,000 clusters / mm 2 In other embodiments, the density of the clusters is about 1,000,000 clusters / mm 2 Up to about 10,000,000 clusters / mm 2 within the range.

[0388] In one implementation, the initial center coordinates of the clusters determined by the base caller are defined in the template image of the patch. In some implementations, the pixel resolution, image coordinate system, and measurement scale of the image coordinate system are the same for both the template image and the image.

[0389] In another specific implementation, the disclosed technology involves determining metadata about clusters on a block of a flow cell. First, the disclosed technology accesses (1) a set of images of the block captured during a sequencing run and (2) initial center coordinates of the clusters determined by a base caller.

[0390] Then, for each image set, the disclosed technique obtains base calls that classify the following into one of four bases: (1) an origin sub-pixel containing the initial center coordinates, and (2) a predetermined neighborhood of adjacent sub-pixels that are contiguously adjacent to a corresponding one of the origin sub-pixels. This produces a sequence of base calls for each of the origin sub-pixels and each of the predetermined neighborhoods of adjacent sub-pixels. The predetermined neighborhood of adjacent sub-pixels may be an m×n sub-pixel patch centered on the sub-pixel containing the origin sub-pixel. In one implementation, the sub-pixel patch is a 3×3 sub-pixel. In other implementations, the image patch may have any size, such as 5×5, 15×15, 20×20, etc. In other implementations, the predetermined neighborhood of adjacent sub-pixels may be an n-connected sub-pixel neighborhood centered on the sub-pixel containing the origin sub-pixel.

[0391] In one specific implementation, the disclosed technique identifies those sub-pixels in the cluster map that do not belong to any of the disjoint regions as background.

[0392] The disclosed technique then generates a cluster map that identifies clusters as non-intersecting regions of contiguous sub-pixels that: (a) are contiguously contiguous with at least some of the corresponding ones of the origin sub-pixels, and (b) share substantially matching base call sequences of one of the four bases with at least some of the corresponding ones of the origin sub-pixels.

[0393] The disclosed technology then stores the cluster map in a memory and determines the shape and size of the cluster based on the disjoint areas in the cluster map. In other implementations, the center of the cluster is also determined.

[0394] Generating training data for the template generator

[0395] Fig.15 is a block diagram illustrating one specific implementation of generating training data for training a neural network-based template generator 1512 and a neural network-based base caller 1514.

[0396] Fig.16Features of the disclosed training examples used to train the neural network-based template generator 1512 and the neural network-based base caller 1514 are shown. Each training example corresponds to a block and is labeled with a corresponding ground truth data representation. In some implementations, the ground truth data representation is a ground truth mask or ground truth map that identifies ground truth cluster metadata in the form of a ground truth decay map 1204, a ground truth ternary map 1304, or a ground truth binary map 1404. In some implementations, multiple training examples correspond to the same block.

[0397] In one specific implementation, the disclosed technology involves generating training data 1504 for neural network-based template generation and base calling. First, the disclosed technology accesses multiple images 108 of the flow cell 202 captured within multiple cycles of a sequencing run. The flow cell 202 has multiple blocks. In the multiple images 108, each block in the block has a sequence of image sets generated within the multiple cycles. Each image in the sequence of image sets 108 depicts the intensity emission of a cluster 302 and its surrounding background 304 at a specific block in the block in a specific cycle.

[0398] Then, the training set constructor 1502 constructs a training set 1504 having a plurality of training examples. Fig.16 As shown, each training example corresponds to a specific block in the block and includes image data from at least some of the image sets in the sequence of image sets 1602 for the specific block in the block. In one specific implementation, the image data includes images from at least some of the image sets in the sequence of image sets 1602 for the specific block in the block. For example, the image may have a resolution of 1800×1800. In other specific implementations, the image may have any resolution, such as 100×100, 3000×3000, 10000×10000, etc. In other specific implementations, the image data includes at least one image patch from each image. In one specific implementation, the image patch covers a portion of the specific block in the block. In one example, the image patch may have a resolution of 20×20. In other specific implementations, the image patch may have any resolution, such as 50×50, 70×70, 90×90, 100×100, 3000×3000, 10000×10000, etc.

[0399] In some implementations, the image data includes an upsampled representation of the image patch. The upsampled representation may have a resolution of, for example, 80×80. In other implementations, the upsampled representation may have any resolution, such as 50×50, 70×70, 90×90, 100×100, 3000×3000, 10000×10000, etc.

[0400] In some implementations, the plurality of training examples correspond to the same specific block in the blocks and respectively include as image data different image patches of each image in each of at least some of the image sets in the sequence of image sets 1602 for the same specific block in the blocks. In such implementations, at least some of the different image patches overlap with each other.

[0401] Then, a ground truth generator 1506 generates at least one ground truth data representation for each of the training examples. The ground truth data representation identifies at least one of the spatial distribution of clusters and their surrounding background on a particular one of the blocks, and identifies at least one of the cluster shapes, cluster sizes, and / or cluster boundaries and / or cluster centers, the intensity emissions of which are depicted by the image data, and their surrounding background.

[0402] In one embodiment, the reference truth data indicates that a cluster is identified as a non-intersecting area of ​​adjacent sub-pixels, the center of the cluster is identified as a centroid sub-pixel within a corresponding non-intersecting area in the non-intersecting area, and its surrounding background is identified as a sub-pixel that does not belong to any of the non-intersecting areas.

[0403] In one implementation, the ground truth data representation has an upsampled resolution of 80 × 80. In other implementations, the ground truth data representation may have any resolution, such as 50 × 50, 70 × 70, 90 × 90, 100 × 100, 3000 × 3000, 10000 × 10000, etc.

[0404] In one specific implementation, the ground truth data indicates that each sub-pixel is identified as a cluster center or a non-center. In another specific implementation, the ground truth data indicates that each sub-pixel is identified as a cluster interior, a cluster center, or a surrounding background.

[0405] In some implementations, the disclosed technology stores the training examples in the training set 1504 and the associated ground truth data 1508 in a memory as training data 1504 for training a neural network-based template generator 1512 and a neural network-based base caller 1514. The training is performed by a trainer 1510.

[0406] In some implementations, the disclosed technology generates training data for a variety of flow cells, sequencing instruments, sequencing protocols, sequencing chemistries, sequencing reagents, and cluster densities.

[0407] Neural network based template generator

[0408] In an inference or production implementation, the disclosed technology uses peak detection and segmentation to determine cluster metadata. The disclosed technology processes input image data 1702 derived from a series of image sets 1602 through a neural network 1706 to generate an alternative representation 1708 of the input image data 1702. For example, an image set may be used for a particular sequencing cycle and include four images (A, C, T, and G images), one image for each imaging channel. Then, for a sequencing run with fifty sequencing cycles, there will be fifty such image sets, i.e., a total of 200 images. When temporarily arranged, fifty image sets, each of which has four images, will form a series of image sets 1602. In some implementations, an image patch of a specific size is extracted from each image in the fifty image sets, thereby forming fifty image patch sets, each of which has four image patches, and in one implementation, this is the input image data 1702. In other specific implementations, the input image data 1702 includes image patch sets where each image patch set has four image patches, and the image patch sets are used for less than fifty sequencing cycles, i.e., only one, two, three, fifteen, twenty sequencing cycles.

[0409] Fig.17 One implementation is shown of processing input image data 1702 through a neural network-based template generator 1512 and generating output values ​​for each cell in an array. In one implementation, the array is an attenuation map 1716. In another implementation, the array is a ternary map 1718. In yet another implementation, the array is a binary map 1720. Thus, the array may represent one or more attributes of each of a plurality of locations represented in the input image data 1702.

[0410] Different from using the structure in the previous figure to train the template generator, the reference truth decay map 1204, the reference truth ternary map 1304 and the reference truth binary map 1404, the decay map 1716, the ternary map 1718 and / or the binary map 1720 are generated by the forward propagation of the trained neural network-based template generator 1512. Forward propagation can be performed during training or during inference. During training, due to the gradient update based on backpropagation, the decay map 1716, the ternary map 1718 and the binary map 1720 (i.e., cumulatively the output 1714) gradually match or approach the reference truth decay map 1204, the reference truth ternary map 1304 and the reference truth binary map 1404, respectively.

[0411] According to one implementation, the size of the image array analyzed during inference depends on the size of the input image data 1702 (e.g., the same or an enlarged or reduced version of the size). Each cell may represent a pixel, sub-pixel, or super-pixel. The cell-by-cell output values ​​of the array may characterize / represent / indicate the attenuation map 1716, the ternary map 1718, or the binary map 1720. In some implementations, the input image data 1702 is also an array of cells at pixel, sub-pixel, or super-pixel resolution. In such implementations, the neural network-based template generator 1512 uses semantic segmentation techniques to generate output values ​​for each cell in the input array. Additional details about the input image data 1702 can be found at Figure 21b , Fig. 22 , Fig.23 and Fig.24 and discussion of these figures.

[0412] In some specific implementations, the neural network-based template generator 1512 is a fully convolutional network, such as the fully convolutional network described in "Fully convolutional networks for semantic segmentation" (2015) by J. Long, E. Shelhamer, and T. Darrell in CVPR, which is incorporated herein by reference. In other specific implementations, the neural network-based template generator 1512 is a U-Net network with jump connections between the decoder and the encoder, such as the U-Net network described in "U-net: Convolutional networks for biomedical image segmentation" (2015) by Ronneberger O, Fischer P, Brox T. in Med. Image Comput. Comput. Assist. Interv (the document is obtained from http: / / link.springer.com / chapter / 10.1007 / 978-3-319-24574-4_28), which is incorporated herein by reference. The U-Net architecture is similar to an autoencoder with two main substructures: 1) an encoder that takes an input image and reduces its spatial resolution through multiple convolutional layers to create a representation encoding. 2) a decoder that takes the representation encoding and increases the spatial resolution back to produce a reconstructed image as output. U-Net introduces two innovations to the architecture: first, setting the objective function to reconstruct the segmentation mask using a loss function; second, using skip connections to connect the convolutional layers of the encoder to the corresponding layers of the decoder with the same resolution. In another embodiment, the neural network-based template generator 1512 is a deep fully convolutional segmentation neural network with an encoder subnetwork and a corresponding decoder network. In such embodiments, the encoder subnetwork includes an encoder hierarchy, and the decoder subnetwork includes a decoder hierarchy that maps the low-resolution encoder feature map to the full-input resolution feature map. Additional details about the segmentation network can be found in the Appendix entitled "Segmentation Network".

[0413] In one implementation, the neural network based template generator 1512 is a convolutional neural network. In another implementation, the neural network based template generator 1512 is a recursive neural network. In yet another implementation, the neural network based template generator 1512 is a residual neural network with residual blocks and residual connections. In another implementation, the neural network based template generator 1512 is a combination of a convolutional neural network and a recursive neural network.

[0414] Those skilled in the art will appreciate that the neural network-based template generator 1512 (i.e., the neural network 1706 and / or the output layer 1710) can use various padding and stride configurations. It can use different output functions (e.g., classification or regression) and may or may not include one or more fully connected layers. It can use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or dilated convolution, transposed convolution, depthwise separable convolution, pointwise convolution, 1×1 convolution, grouped convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It can use one or more loss functions, such as logistic regression / logarithmic loss function, multi-class cross entropy / softmax loss function, binary cross entropy loss function, mean square error loss function, L1 loss function, L2 loss function, smoothed L1 loss function, and Huber loss function. It can use any parallelism, efficiency and compression scheme, such as TFRecords, compression encoding (e.g., PNG), sharpening, parallel detection of map conversion, batching, prefetching, model parallelism, data parallelism and synchronous / asynchronous SGD. It can include upsampling layers, downsampling layers, recurrent connections, gate and gate memory units (such as LSTM or GRU), residual blocks, residual connections, high-speed connections, skip connections, peephole connections, activation functions (e.g., non-linear transformation functions such as rectified linear units (ReLU), leaky ReLU, exponential lining units (ELU), sigmoid and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout layers, pooling layers (e.g., max or average pooling), global average pooling layers and attention mechanisms.

[0415] In some implementations, each image in the sequence of image set 1602 covers a block and depicts the intensity emission captured for a specific imaging channel in a specific sequencing cycle of multiple sequencing cycles of a sequencing run performed on a flow cell for a cluster on the block and its surrounding background. In one implementation, the input image data 1702 includes at least one image patch from each image in the sequence of images from the image set 1602. In such implementations, the image patch covers a portion of the block. In one example, the image patch has a resolution of 20×20. In other cases, the resolution of the image patch may be in the range of 20×20 to 10000×10000. In another implementation, the input image data 1702 includes an upsampled sub-pixel resolution representation of the image patch from each image in the sequence of images from the image set 1602. In one example, the upsampled sub-pixel representation has a resolution of 80×80. In other cases, the resolution of the upsampled sub-pixel representation may be in the range of 80×80 to 10000×10000.

[0416] The input image data 1702 has a cell array 1704 depicting a cluster and its surrounding background. For example, an image set may be used for a particular sequencing cycle and include four images (A, C, T, and G images), one for each imaging channel. Then, for a sequencing run with fifty sequencing cycles, there will be fifty such image sets, i.e., 200 images in total. When temporarily arranged, fifty image sets, each of which has four images, will form a series of image sets 1602. In some implementations, an image patch of a specific size is extracted from each image in the fifty image sets, thereby forming fifty image patch sets, each of which has four image patches, and in one implementation, this is the input image data 1702. In other implementations, the input image data 1702 includes image patch sets, each of which has four image patches, which are used for less than fifty sequencing cycles, i.e., only one, two, three, fifteen, twenty sequencing cycles. An alternative representation is a feature map. When the neural network is a convolutional neural network, the feature map may be a convolutional feature or a convolutional representation. When the neural network is a recurrent neural network, the feature map may be a hidden state feature or a hidden state representation.

[0417] The disclosed technology then processes the alternative representation 1708 through an output layer 1710 to generate an output 1714 having an output value 1712 for each cell in the array 1704. The output layer can be a classification layer such as softmax or sigmoid that produces cell-by-cell output values. In one specific implementation, the output layer is a ReLU layer or any other activation function layer that produces cell-by-cell output values.

[0418] In one implementation, the units in the input image data 1702 are pixels, and thus pixel-by-pixel output values ​​1712 are generated in the output 1714. In another implementation, the units in the input image data 1702 are sub-pixels, and thus sub-pixel-by-sub-pixel output values ​​1712 are generated in the output 1714. In yet another implementation, the units in the input image data 1702 are super-pixels, and thus super-pixel-by-super-pixel output values ​​1712 are generated in the output 1714.

[0419] Export cluster metadata from decay, ternary, and / or binary maps

[0420] Fig.18One implementation of a post-processing technique is shown, which is applied to the attenuation map 1716, ternary map 1718, or binary map 1720 generated by the neural network-based template generator 1512 to derive cluster metadata, including cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries. In some implementations, the post-processing technique is applied by a post-processor 1814, which also includes a thresholder 1802, a peak locator 1806, and a segmenter 1810.

[0421] The input to the thresholder 1802 is an attenuation map 1716, a ternary map 1718, or a binary map 1720 generated by a template generator 1512 (such as the disclosed neural network-based template generator). In one specific implementation, the thresholder 1802 applies thresholding to the values ​​in the attenuation map, the ternary map, or the binary map to identify background cells 1804 (i.e., sub-pixels that characterize the non-cluster background) and non-background cells. In other words, once the output 1714 is generated, the thresholder 1802 thresholds the output values ​​of the cells 1712 and classifies or reclassifies a first subset of the cells 1712 as "background cells" 1804, which depict the surrounding background of the cluster, and "non-background cells" that depict cells that may belong to the cluster. The thresholds applied by the thresholder 1802 can be preset.

[0422] The input to the peak locator 1806 is also the attenuation map 1716, the ternary map 1718, or the binary map 1720 generated by the neural network-based template generator 1512. In one implementation, the peak locator 1806 applies peak detection to the values ​​in the attenuation map 1716, the ternary map 1718, or the binary map 1720 to identify the center cell 1808 (i.e., the center sub-pixel that characterizes the center of the cluster). In other words, the peak locator 1806 processes the output values ​​of the cells 1712 in the output 1714 and classifies a second subset of the cells 1712 as "center cells" 1808 that contain cluster centers. In some implementations, the cluster centers detected by the peak locator 1806 are also cluster centroids. The center cells 1808 are then provided to the segmenter 1810. Additional details about the peak locator 1806 can be found in the Appendix entitled "Peak Detection".

[0423] Thresholding and peak detection can be performed in parallel or sequentially. That is, they are independent of each other.

[0424] The input to the segmenter 1810 is also the attenuation map 1716, ternary map 1718, or binary map 1720 generated by the neural network-based template generator 1512. Additional complementary inputs to the segmenter 1810 include the threshold cells (background, non-background cells) 1804 identified by the thresholder 1802, and the center cell 1808 identified by the peak locator 1806. The segmenter 1810 uses the background, non-background cells 1804, and the center cell 1808 to identify the disjoint regions 1812 (i.e., non-overlapping adjacent clusters / cluster-internal sub-pixel groups that characterize clusters). In other words, the segmenter 1810 processes the output values ​​of the cells 1712 in the output 1714, and uses the background, non-background cells 1804, and the center cell 1808 to determine the shape 1812 of the cluster as a non-overlapping region of adjacent cells separated by the background cells 1804 and centered on the center cell 1808. The output of the segmenter 1810 is the cluster metadata 1812. Cluster metadata 1812 identifies cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries.

[0425] In one implementation, segmenter 1810 starts with center cell 1808 and determines for each center cell a set of consecutive adjacent cells that depict the same cluster whose centroid is contained in the center cell. In one implementation, segmenter 1810 uses a so-called "watershed" segmentation technique to subdivide adjacent clusters into multiple adjacent clusters at valley intensities. Additional details on the watershed segmentation technique and other segmentation techniques can be found in the appendix entitled "Watershed Segmentation".

[0426] In one implementation, the output values ​​of the units 1712 in the output 1714 are continuous values, such as the values ​​encoded in the ground truth decay map 1204. In another implementation, the output values ​​are softmax scores, such as the scores encoded in the ground truth ternary map 1304 and the ground truth binary map 1404. In the ground truth decay map 1204, according to one implementation, the neighboring cells in the corresponding non-overlapping regions of the non-overlapping regions have output values ​​weighted according to the distance of the neighboring cells from the center cell in the non-overlapping region to which the neighboring cells belong. In this implementation, the center cell has the highest output value in the corresponding non-overlapping region of the non-overlapping regions. As described above, during training, due to back-propagation-based gradient updates, the attenuation map 1716, the ternary map 1718, and the binary map 1720 (i.e., cumulatively the output 1714) gradually match or approach the reference truth attenuation map 1204, the reference truth ternary map 1304, and the reference truth binary map 1404, respectively.

[0427] Pixel Domain—Intensity Extraction for Irregular Cluster Shapes

[0428] We now discuss how the cluster shape determined by the disclosed techniques can be used to extract the intensity of the cluster. Since clusters typically have irregular shapes and contours, the disclosed techniques can be used to identify which sub-pixels contribute to the non-intersecting / non-overlapping regions of the irregular shape representing the cluster shape.

[0429] Fig.19 A specific implementation of extracting cluster intensities in the pixel domain is depicted. A "template image" or "template" may refer to a data structure that contains or identifies cluster metadata 1812 derived from the attenuation map 1716, the ternary map 1718, and / or the binary map 1718. The cluster metadata 1812 identifies cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries.

[0430] In some implementations, the template image is in the upsampled sub-pixel domain to distinguish cluster boundaries at a fine-grained level. However, the sequencing image 108 containing cluster and background intensity data is usually in the pixel domain. Therefore, the disclosed technology proposes two methods to use the cluster shape information encoded in the template image at the upsampled sub-pixel resolution to extract the intensity of irregular-shaped clusters from the optical pixel resolution sequencing image. In the first method, as Fig.19 As depicted, non-overlapping groups of adjacent sub-pixels identified in the template image are located in the pixel-resolution sequencing image, and the intensities of these non-overlapping groups of adjacent sub-pixels are extracted by interpolation. Additional details on this intensity extraction technique can be found in Fig.33 and a discussion about the figure.

[0431] In one implementation, when the non-overlapping regions have irregular contours and the units are sub-pixels, the cluster intensity 1912 of a given cluster is determined by the intensity extractor 1902 as follows.

[0432] First, sub-pixel locator 1904 identifies sub-pixels that contribute to the cluster intensity of a given cluster based on corresponding non-overlapping areas of adjacent sub-pixels that identify the shape of the given cluster.

[0433] The sub-pixel locator 1904 then locates the identified sub-pixels in one or more optical pixel resolution images 1918 generated for one or more imaging channels in the current sequencing cycle. In one implementation, integer or non-integer coordinates (e.g., floating point) are located in the optical pixel resolution images after downscaling based on a downscaling factor that matches the upsampling factor used to create the sub-pixel domain.

[0434] Then, the interpolator and sub-pixel intensity combiner 1906 performs the following operations: interpolating the intensities of the sub-pixels identified in the processed image, combining the interpolated intensities and normalizing the combined interpolated intensities to produce a per-image cluster intensity for a given cluster in each image in the image. Normalization is performed by a normalizer 1908 and is based on a normalization factor. In one specific implementation, the normalization factor is the number of sub-pixels identified. This is done to normalize / account for different cluster sizes and the uneven illumination received by the clusters (which depends on their position in the flow cell).

[0435] Finally, a cross-channel sub-pixel intensity accumulator 1910 combines the per-image cluster intensities for each image in the image to determine a cluster intensity 1912 for a given cluster in the current sequencing cycle.

[0436] The given cluster is then base called based on the cluster intensity 1912 in the current sequencing cycle by any of the base callers discussed in this application, resulting in base calls 1916.

[0437] However, in some implementations, when the cluster size is large enough, the output of the neural network-based base caller 1514 (i.e., the attenuation map 1716, the ternary map 1718, and the binary map 1720) is in the optical pixel domain. Therefore, in such implementations, the template image is also in the optical pixel domain.

[0438] Sub-pixel domain—intensity extraction of irregular cluster shapes

[0439] Fig. 20 A second method for extracting cluster intensities in the sub-pixel domain is depicted. In this second method, the sequencing image at optical pixel resolution is upsampled to sub-pixel resolution. This results in a correspondence between the "cluster shapes depicted in sub-pixels" in the template image and the "cluster intensities depicted in sub-pixels" in the upsampled sequencing image. Cluster intensities are then extracted based on the correspondence. Additional details on this intensity extraction technique can be found in Fig.33 and a discussion about the figure.

[0440] In one implementation, when the non-overlapping regions have irregular contours and the units are sub-pixels, the cluster intensity 2012 of a given cluster is determined by the intensity extractor 2002 as follows.

[0441] First, sub-pixel locator 2004 identifies sub-pixels that contribute to the cluster intensity of a given cluster based on corresponding non-overlapping areas of adjacent sub-pixels that identify the shape of the given cluster.

[0442] The sub-pixel locator 2004 then locates the identified sub-pixels in one or more optical pixel resolution images 2018 upsampled from the corresponding optical pixel resolution images 1918 generated for one or more imaging channels in the current sequencing cycle. Upsampling can be performed by: nearest neighbor intensity extraction, Gaussian-based intensity extraction, intensity extraction based on the average of a 2×2 sub-pixel area, intensity extraction based on the brightest point in a 2×2 sub-pixel area, intensity extraction based on the average of a 3×3 sub-pixel area, bilinear intensity extraction, bicubic intensity extraction, and / or intensity extraction based on weighted area coverage. These techniques are described in detail in the Appendix entitled "Intensity Extraction Methods". In some embodiments, a template image can be used as a mask for intensity extraction.

[0443] Then, in each of the upsampled images, a subpixel intensity combiner 2006 combines the intensities of the identified subpixels and normalizes the combined intensities to produce a per-image cluster intensity for a given cluster in each of the upsampled images. Normalization is performed by a normalizer 2008 and is based on a normalization factor. In one specific implementation, the normalization factor is the number of identified subpixels. This is done to normalize / account for different cluster sizes and the uneven illumination received by the clusters (depending on their position in the flow cell).

[0444] Finally, a cross-channel sub-pixel intensity accumulator 2010 combines the per-image cluster intensities for each of the upsampled images to determine a cluster intensity 2012 for a given cluster in the current sequencing cycle.

[0445] The given cluster is then base called based on the cluster intensity 2012 in the current sequencing cycle by any of the base callers discussed in this application, resulting in base calls 2016.

[0446] Types of Neural Network-Based Template Generators

[0447] The details of three different implementations of the neural network-based template generator 1512 are now discussed. Fig.21a , and includes: (1) a template generator 2600 based on an attenuation map (also known as a regression model), (2) a template generator 4600 based on a binary map (also known as a binary classification model), and (3) a template generator 5400 based on a ternary map (also known as a ternary classification model).

[0448] In one implementation, the regression model 2600 is a fully convolutional network. In another implementation, the regression model 2600 is a U-Net network with skip connections between the decoder and the encoder. In one implementation, the binary classification model 4600 is a fully convolutional network. In another implementation, the binary classification model 4600 is a U-Net network with skip connections between the decoder and the encoder. In one implementation, the ternary classification model 5400 is a fully convolutional network. In another implementation, the ternary classification model 5400 is a U-Net network with skip connections between the decoder and the encoder.

[0449] Input image data

[0450] Figure 21b Depicted is one specific implementation of input image data 1702 fed as input to a neural network-based template generator 1512. The input image data 1702 includes a series of image sets 2100 having sequencing images 108 generated during a certain number of initial sequencing cycles (e.g., the first 2 to 7 sequencing cycles) of a sequencing run.

[0451] In some implementations, the intensity of the sequencing images 108 is corrected for background and / or the images are aligned to each other using an affine transform. In one implementation, the sequencing run utilizes a four-channel chemistry, and each image set has four images. In another implementation, the sequencing run utilizes a dual-channel chemistry, and each image set has two images. In yet another implementation, the sequencing run utilizes a single-channel chemistry, and each image set has two images. In other implementations, each image set has only one image. These and other different implementations are described in Appendix 6 and 9.

[0452] Each image 2116 in the series of image sets 2100 covers a block 2104 of the flow cell 2102 and depicts the intensity emission of the cluster 2106 and its surrounding background on the block 2104 captured for a particular imaging channel in a particular sequencing cycle among a plurality of sequencing cycles of a sequencing run. In one example, for cycle t1, the image set includes four images 2112A, 2112C, 2112T, and 2112G: one image for each base A, C, T, and G, which are labeled with corresponding fluorescent dyes and imaged in corresponding wavelength bands (images / imaging channels).

[0453] For schematic illustration, in image 2112G, Figure 21bCluster intensity emission is depicted as 2108 and background intensity emission is depicted as 2110. In another example, for cycle tn, the image set includes four images 2114A, 2114C, 2114T, and 2114G: one image for each base A, C, T, and G, which are labeled with corresponding fluorescent dyes and imaged in corresponding wavelength bands (images / imaging channels). Also for schematic illustration, in image 2114A, Figure 21b The cluster intensity emission is depicted as 2118 and in image 2114T, the background intensity emission is depicted as 2120.

[0454] Non-image data

[0455] Input image data 1702 is encoded using intensity channels (also referred to as imaging channels). For each image in the c-image obtained from the sequencer for a particular sequencing cycle, its intensity signal data is encoded using a separate imaging channel. For example, considering that the sequencing run uses a dual-channel chemistry that produces a red image and a green image in each sequencing cycle. In this case, the input data 2632 includes (i) a first red imaging channel with w×h pixels, which depicts the intensity emission captured in the red image of the one or more clusters and their surrounding background; and (ii) a second green imaging channel with w×h pixels, which depicts the intensity emission captured in the green image of the one or more clusters and their surrounding background.

[0456] In another specific implementation, image data is not used as input to the neural network-based template generator 1512 or the neural network-based base caller 1514. Instead, the input to the neural network-based template generator 1512 and the neural network-based base caller 1514 is based on pH changes due to the release of hydrogen ions during molecular extension. The pH change is detected and converted into a voltage change that is proportional to the number of bases incorporated (e.g., in the case of Ion Torrent).

[0457] In yet another specific implementation, the inputs of the neural network-based template generator 1512 and the neural network-based base detector 1514 are constructed according to nanopore sensing, which uses a biosensor to measure the interruption of the current when the analyte passes through the nanopore or approaches its orifice, while determining the type of base. For example, Oxford Nanopore Technology (ONT) sequencing is based on the following concept: single-stranded DNA (or RNA) is passed through a membrane via a nanopore, and a voltage difference is applied across the membrane. The nucleotides present in the hole will affect the resistance of the hole, so the current measurement results over time can indicate the sequence of DNA bases passing through the hole. This current signal (called a "squiggle" because of its appearance when plotted) is the raw data collected by the ONT sequencer. These measurements are stored as 16-bit integer data acquisition (DAC) values ​​acquired at a frequency of (for example) 4kHz. In the case of a DNA chain speed of about 450 base pairs / second, this gives an average of about nine raw observations per base. The signal is then processed to identify the interruption of the opening signal corresponding to each reading. The greatest use of these raw signals is to perform base calling, the process of converting DAC values ​​into DNA base sequences. In some implementations, input data 2632 includes normalized or scaled DAC values.

[0458] Patch extraction

[0459] Fig. 22 Shows from Figure 21b In one embodiment, the sequenced images 108 in the sequence of image sets 2100 are extracted from the sequence of image sets 2100 to generate a series of "reduced size" image sets that form the input image data 1702. In the illustrated embodiment, the size of the sequencing images 108 in the sequence of image sets 2100 is L×L (e.g., 2000×2000). In other embodiments, L is any number in the range of 1 to 10,000.

[0460] In one specific implementation, the patch extractor 2202 extracts patches from the sequencing images 108 in the series of image sets 2100, and generates a series of reduced-size image sets 2206, 2208, 2210, and 2212. Each image in the series of reduced-size image sets is a patch of size M×M (e.g., 20×20) extracted from a corresponding sequencing image in the series of image sets 2100. The size of the patch can be preset. In other specific implementations, M is any number in the range of 1 to 1000.

[0461] exist Fig. 22, four exemplary series of reduced-size image sets are shown. A first exemplary series of reduced-size image sets 2206 are extracted from the sequenced images 108 (coordinates 0, 0 to 20, 20) in the series of image sets 2100. A second exemplary series of reduced-size image sets 2208 are extracted from the sequenced images 108 (coordinates 20, 20 to 40, 40) in the series of image sets 2100. A third exemplary series of reduced-size image sets 2210 are extracted from the sequenced images 108 (coordinates 40, 40 to 60, 60) in the series of image sets 2100. A fourth exemplary series of reduced-size image sets 2212 are extracted from the sequenced images 108 (coordinates 60, 60 to 80, 80) in the series of image sets 2100.

[0462] In some implementations, the series of reduced-size image sets form input image data 1702, which is fed as input to the neural network-based template generator 1512. Multiple series of reduced-size image sets can be fed simultaneously as an input batch, and a separate output can be generated for each series in the input batch.

[0463] Upsampling

[0464] Fig.23 Describes the Figure 21b A specific implementation of upsampling a series of image sets 2100 in to produce a series of “upsampled” image sets 2300 that form the input image data 1702.

[0465] In one specific implementation, the upsampler 2302 uses an interpolation method (eg, bicubic interpolation) to upsample the sequenced images 108 in the series of image sets 2100 by an upsampling factor (eg, 4 times) to generate a series of upsampled image sets 2300 .

[0466] In the illustrated implementation, the sequencing images 108 in the series of image sets 2100 have a size of L×L (e.g., 2000×2000) and are upsampled by an upsampling factor of 4 to produce upsampled images of size U×U (e.g., 8000×8000) in the series of upsampled image sets 2300.

[0467] In one specific implementation, the sequencing images 108 in the series of image sets 2100 are directly fed to the neural network-based template generator 1512, and upsampling is performed by the initial layer of the neural network-based template generator 1512. That is, the upsampler 2302 is part of the neural network-based template generator 1512, and operates as its first layer, which upsamples the sequencing images 108 in the series of image sets 2100 and generates a series of upsampled image sets 2300.

[0468] In some implementations, a series of upsampled image sets 2300 form input image data 1702, which is fed as input to the neural network-based template generator 1512.

[0469] Fig.24 Shows from Fig.23 A specific implementation of extracting patches from the series of upsampled image sets 2300 to produce a series of “upsampled and downsized” image sets 2406 , 2408 , 2410 , and 2412 that form the input image data 1702 .

[0470] In one implementation, the patch extractor 2202 extracts patches from the upsampled images in the series of upsampled image sets 2300, and generates a series of upsampled and downsized image sets 2406, 2408, 2410, and 2412. Each upsampled image in the series of upsampled and downsized image sets is a patch of size M×M (e.g., 80×80) extracted from a corresponding upsampled image in the series of upsampled image sets 2300. The size of the patch may be preset. In other implementations, M is any number in the range of 1 to 1000.

[0471] exist Fig.24 , four exemplary series of upsampled and downsized image sets are shown. A first exemplary series of upsampled and downsized image sets 2406 are extracted from the upsampled images (coordinates 0,0 to 80,80) in the series of upsampled image sets 2300. A second exemplary series of upsampled and downsized image sets 2408 are extracted from the upsampled images (coordinates 80,80 to 160,160) in the series of upsampled image sets 2300. A third exemplary series of upsampled and downsized image sets 2410 are extracted from the upsampled images (coordinates 160,160 to 240,240) in the series of upsampled image sets 2300. A fourth exemplary series of upsampled and downsized image sets 2412 are extracted from the upsampled images (coordinates 240,240 to 320,320) in the series of upsampled image sets 2300.

[0472] In some implementations, the series of upsampled and downsized image sets form input image data 1702, which is fed as input to the neural network based template generator 1512. Multiple series of upsampled and downsized image sets can be fed simultaneously as an input batch, and a separate output can be generated for each series in the input batch.

[0473] Output

[0474] The three models are trained to produce different outputs. This is achieved by using different types of ground truth data representations as training labels. The regression model 2600 is trained to produce an output that characterizes / represents / indicates the so-called "attenuation map" 1716. The binary classification model 4600 is trained to produce an output that characterizes / represents / indicates the so-called "binary map" 1720. The ternary classification model 5400 is trained to produce an output that characterizes / represents / indicates the so-called "ternary map" 1718.

[0475] The output 1714 of each type of model includes an array of cells 1712. Cells 1712 can be pixels, sub-pixels, or super-pixels. The output of each type of model includes cell-by-cell output values, such that the output values ​​of the cell arrays together characterize / represent / indicate an attenuation map 1716 in the case of the regression model 2600, a binary map 1720 in the case of the binary classification model 4600, and a ternary map 1718 in the case of the ternary classification model 5400. More details are as follows.

[0476] Benchmark truth data generation

[0477] Fig.25 One specific implementation of the overall exemplary process for generating ground truth data for training a neural network-based template generator 1512 is shown. For a regression model 2600, the ground truth data may be a decay map 1204. For a binary classification model 4600, the ground truth data may be a binary map 1404. For a ternary classification model 5400, the ground truth data may be a ternary map 1304. The ground truth data is generated from cluster metadata. The cluster metadata is generated by a cluster metadata generator 122. The ground truth data is generated by a ground truth data generator 1506.

[0478] In the illustrated implementation, ground truth data is generated for block A on channel A of flow cell A. Ground truth data is generated from sequencing images 108 of block A captured during sequencing run A. Sequencing images 108 of block A are in the pixel domain. In one example involving four-channel chemistry generating four sequencing images per sequencing cycle, fifty sequencing cycles yield two hundred sequencing images 108. Each of the two hundred sequencing images 108 depicts the intensity emission of a cluster on block A and its surrounding background captured in a particular imaging channel in a particular sequencing cycle.

[0479] The sub-pixel addresser 110 converts the sequenced image 108 into the sub-pixel domain (eg, by dividing each pixel into a plurality of sub-pixels) and generates a sequenced image 112 in the sub-pixel domain.

[0480] Then, a base caller 114 (eg, RTA) processes the sequencing image 112 in the sub-pixel domain and generates a base call for each sub-pixel and each of the fifty sequencing cycles. This is referred to herein as "sub-pixel base calling."

[0481] The sub-pixel base calls are then merged 116 to produce a base call sequence for each sub-pixel across fifty sequencing cycles. The base call sequence for each sub-pixel has fifty base calls, ie, one base call for each of the fifty sequencing cycles.

[0482] Searcher 118 evaluates the base call sequences of adjacent sub-pixels pair by pair. The search involves evaluating each sub-pixel to determine which of its adjacent sub-pixels it shares a substantially matching base call sequence with. The base call sequences of adjacent sub-pixels are "substantially matched" when a predetermined portion of the base calls match in positional order (e.g., there are ≥41 matches in 45 cycles, ≤4 mismatches in 45 cycles, ≤4 mismatches in 50 cycles, or ≤2 mismatches in 34 cycles).

[0483] In some implementations, the base caller 114 also identifies the initial center coordinates of the cluster. The sub-pixel containing the initial center coordinates is referred to as the center or origin sub-pixel. Figure 6 604a-c) and the corresponding origin sub-pixels (606a-c) identified by the base caller 114 are shown in FIG. However, the origin sub-pixel (the initial center coordinate of the cluster) need not be identified, as described below. In some implementations, the searcher 118 uses a breadth-first search to identify a substantially matching base call sequence of sub-pixels by starting with the origin sub-pixels 606a-c and continuing with consecutive adjacent non-origin sub-pixels 702a-c. This is also optional, as described below.

[0484] The search for a substantially matching base call sequence for a sub-pixel does not require identification of the origin sub-pixel (the initial center coordinate of the cluster) because the search can be performed on all sub-pixels, and the search does not have to start from the origin sub-pixel, but can start from any sub-pixel (e.g., the 0,0 sub-pixel or any random sub-pixel). Therefore, since each sub-pixel is evaluated to determine whether it shares a substantially matching base call sequence with another adjacent sub-pixel, the search does not have to utilize the origin sub-pixel and can start at any sub-pixel.

[0485] Regardless of whether the origin sub-pixel is used, certain clusters are identified that do not contain the origin sub-pixel (the initial center coordinate of the cluster) predicted by base caller 114. Some examples of clusters identified by merging sub-pixel base calling and that do not contain the origin sub-pixel are Figure 8aThus, using base caller 114 to identify the origin sub-pixel (initial center coordinate of the cluster) is optional and not necessary for searching for substantially matching base call sequences of sub-pixels.

[0486] Searcher 118: (1) identifies adjacent sub-pixels having substantially matching base call sequences as so-called "disjoint regions," (2) further evaluates the base call sequences of those sub-pixels that do not belong to any of the disjoint regions identified at (1) to generate additional disjoint regions, and (3) then identifies background sub-pixels as those sub-pixels that do not belong to either of the disjoint regions identified at (1) and (2). Action (2) allows the disclosed technique to identify additional or additional clusters whose centers have not yet been identified by base caller 114.

[0487] The results of the searcher 118 are encoded in a so-called "cluster map" for tile A and stored in a cluster map data storage device 120. In the cluster map, each of the clusters on tile A is identified by a corresponding non-intersecting region of adjacent sub-pixels, with background sub-pixels separating the non-intersecting regions to identify the surrounding background on the tile.

[0488] A centroid (COM) calculator 1004 determines the center of each of the clusters on block A by calculating the COM of each of the non-intersecting regions as the average of the coordinates of the corresponding adjacent sub-pixels forming the non-intersecting regions. The centroid of the cluster is stored as COM data 2502 .

[0489] The sub-pixel classifier 2504 uses the cluster map and the COM data 2502 to generate a sub-pixel classification 2506. The sub-pixel classification 2506 classifies the sub-pixels in the cluster map into: (1) background sub-pixels, (2) COM sub-pixels (each non-intersecting region of the COM containing the corresponding non-intersecting region has one COM sub-pixel), and (3) cluster / cluster-internal sub-pixels forming the corresponding non-intersecting region. That is, each sub-pixel in the cluster map is assigned to one of the three categories.

[0490] Based on the sub-pixel classification 2506, in some specific implementations, (i) the baseline truth attenuation map 1204 is generated by the baseline truth attenuation map generator 1202, (ii) the baseline truth binary map 1304 is generated by the baseline truth binary map generator 1302, and (iii) the baseline truth ternary map 1404 is generated by the baseline truth ternary map generator 1402.

[0491] 1. Regression Model

[0492] Fig.26One implementation of a regression model 2600 is shown. In the illustrated implementation, the regression model 2600 is a fully convolutional network 2602 that processes input image data 1702 through an encoder subnetwork and a corresponding decoder subnetwork. The encoder subnetwork includes an encoder hierarchy. The decoder subnetwork includes a decoder hierarchy that maps a low-resolution encoder feature map to a full input resolution attenuation map 1716. In another implementation, the regression model 2600 is a U-Net network 2604 with jump connections between the decoder and the encoder. Additional details about the segmentation network can be found in the Appendix entitled "Segmentation Network".

[0493] Attenuation Map

[0494] Fig. 27 A specific implementation of generating a reference truth attenuation map 1204 from a cluster map 2702 is depicted. The reference truth attenuation map 1204 is used as reference truth data for training the regression model 2600. In the reference truth attenuation map 1204, the reference truth attenuation map generator 1202 assigns a weighted attenuation value to each adjacent sub-pixel in the non-intersecting region based on a weighted attenuation factor. The weighted attenuation value is proportional to the Euclidean distance of the adjacent sub-pixel from the centroid (COM) sub-pixel in the non-intersecting region to which the adjacent sub-pixel belongs, so that the weighted attenuation value is the highest (e.g., 1 or 100) for the COM sub-pixel, and is lower for the sub-pixel farther away from the COM sub-pixel. In some specific implementations, the weighted attenuation value is multiplied by a preset factor, such as 100.

[0495] Additionally, the ground truth attenuation map generator 1202 assigns the same predetermined value (eg, a floor background value) to all background sub-pixels.

[0496] The reference truth attenuation map 1204 expresses neighboring sub-pixels and background sub-pixels in the disjoint region based on the assigned values. The reference truth attenuation map 1204 also stores the assigned values ​​in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0497] train

[0498] Fig.28 It is a specific implementation of training 2800 the regression model 2600 using a back-propagation based gradient update technique, which modifies the parameters of the regression model 2600 until the training output of the regression model 2600 during training 2800 gradually approaches or matches the reference true value attenuation map 1204 to generate an attenuation map 1716.

[0499] The training 2800 includes iteratively optimizing a loss function that minimizes an error 2806 between the attenuation map 1716 and the ground truth attenuation map 1204, and updating parameters of the regression model 2600 based on the error 2806. In one specific implementation, the loss function is a mean squared error, and the error is minimized sub-pixel by sub-pixel between the weighted attenuation values ​​of the corresponding sub-pixels in the attenuation map 1716 and the ground truth attenuation map 1204.

[0500] Training 2800 includes hundreds, thousands, and / or millions of iterations of forward propagation 2808 and backward propagation 2810, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 is operated by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0501] infer

[0502] Fig.29 is a specific implementation of generating a template by the regression model 2600 during inference 2900, wherein the attenuation map 1716 is produced by the regression model 2600 as an inference output during inference 2900. An example of the attenuation map 1716 is disclosed in the Appendix entitled "Regression_Model_Sample_Ouput". The Appendix includes unit-by-unit weighted attenuation output values ​​2910 that together represent the attenuation map 1716.

[0503] Inference 2900 includes hundreds, thousands, and / or millions of iterations of forward propagation 2904, including parallelization techniques such as batch processing. Inference 2900 is performed on inference data 2908, which includes a series of upsampled and downsized image sets as input image data 1702. Inference 2900 is operated by a tester 2906.

[0504] Watershed segmentation

[0505] Fig.30 One specific implementation of subjecting the attenuation map 1716 to (i) thresholding to identify background sub-pixels that characterize the background of a cluster and (ii) peak detection to identify center sub-pixels that characterize the center of a cluster is shown. Thresholding is performed by a thresholder 1802 that uses a local threshold binary to produce a binarized output. Peak detection is performed by a peak locator 1806 to identify the center of a cluster. Additional details about the peak locator can be found in the Appendix entitled "Peak Detection".

[0506] Fig.31One specific implementation of a watershed segmentation technique is depicted that takes as input the background sub-pixels identified by thresholder 1802 and the center sub-pixels identified by peak locator 1806, finds the valley intensities between adjacent clusters, and outputs non-overlapping sets of adjacent clusters / cluster interior sub-pixels that characterize the clusters. Additional details on the watershed segmentation technique can be found in the appendix entitled "Watershed Segmentation".

[0507] In one specific implementation, the watershed segmenter 3102 takes as input the following data: (1) the negative output values ​​2910 in the attenuation map 1716, (2) the binarized output of the thresholder 1802, and (3) the cluster centers identified by the peak locator 1806. The watershed segmenter 3102 then generates an output 3104 based on the input. In the output 3104, each cluster center is identified as a set of unique sub-pixels that belong to the cluster center (as long as the sub-pixel is "1" in the binarized output, that is, it is not a background sub-pixel, it belongs to the cluster center). In addition, the clusters are filtered based on containing at least four sub-pixels. The watershed segmenter 3102 can be part of the segmenter 1810, which in turn can be part of the post-processor 1814.

[0508] Network Architecture

[0509] Fig.32 is a table showing an exemplary U network architecture of the regression model 2600, as well as details of the layers of the regression model 2600, the dimensions of the outputs of the layers, the magnitudes of the model parameters, and the interconnections between the layers. Similar details are disclosed in a file entitled “Regression_Model_Example_Architecture”, which is filed as an appendix to this application.

[0510] Cluster strength extraction

[0511] Fig.33 Different methods of extracting cluster intensities using cluster shape information identified in a template image are shown. As described above, the template image identifies cluster shape information at upsampled sub-pixel resolution. However, the cluster intensity information is in the sequencing images 108, which are typically at optical pixel resolution.

[0512] According to a first method, the coordinates of the sub-pixels are located in the sequencing image 108 and their respective intensities are extracted using bilinear interpolation and normalized based on the count of sub-pixels contributing to the cluster.

[0513] The second method uses a weighted area coverage technique to modulate the intensity of a pixel according to the number of sub-pixels contributing to the pixel. Here again, the modulated pixel intensity is normalized by the sub-pixel count parameter.

[0514] The third method upsamples the sequenced image into the sub-pixel domain using bicubic interpolation, sums the intensities of the upsampled pixels belonging to the cluster, and normalizes the summed intensities based on the counts of the upsampled pixels belonging to the cluster.

[0515] Experimental results and observations

[0516] Fig.34 Different methods of base calling using the output of the regression model 2600 are shown. In the first method, cluster centers identified from the output of the neural network-based template generator 1512 in the template image are fed to a base caller (e.g., Illumina's Real-Time Analysis software, referred to herein as the "RTA Base Caller") for base calling.

[0517] In the second method, instead of cluster centers, cluster intensities extracted from the sequencing image based on cluster shape information in the template image are fed to the RTA base caller for base calling.

[0518] Fig.35 The difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center is shown. The results show that using the COM improves base calling.

[0519] Example Model Output

[0520] Fig.36 An exemplary attenuation map 1716 produced by the regression model 2600 is shown on the left. Fig.36 Also shown on the right is an exemplary ground truth decay map 1204 that the regression model 2600 approximated during training.

[0521] Both the attenuation map 1716 and the baseline truth attenuation map 1204 depict clusters as non-intersecting regions of adjacent sub-pixels, depict the center of the cluster as the central sub-pixel at the centroid of the corresponding non-intersecting regions in the non-intersecting regions, and depict its surrounding background as background sub-pixels that do not belong to any of the non-intersecting regions.

[0522] In addition, adjacent sub-pixels in corresponding ones of the disjoint regions have values ​​weighted according to the distance of the adjacent sub-pixels from the center sub-pixel in the disjoint regions to which the adjacent sub-pixels belong. In one implementation, the center sub-pixel has the highest value in the corresponding ones of the disjoint regions. In one implementation, the background sub-pixels all have the same minimum background value in the attenuation map.

[0523] Fig.37One specific implementation of peak locator 1806 is depicted for identifying cluster centers in an attenuation map by detecting peaks 3702. Additional details regarding the peak locator can be found in the Appendix entitled "Peak Detection."

[0524] Fig.38 The peaks detected by peak locator 1806 in attenuation map 1716 produced by regression model 2600 are compared to the corresponding peaks in ground truth attenuation map 1204. The red markers are the peaks predicted by regression model 2600 to be cluster centers, and the green markers are the ground truth centroids of the clusters.

[0525] More experimental results and observations

[0526] Fig.39 The performance of the regression model 2600 using precision and recall statistics is shown. The precision and recall statistics indicate that the regression model 2600 is suitable for recovering all identified cluster centers.

[0527] Fig.40 The performance of regression model 2600 was compared to the RTA base caller for a 20 pM library concentration (normal operation). Outperforming the RTA base caller, regression model 2600 identified 34,323 more (4.46%) clusters (i.e., 988,884 clusters) in a higher cluster density environment.

[0528] Fig.40 Results for other sequencing metrics are also shown, such as the number of clusters passing the purity filter ("% PF" (passed filter)), the number of aligned reads ("% Aligned"), the number of duplicate reads ("% Duplicate"), the number of reads that mismatched the reference sequence among all reads aligned to the reference sequence ("% Mismatch"), the detected bases with a quality score of 30 and above ("% Q30 Bases"), and so on.

[0529] Fig.41 The performance of regression model 2600 was compared to that of the RTA base caller for a 30 pM library concentration (dense run). Outperforming the RTA base caller, regression model 2600 identified 34,323 more (6.27%) clusters (i.e., 1,351,588 clusters) in a much higher cluster density environment.

[0530] Fig.41 Results for other sequencing metrics are also shown, such as the number of clusters passing the purity filter ("% PF" (passed filter)), the number of aligned reads ("% Aligned"), the number of duplicate reads ("% Duplicate"), the number of reads that mismatched the reference sequence among all reads aligned to the reference sequence ("% Mismatch"), the detected bases with a quality score of 30 and above ("% Q30 Bases"), and so on.

[0531] Fig.42 The number of non-duplicate (unique or de-duplicated) correct read pairs detected by the regression model 2600 (i.e., the number of paired reads where both reads are aligned within a reasonable distance) was compared to the number of non-duplicate correct read pairs detected by the RTA base caller. Both the 20pM normal run and the 30pM dense run were compared.

[0532] More importantly, Fig.42 It is shown that the disclosed neural network-based template generator is able to detect more clusters in sequencing cycles with fewer template generation inputs than the RTA base caller. In only four sequencing cycles, the regression model 2600 identified 11% more non-duplicate correct read pairs than the RTA base caller during normal operation at 20 pM and 33% more non-duplicate correct read pairs than the RTA base caller during intensive operation at 30 pM. In only seven sequencing cycles, the regression model 2600 identified 4.5% more non-duplicate correct read pairs than the RTA base caller during normal operation at 20 pM and 6.3% more non-duplicate correct read pairs than the RTA base caller during intensive operation at 30 pM.

[0533] Fig.43 On the right is shown a first attenuation map generated by the regression model 2600. The first attenuation map identifies clusters imaged during normal operation at 20 pM and their surrounding background, and its spatial distribution depicting cluster shape, cluster size, and cluster centers.

[0534] Fig.43 Shown on the left is a second attenuation map generated by the regression model 2600. The second attenuation map identifies clusters imaged during the 30 pM intensive run and their surrounding background, and its spatial distribution depicting cluster shape, cluster size, and cluster centers.

[0535] Fig.44 The performance of regression model 2600 and RTA base caller for 40 pM library concentration (highly dense run) were compared. Regression model 2600 produced 89,441,688 more aligned bases than RTA base caller in a much higher cluster density environment (ie, 1,509,395 clusters).

[0536] Fig.44 Results for other sequencing metrics are also shown, such as the number of clusters passing the purity filter ("% PF" (passed filter)), the number of aligned reads ("% Aligned"), the number of duplicate reads ("% Duplicate"), the number of reads that mismatched the reference sequence among all reads aligned to the reference sequence ("% Mismatch"), the detected bases with a quality score of 30 and above ("% Q30 Bases"), and so on.

[0537] More example model outputs

[0538] Fig.45 Shown on the left is a first attenuation map generated by the regression model 2600. The first attenuation map identifies clusters imaged during normal operation at 40 pM and their surrounding background, and its spatial distribution depicting cluster shape, cluster size, and cluster centers.

[0539] Fig.45 The results of the following operations are shown on the right: thresholding and peak location are applied to the first attenuation map to distinguish the corresponding clusters from each other and from the background, and to identify their corresponding cluster centers. In some implementations, the intensity of the corresponding clusters is identified, and a purity filter (or filtering) is applied to reduce the mismatch rate.

[0540] 2. Binary Classification Model

[0541] Fig.46 One implementation of a binary classification model 4600 is shown. In the illustrated implementation, the binary classification model 4600 is a deep fully convolutional segmentation neural network that processes input image data 1702 through an encoder subnetwork and a corresponding decoder subnetwork. The encoder subnetwork includes an encoder hierarchy. The decoder subnetwork includes a decoder hierarchy that maps a low-resolution encoder feature map to a full-input resolution binary map 1720. In another implementation, the binary classification model 4600 is a U-Net network with jump connections between the decoder and the encoder. Additional details about the segmentation network can be found in the appendix entitled "Segmentation Network".

[0542] Binary Map

[0543] The final output layer of the binary classification model 4600 is a unit-by-unit classification layer that produces a classification label for each unit in the output array. In some implementations, the unit-by-unit classification layer is a sub-pixel classification layer that produces a softmax classification score distribution across two classes (i.e., cluster center class and non-cluster class) for each sub-pixel in the binary map 1720, and the classification label of a given sub-pixel is determined by the corresponding softmax classification score distribution.

[0544] In other embodiments, the unit-by-unit classification layer is a sub-pixel-by-sub-pixel classification layer that produces a sigmoid classification score for each sub-pixel in the binary map 1720, such that the activation of a unit is interpreted as the probability that the unit belongs to the first class, and conversely, 1 minus the activation gives the probability that it belongs to the second class.

[0545] The binary map 1720 represents each sub-pixel based on the predicted classification score. The binary map 1720 also stores the predicted classification scores in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0546] train

[0547] Fig.47 It is a specific implementation of training 4700 the binary classification model 4600 using a back-propagation based gradient update technique, which modifies the parameters of the binary classification model 4600 until the binary map 1720 of the binary classification model 4600 gradually approaches or matches the reference truth binary map 1404.

[0548] In the illustrated implementation, the final output layer of the binary classification model 4600 is a softmax-based sub-pixel classification layer. In the softmax implementation, the ground truth binary map generator 1402 assigns each ground truth sub-pixel to either (i) a cluster center value pair (e.g., [1,0]) or (ii) a non-center value pair (e.g., [0,1]).

[0549] In the cluster center value pair [1,0], the first value [1] represents the cluster center class label, and the second value [0] represents the non-center class label. In the non-center value pair [0,1], the first value [0] represents the cluster center class label, and the second value [1] represents the non-center class label.

[0550] The ground truth binary map 1404 expresses each sub-pixel based on an assigned value pair / value. The ground truth binary map 1404 also stores the assigned value pairs / values ​​in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0551] Training includes iteratively optimizing a loss function that minimizes an error 4706 (eg, a softmax error) between the binary map 1720 and the ground truth binary map 1404 , and updating parameters of the binary classification model 4600 based on the error 4706 .

[0552] In one specific implementation, the loss function is a custom weighted binary cross entropy loss, and the error 4706 is minimized on a sub-pixel-by-sub-pixel basis between the predicted classification scores (e.g., softmax scores) and the labeled class scores (e.g., softmax scores) of the corresponding sub-pixels in the binary map 1720 and the ground truth binary map 1404, as Fig.47 shown.

[0553] The custom weighted loss function will give more weight to COM sub-pixels, so that whenever a COM sub-pixel is misclassified, the cross entropy loss is multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Additional details about the custom weighted loss function can be found in the appendix titled "Custom Weighted Loss Function".

[0554] Training 4700 includes hundreds, thousands, and / or millions of iterations of forward propagation 4708 and backward propagation 4710, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 is operated by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0555] Fig.48 is another specific implementation of training 4800 the binary classification model 4600, wherein the final output layer of the binary classification model 4600 is a sigmoid-based sub-pixel classification layer.

[0556] In a sigmoid implementation, the ground truth binary map generator 1302 assigns each ground truth sub-pixel either (i) a cluster center value (e.g., [1]) or (ii) a non-center value (e.g., [0]). The COM sub-pixel is assigned a cluster center value pair / value, and all other sub-pixels are assigned a non-center value pair / value.

[0557] For cluster center values, values ​​above a threshold midpoint between 0 and 1 (e.g., values ​​above 0.5) represent center class labels. For non-center values, values ​​below a threshold midpoint between 0 and 1 (e.g., values ​​below 0.5) represent non-center class labels.

[0558] The ground truth binary map 1404 expresses each sub-pixel based on an assigned value pair / value. The ground truth binary map 1404 also stores the assigned value pairs / values ​​in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0559] Training includes iteratively optimizing a loss function that minimizes an error 4806 (eg, a sigmoid error) between the binary map 1720 and the ground truth binary map 1404 , and updating parameters of the binary classification model 4600 based on the error 4806 .

[0560] In one specific implementation, the loss function is a custom weighted binary cross entropy loss, and the error 4806 is minimized on a sub-pixel-by-sub-pixel basis between the predicted scores (e.g., sigmoid scores) and the labeled scores (e.g., sigmoid scores) of the corresponding sub-pixels in the binary map 1720 and the ground truth binary map 1404, as Fig.48 shown.

[0561] The custom weighted loss function will give more weight to COM sub-pixels, so that whenever a COM sub-pixel is misclassified, the cross entropy loss is multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Additional details about the custom weighted loss function can be found in the appendix titled "Custom Weighted Loss Function".

[0562] Training 4800 includes hundreds, thousands, and / or millions of iterations of forward propagation 4808 and backward propagation 4810, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 is operated by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0563] Fig.49 Another specific implementation of the input image data 1702 fed to the binary classification model 4600 and the corresponding class labels 4904 used to train the binary classification model 4600 is shown.

[0564] In the illustrated implementation, the input image data 1702 includes a series of upsampled and downsized image sets 4902. The class labels 4904 include two classes: (1) "no cluster center" and (2) "cluster center", which are distinguished using different output values. That is, (1) light green cells / sub-pixels 4906 represent sub-pixels predicted by the binary classification model 4600 as not containing cluster centers, and (2) dark green sub-pixels 4908 represent cells / sub-pixels predicted by the binary classification model 4600 as containing cluster centers.

[0565] infer

[0566] Fig.50is a specific implementation of generating a template by the binary classification model 4600 during inference 5000, wherein the binary map 1720 is generated by the binary classification model 4600 as an inference output during inference 5000. An example of the binary map 1720 includes unit-by-unit binary classification scores 5010 that together represent the binary map 1720. In a softmax application, the binary map 1720 has a first array 5002a of unit-by-unit classification scores for non-center classes and a second array 5002b of unit-by-unit classification scores for cluster center classes.

[0567] Inference 5000 includes hundreds, thousands, and / or millions of iterations of forward propagation 5004, including parallelization techniques such as batch processing. Inference 5000 is performed on inference data 2908, which includes a series of upsampled and downsized image sets as input image data 1702. Inference 5000 is operated by tester 2906.

[0568] In some implementations, the binary map 1720 is subjected to the post-processing techniques discussed above, such as thresholding, peak detection, and / or watershed segmentation, to generate cluster metadata.

[0569] Peak Detection

[0570] Fig.51 A specific implementation of peak detection for a binary map 1720 to identify cluster centers is depicted. As described above, the binary map 1720 is an array of cells that classify each sub-pixel based on a predicted classification score, where each cell in the array represents a corresponding sub-pixel in the input. The classification score can be a softmax score or a sigmoid score.

[0571] In the softmax application, the binary map 1720 includes two arrays: (1) a first array 5002a of cell-by-cell classification scores for the non-center class and (2) a second array 5002b of cell-by-cell classification scores for the cluster center class. In both arrays, each cell represents a corresponding sub-pixel in the input.

[0572] To determine which sub-pixels in the input contain cluster centers and which sub-pixels do not contain cluster centers, the peak locator 1806 applies peak detection to the cells in the binary map 1720. Peak detection identifies those cells whose classification scores (e.g., softmax / sigmoid scores) are above a preset threshold. The identified cells are inferred to be cluster centers, and the corresponding sub-pixels of these cells in the input are determined to contain cluster centers and are stored as cluster center sub-pixels in the sub-pixel classification data storage device 5102. Additional details about the peak locator 1806 can be found in the Appendix entitled "Peak Detection".

[0573] The remaining cells and their corresponding sub-pixels in the input are determined to not contain the cluster center and are stored as non-center sub-pixels in the sub-pixel classification data storage device 5102.

[0574] In some implementations, those cells whose classification scores are below a certain background threshold (e.g., 0.3) are set to zero before peak detection is applied. In some implementations, such cells and their corresponding sub-pixels in the input are inferred to represent the background around the cluster and are stored as background sub-pixels in the sub-pixel classification data storage 5102. In other implementations, such cells may be considered noise and ignored.

[0575] Example Model Output

[0576] Fig.52a An exemplary binary map produced by the binary classification model 4600 is shown on the left. Fig.52a Also shown on the right is an exemplary ground truth binary map approximated during training by the binary classification model 4600. The binary map has multiple sub-pixels, and each sub-pixel is classified as a cluster center or a non-center. Similarly, the ground truth binary map has multiple sub-pixels, and each sub-pixel is classified as a cluster center or a non-center.

[0577] Experimental results and observations

[0578] Fig.52b The performance of the binary classification model 4600 using recall and precision statistics is shown. Applying these statistics, the binary classification model 4600 outperforms the RTA base caller.

[0579] Network Architecture

[0580] Fig.53 is a table showing an example architecture of a binary classification model 4600, as well as details of the layers of the binary classification model 4600, the dimensions of the outputs of the layers, the magnitudes of the model parameters, and the interconnections between the layers. Similar details are disclosed in the Appendix titled "Binary_Classification_Model_Example_Architecture".

[0581] 3. Ternary (three-category) classification model

[0582] Fig.54One implementation of a ternary classification model 5400 is shown. In the illustrated implementation, the ternary classification model 5400 is a deep fully convolutional segmentation neural network that processes input image data 1702 through an encoder subnetwork and a corresponding decoder subnetwork. The encoder subnetwork includes an encoder hierarchy. The decoder subnetwork includes a decoder hierarchy that maps a low-resolution encoder feature map to a full-input resolution ternary map 1718. In another implementation, the ternary classification model 5400 is a U-Net network with jump connections between the decoder and the encoder. Additional details about the segmentation network can be found in the appendix entitled "Segmentation Network".

[0583] Ternary Map

[0584] The final output layer of the ternary classification model 5400 is a unit-by-unit classification layer that produces a classification label for each unit in the output array. In some specific implementations, the unit-by-unit classification layer is a sub-pixel classification layer that produces a softmax classification score distribution across three classes (i.e., background class, cluster center class, and cluster / cluster interior class) for each sub-pixel in the ternary map 1718, and the classification label of a given sub-pixel is determined by the corresponding softmax classification score distribution.

[0585] The ternary map 1718 expresses each sub-pixel based on the predicted classification score. The ternary map 1718 also stores the predicted classification scores in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0586] train

[0587] Fig.55 It is a specific implementation of training 5500 the ternary classification model 5400 using a back-propagation based gradient update technique, which modifies the parameters of the ternary classification model 5400 until the ternary map 1718 of the ternary classification model 5400 gradually approaches or matches the reference truth ternary map 1304.

[0588] In the illustrated implementation, the final output layer of the ternary classification model 5400 is a softmax-based sub-pixel classification layer. In the softmax implementation, the ground truth ternary map generator 1402 assigns each ground truth sub-pixel to (i) a background value triplet (e.g., [1,0,0]), (ii) a cluster center value triplet (e.g., [0,1,0]), or (iii) a cluster / cluster internal value triplet (e.g., [0,0,1]).

[0589] Background sub-pixels are assigned a background value triplet. Centroid (COM) sub-pixels are assigned a cluster center value triplet. Cluster / cluster interior sub-pixels are assigned a cluster / cluster interior value triplet.

[0590] In the background value triplet [1,0,0], the first value [1] represents the background class label, the second value [0] represents the cluster center label, and the third value [0] represents the cluster / cluster interior class label.

[0591] In the cluster center value triplet [0, 1, 0], the first value [0] represents the background class label, the second value [1] represents the cluster center label, and the third value [0] represents the cluster / cluster internal class label.

[0592] In the cluster / cluster internal value triplet [0,0,1], the first value [0] represents the background class label, the second value [0] represents the cluster center label, and the third value [1] represents the cluster / cluster internal class label.

[0593] The ground truth ternary map 1304 expresses each sub-pixel based on the assigned value triplet. The ground truth ternary map 1304 also stores the assigned triplet in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0594] Training includes iteratively optimizing a loss function that minimizes an error 5506 (eg, a softmax error) between the ternary map 1718 and the ground truth ternary map 1304 , and updating parameters of the ternary classification model 5400 based on the error 5506 .

[0595] In one specific implementation, the loss function is a custom weighted categorical cross entropy loss, and the error 5506 is minimized on a sub-pixel basis between the predicted classification scores (e.g., softmax scores) and the labeled class scores (e.g., softmax scores) of the corresponding sub-pixels in the ternary map 1718 and the ground truth ternary map 1304, as Fig.54 shown.

[0596] The custom weighted loss function will give more weight to COM sub-pixels, so that whenever a COM sub-pixel is misclassified, the cross entropy loss is multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Additional details about the custom weighted loss function can be found in the appendix titled "Custom Weighted Loss Function".

[0597] Training 5500 includes hundreds, thousands, and / or millions of iterations of forward propagation 5508 and backward propagation 5510, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 5500 is operated by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0598] Fig.56 A specific implementation of the input image data 1702 fed to the ternary classification model 5400 and the corresponding class labels used to train the ternary classification model 5400 is shown.

[0599] In the illustrated implementation, the input image data 1702 includes a series of upsampled and downsized image sets 5602. The class labels 5604 include three classes: (1) "background class", (2) "cluster center class", and (3) "cluster interior class", which are distinguished using different output values. For example, some of these different output values ​​can be visually represented as follows: (1) gray cells / sub-pixels 5606 represent sub-pixels predicted by the ternary classification model 5400 to be background, (2) dark green cells / sub-pixels 5608 represent sub-pixels predicted by the ternary classification model 5400 to contain cluster centers, and (3) light green sub-pixels 5610 represent sub-pixels predicted by the ternary classification model 5400 to contain the interior of the cluster.

[0600] Network Architecture

[0601] Fig.57 is a table showing an example architecture of a ternary classification model 5400, as well as details of the layers of the ternary classification model 5400, the dimensions of the outputs of the layers, the magnitudes of the model parameters, and the interconnections between the layers. Similar details are disclosed in the Appendix entitled "Ternary_Classification_Model_Example_Architecture".

[0602] infer

[0603] Fig.58 is a specific implementation of generating a template by the ternary classification model 5400 during inference 5800, wherein the ternary map 1718 is generated by the ternary classification model 5400 as an inference output during inference 5800. An example of the ternary map 1718 is disclosed in the Appendix entitled "Ternary_Classification_Model_Sample_Ouput". The Appendix includes unit-by-unit binary classification scores 5810 that together represent the ternary map 1718. In the softmax application, the Appendix has a first array 5802a of unit-by-unit classification scores for background classes, a second array 5802b of unit-by-unit classification scores for cluster center classes, and a third array 5802c of unit-by-unit classification scores for cluster / cluster internal classes.

[0604] Inference 5800 includes hundreds, thousands, and / or millions of iterations of forward propagation 5804, including parallelization techniques such as batch processing. Inference 5800 is performed on inference data 2908, which includes a series of upsampled and downsized image sets as input image data 1702. Inference 5000 is operated by tester 2906.

[0605] In some implementations, the ternary map 1718 is generated by the ternary classification model 5400 using the post-processing techniques discussed above, such as thresholding, peak detection, and / or watershed segmentation.

[0606] Fig.59 The ternary map 1718 produced by the ternary classification model 5400 is graphically depicted, where each sub-pixel has a three-way softmax classification score distribution for three corresponding classes (i.e., background class 5906, cluster center class 5902, and cluster / cluster interior class 5904).

[0607] Fig.60 Depicted is an array of cells generated by the ternary classification model 5400, and the cell-by-cell output values. As depicted, each cell has three output values ​​for three corresponding classes, namely, background class 5906, cluster center class 5902, and cluster / cluster interior class 5904. For each classification (column-by-column), each cell is assigned the class with the highest output value, as indicated by the class in parentheses below each cell. In some specific implementations, the output values ​​6002, 6004, and 6006 (row-by-row) are analyzed for each of the corresponding classes 5906, 5902, and 5904.

[0608] Peak detection and watershed segmentation

[0609] Fig.61 A specific implementation of post-processing the ternary map 1718 to identify cluster centers, cluster backgrounds, and cluster interiors is shown. As described above, the ternary map 1718 is an array of cells that classify each sub-pixel based on a predicted classification score, where each cell in the array represents a corresponding sub-pixel in the input. The classification score can be a softmax score.

[0610] In the softmax application, the ternary map 1718 includes three arrays: (1) a first array 5802a of unit-by-unit classification scores for the background class, (2) a second array 5802b of unit-by-unit classification scores for the cluster center class, and (3) a third array 5802c of unit-by-unit classification scores for the cluster interior class. In all three arrays, each cell represents a corresponding sub-pixel in the input.

[0611] To determine which sub-pixels in the input contain cluster centers, which sub-pixels contain the interior of the cluster, and which sub-pixels contain the background, the peak locator 1806 applies peak detection to the softmax values ​​in the ternary map 1718 of the cluster center class 5802b. Peak detection identifies those cells whose classification scores (e.g., softmax scores) are above a preset threshold. The identified cells are inferred to be cluster centers, and the corresponding sub-pixels of these cells in the input are determined to contain cluster centers and are stored as cluster center sub-pixels in the sub-pixel classification and segmentation data storage device 6102. Additional details about the peak locator 1806 can be found in the Appendix entitled "Peak Detection".

[0612] In some implementations, before applying peak detection, those cells whose classification scores are below a certain noise threshold (e.g., 0.3) are set to zero. Such cells can be considered noise and ignored.

[0613] In addition, cells of the background class 5802a whose classification scores are above a certain background threshold (e.g., equal to or greater than 0.5) and their corresponding sub-pixels in the input are inferred to represent the background surrounding the cluster and are stored as background sub-pixels in the sub-pixel classification and segmentation data storage device 6102.

[0614] Then, a watershed segmentation algorithm operated by the watershed segmenter 3102 is used to determine the shape of the cluster. In some specific implementations, the background cells / sub-pixels are used as masks by the watershed segmentation algorithm. The classification scores of the cells / sub-pixels inferred as cluster centers and cluster interiors are added to produce a so-called "cluster marker". The cluster center is used as a watershed marker for separation by the intensity valley by the watershed segmentation algorithm.

[0615] In one implementation, the negative feature cluster markers are provided as an input image to a watershed segmenter 3102, which performs segmentation and generates cluster shapes as non-intersecting regions of adjacent cluster interior sub-pixels separated by background sub-pixels. In addition, each non-intersecting region includes a corresponding cluster center sub-pixel. In some implementations, the corresponding cluster center sub-pixel is the center of the non-intersecting region to which it belongs. In other implementations, the centroid (COM) of the non-intersecting region is calculated based on the base position coordinates and is stored as the new center of the cluster.

[0616] The output of the watershed segmenter 3102 is stored in the sub-pixel classification and segmentation data storage device 6102. Additional details on the watershed segmentation algorithm and other segmentation algorithms can be found in the Appendix entitled "Watershed Segmentation".

[0617] Example outputs of the peak locator 1806 and the watershed segmenter 3102 are shown in Fig.62a , Figure 62b , Fig.63 and Fig.64 Shown in.

[0618] Example Model Output

[0619] Fig.62a An exemplary prediction of the ternary classification model 5400 is shown. Fig.62a Four maps are shown, and each map has an array of cells. The first map 6202 (leftmost) shows the output value of each cell of the cluster center class 5802b. The second map 6204 shows the output value of each cell of the cluster / cluster interior class 5802c. The third map 6206 (rightmost) shows the output value of each cell of the background class 5802a. The fourth map 6208 (bottom) is a binary mask of the reference truth ternary map 6008, which assigns the class label with the highest output value to each cell.

[0620] Figure 62b Other exemplary predictions of the ternary classification model 5400 are shown. Figure 62b Four maps are shown, and each map has an array of cells. The first map 6212 (bottom left) shows the output value of each cell of the cluster / cluster interior class. The second map 6214 shows the output value of each cell of the cluster center class. The third map 6216 (bottom right) shows the output value of each cell of the background class. The fourth map (top) 6210 is a reference truth ternary map, which assigns the class label with the highest output value to each cell.

[0621] Figure 62c Other exemplary predictions of the ternary classification model 5400 are also shown. Fig.64 Four maps are shown, and each map has an array of cells. The first map 6220 (bottom left) shows the output value of each cell of the cluster / cluster interior class. The second map 6222 shows the output value of each cell of the cluster center class. The third map 6224 (bottom right) shows the output value of each cell of the background class. The fourth map 6218 (top) is a reference truth ternary map, which assigns the class label with the highest output value to each cell.

[0622] Fig.63 Describes the output from post-processing Fig.62a The output of the ternary classification model 5400 in derives a specific implementation of cluster centers and cluster shapes. Post-processing (e.g., peak location, watershed segmentation) generates cluster shape data and other metadata identified in the cluster map 6310.

[0623] Experimental results and observations

[0624] Fig.64The performance of the binary classification model 4600, the regression model 2600, and the RTA base caller were compared. Multiple sequencing metrics were used to evaluate performance. One metric is the total number of clusters detected ("# clusters"), which can be measured by the number of unique cluster centers detected. Another metric is the number of clusters detected that passed the purity filter ("% PF" (passed filtering)). During the 1st-25th cycles of the sequencing run, the purity filter removes the least reliable clusters from the image extraction results. A cluster "passes filtering" if no more than one base call has a purity value below 0.6 in the first 25 cycles. Purity is defined as the ratio of the brightest base signal intensity divided by the sum of the brightest and second brightest base signal intensities. This metric goes beyond the number of clusters detected and also conveys their quality, i.e., how many of the detected clusters can be used for accurate base calling and downstream binary and ternary analysis (such as variant calling and variant pathogenicity annotation).

[0625] Other metrics for measuring how good a detected cluster is for downstream analysis include the number of aligned reads generated by the detected cluster ("% Aligned"), the number of duplicate reads generated by the detected cluster ("% Duplicates"), the number of reads generated by the detected cluster that mismatch the reference sequence among all reads aligned to the reference sequence ("% Mismatches"), the number of reads generated by the detected cluster whose portions did not match the reference sequence on either side well and were therefore ignored in the alignment ("% Soft Clipping"), the number of bases detected for the detected cluster with a quality score of 30 or above ("% Q30 Bases"), the number of paired reads generated by the detected cluster in which both reads are aligned within a reasonable distance ("Total Correct Read Pairs"), and the number of unique or deduplicated correct read pairs generated by the detected cluster ("Non-Duplicate Correct Read Pairs").

[0626] like Fig.64 As shown, in template generation, the binary classification model 4600 and the regression model 2600 outperform the RTA base caller in most metrics.

[0627] Fig.65 The performance of the ternary classification model 5400 was compared to that of the RTA base caller under three contexts, five sequencing metrics, and two run densities.

[0628] In a first context, referred to as "RTA", cluster centers are detected by the RTA base caller, intensity extraction of clusters is performed by the RTA base caller, and clusters are also base called using the RTA base caller. In a second context, referred to as "RTA IE", cluster centers are detected by the ternary classification model 5400; however, intensity extraction of clusters is performed by the RTA base caller, and clusters are also base called using the RTA base caller. In a third context, referred to as "Self IE", cluster centers are detected by the ternary classification model 5400, and intensity extraction of clusters is performed using the cluster shape-based intensity extraction technique disclosed herein (note that the cluster shape information is generated by the ternary classification model 5400); however, clusters are base called using the RTA base caller.

[0629] Performance was compared between the ternary classification model 5400 and the RTA base caller based on five metrics: (1) the total number of clusters detected ("#clusters"), (2) the number of clusters detected that passed the purity filter ("#PF"), (3) the number of unique or deduplicated correct read pairs generated by the detected clusters ("#non-duplicate correct read pairs"), (4) the mismatch rate between the sequence reads generated by the detected clusters and the reference sequence after alignment ("%Mismatch Rate"), and (5) the bases called with a quality score of 30 or above for the detected clusters ("%Q30").

[0630] The performance between the ternary classification model 5400 and the RTA base caller was compared under five metrics in three contexts and two types of sequencing runs: (1) normal run with 20 pM library concentration and (2) dense run with 30 pM library concentration.

[0631] like Fig.65 As shown, the ternary classification model 5400 outperforms the RTA base caller in all metrics.

[0632] Under the same three contexts, five metrics, and two run densities, Fig.66 It is shown that regression model 2600 outperforms the RTA base caller in all metrics.

[0633] Fig.67 Focus on the penultimate layer 6702 of the neural network based template generator 1512.

[0634] Fig.68 Visualizes what the penultimate layer 6702 of the neural network-based template generator 1512 has learned as a result of back-propagation-based gradient update training. The illustrated implementation visualizes 24 of the 32 convolutional filters of the penultimate layer 6702 superimposed on the ground truth cluster shapes. Fig.68As shown, the penultimate layer 6702 has learned cluster metadata, including the spatial distribution of clusters, such as cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and cluster boundaries.

[0635] Fig.69 Cluster center predictions (blue) from the binary classification model 4600 are superimposed on those from the RTA base caller (pink). Predictions were made on sequencing image data from an Illumina NextSeq sequencer.

[0636] Fig.70 Cluster center predictions (pink) made by the RTA base caller are superimposed on a visualization of the trained convolutional filters of the penultimate layer of the binary classification model 4600. These convolutional filters were learned as a result of training on sequencing image data from an Illumina NextSeq sequencer.

[0637] Fig.71 One implementation of training data for training a neural network-based template generator 1512 is shown. In this implementation, the training data is obtained from a dense flow cell that produces data with STORM probe images. In another implementation, the training data is obtained from a dense flow cell that produces data with fewer bridge amplification cycles.

[0638] Fig.72 It is a specific implementation of image registration using beads based on cluster center prediction of the neural network based template generator 1512.

[0639] Fig.73 One specific implementation of cluster statistics is shown for clusters identified by the neural network based template generator 1512. The cluster statistics include cluster size based on the number of contributing sub-pixels and GC content.

[0640] Fig.74 It shows how the ability of the neural network-based template generator 1512 to distinguish adjacent clusters improves when the number of initial sequencing cycles using the input image data 1702 increases from five to seven. For five sequencing cycles, a single cluster is identified by a single non-intersecting region of adjacent sub-pixels. For seven sequencing cycles, the single cluster is split into two adjacent clusters, each with their own non-intersecting region of adjacent sub-pixels.

[0641] Fig.75 Shown is the difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center.

[0642] Fig.76The performance of the neural network based template generator 1512 for additional detected clusters is depicted.

[0643] Fig.77 Different data sets used to train the neural network based template generator 1512 are shown.

[0644] Fig.78 The processing stages used by an RTA base caller for base calling are shown according to one specific implementation. Fig.78 Also shown are the processing stages used for base calling by the disclosed neural network-based base caller according to two specific implementations. Fig.78 As shown, the neural network-based base caller 1514 can simplify the base calling process by eliminating many processing stages used by the RTA base caller. This simplification improves the accuracy and scale of base calling. In a first specific implementation of the neural network-based base caller 1514, the neural network-based base caller uses the position / location information of the cluster center identified from the output of the neural network-based template generator 1512 to perform base calling. In a second specific implementation, the neural network-based base caller 1514 does not use the position / location information of the cluster center to perform base calling. When a patterned circulation cell is designed for cluster generation, a second specific implementation is used. The patterned circulation cell includes a nanowell that is accurately positioned relative to a known reference position and provides a pre-arranged cluster distribution on the patterned circulation cell. In other specific implementations, the neural network-based base caller 1514 performs base calling on the clusters generated on the random circulation cell.

[0645] Neural network-based base calling

[0646] We now discuss neural network-based base calling, where a neural network is trained to map sequencing images to base calls. This discussion will proceed as follows. First, the input to the neural network is described. Then, the structure and form of the neural network is described. Finally, the output of the neural network is described.

[0647] enter

[0648] Fig.79 One specific implementation of base calling using neural network 7906 is illustrated.

[0649] Main input: image channel

[0650] The primary input to the neural network 7906 is image data 7902. The image data 7902 is derived from a sequencing image 108 generated by the sequencer 102 during a sequencing run. In one implementation, the image data 7902 includes n×n image patches extracted from the sequencing image 102, where n is any number in the range of 1 to 10,000. The sequencing run generates m images per sequencing cycle for the corresponding m image channels, and image patches are extracted from each of the m images to prepare image data for a particular sequencing cycle. In different implementations such as four-channel chemistry, two-channel chemistry, and single-channel chemistry, m is 4 or 2. In other implementations, m is 1, 3, or greater than 4. In some implementations, the image data 7902 is in the optical pixel domain, and in other implementations, in the upsampled sub-pixel domain.

[0651] Image data 7902 includes data for multiple sequencing cycles (e.g., a current sequencing cycle, one or more previous sequencing cycles, and one or more subsequent sequencing cycles). In one implementation, image data 7902 includes data for three sequencing cycles, such that data for pending base calls for the current (time t) sequencing cycle are accompanied by (i) data for the left flank / context / previous / previous / before (time t-1) sequencing cycle and (ii) data for the right flank / context / next / subsequent / after (time t+1) sequencing cycle. In other implementations, image data 7902 includes data for a single sequencing cycle.

[0652] Image data 7902 depicts the intensity emission of one or more clusters and their surrounding background. In one specific implementation, when base calling is to be performed on a single target cluster, image patches are extracted from sequencing image 108 in a manner such that each image patch contains the center of the target cluster in its center pixel, a concept referred to herein as "target cluster-centered patch extraction."

[0653] The image data 7902 is encoded in the input data 7904 using intensity channels (also referred to as image channels). For each of the m images obtained from the sequencer 102 for a particular sequencing cycle, its intensity data is encoded using a separate image channel. For example, consider a sequencing run using a dual-channel chemistry that produces a red image and a green image in each sequencing cycle, then the input data 7904 includes (i) a first red image channel having n×n pixels, the first red image channel depicting the intensity emission captured in the red image of the one or more clusters and their surrounding background; and (ii) a second green imaging channel having n×n pixels, the second green image channel depicting the intensity emission captured in the green image of the one or more clusters and their surrounding background.

[0654] Supplementary input: distance channel

[0655] The image data 7902 is accompanied by supplementary distance data (also referred to as a distance channel). The distance channel provides an additive bias that is incorporated into the feature map generated from the image channel. The additive bias contributes to the accuracy of base calling because it is based on the distance from the pixel center to the cluster center encoded pixel by pixel in the distance channel.

[0656] In a "single target cluster" base calling implementation, for each image channel (image patch) in input data 7904, a supplemental distance channel identifies the distance of the center of its pixels from the center of the target cluster containing its center pixel and for which base calling is to be performed. The distance channel thereby indicates the respective distances of the pixels of the image patch from the center pixel of the image patch.

[0657] In a "multi-cluster" base calling implementation, for each image channel (image patch) in the input data 7904, a supplemental distance channel identifies the center-to-center distance of each pixel from the nearest one of these clusters, which is selected based on the center-to-center distance between the pixel and each of these clusters.

[0658] In a "multi-cluster shape-based" base calling implementation, for each image channel (image patch) in the input data 7904, a supplemental distance channel identifies the center-to-center distance of each cluster pixel from an assigned cluster, which is selected based on classifying each cluster pixel into only one cluster.

[0659] Supplementary input: Scaling channel

[0660] The image data 7902 is accompanied by supplemental scaling data (also referred to as scaling channels) that account for different cluster sizes and non-uniform lighting conditions. The scaling channels also provide an additive bias that is incorporated into the feature map generated from the image channels. The additive bias contributes to the accuracy of base calling because it is based on the average intensity of the center cluster pixels encoded pixel by pixel in the scaling channels.

[0661] Supplementary input: cluster center coordinates

[0662] In some implementations, location / position information 7916 (e.g., xy coordinates) of cluster centers identified from the output of the neural network-based template generator 1512 is fed to the neural network 7906 as a supplemental input.

[0663] Supplementary input: cluster affiliation information

[0664] In some implementations, the neural network 7906 receives as a supplementary input cluster attribution information that classifies pixels or sub-pixels into: background pixels or sub-pixels, cluster center pixels or sub-pixels, and cluster / cluster interior pixels or sub-pixels that depict / contribute to / belong to the same cluster. In other implementations, an attenuation map, a binary map, and / or a ternary map, or variations thereof, are fed as supplementary inputs to the neural network 7906.

[0665] Preprocessing: Strength Modification

[0666] In some implementations, the input data 7904 does not include a range channel, and instead the neural network 7906 receives as input modified image data that is modified according to the output (i.e., attenuation map, binary map, and / or ternary map) of the neural network-based template generator 1512. In such implementations, the intensity of the image data 7902 is modified to account for the absence of a range channel.

[0667] In other embodiments, the image data 7902 is subjected to one or more lossless transform operations (e.g., convolution, deconvolution, Fourier transform), and the resulting modified image data is fed as input to the neural network 7906.

[0668] Network structure and form

[0669] The neural network 7906 is also referred to herein as a "neural network-based base caller" 1514. In one implementation, the neural network-based base caller 1514 is a multi-layer perceptron. In another implementation, the neural network-based base caller 1514 is a feed-forward neural network. In yet another implementation, the neural network-based base caller 1514 is a fully connected neural network. In another implementation, the neural network-based base caller 1514 is a fully convolutional neural network. In yet another implementation, the neural network-based base caller 1514 is a semantic segmentation neural network.

[0670] In one implementation, the neural network based base caller 1514 is a convolutional neural network (CNN) having multiple convolutional layers. In another implementation, the neural network based base caller is a recurrent neural network (RNN), such as a long short-term memory network (LSTM), a bidirectional LSTM (Bi-LSTM), or a gated recurrent unit (GRU). In yet another implementation, the neural network based base caller includes both a CNN and a RNN.

[0671] In other implementations, the neural network based base caller 1514 may use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or atrous convolution, transposed convolution, depthwise separable convolution, pointwise convolution, 1×1 convolution, grouped convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It may use one or more loss functions, such as logistic regression / logarithmic loss, multi-class cross entropy / softmax loss, binary cross entropy loss, mean square error loss, L1 loss, L2 loss, smooth L1 loss, and Huber loss. It may use any parallelism, efficiency, and compression scheme, such as TFRecords, compressed encoding (e.g., PNG), sharpening, parallel calling of map transformations, batching, prefetching, model parallelism, data parallelism, and synchronous / asynchronous SGD. It may include upsampling layers, downsampling layers, recurrent connections, gate and gate memory units (such as LSTM or GRU), residual blocks, residual connections, high-speed connections, skip connections, peephole connections, activation functions (e.g., non-linear transformation functions such as rectified linear units (ReLU), leaky ReLU, exponential lining units (ELU), sigmoid and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout layers, pooling layers (e.g., max or average pooling), global average pooling layers, and attention mechanisms.

[0672] The neural network-based base caller 1514 processes the input data 7904 and generates an alternative representation 7908 of the input data 7904. The alternative representation 7908 is a convolutional representation in some implementations and a hidden representation in other implementations. The alternative representation 7908 is then processed by an output layer 7910 to generate an output 7912. The output 7912 is used to generate base calls as described below.

[0673] Output

[0674] In one implementation, the neural network-based base caller 1514 outputs base calls for a single target cluster for a particular sequencing cycle. In another implementation, the neural network-based base caller outputs base calls for each of a plurality of target clusters for a particular sequencing cycle. In yet another implementation, the neural network-based base caller outputs base calls for each of a plurality of target clusters for each sequencing cycle in a plurality of sequencing cycles, thereby generating a base call sequence for each target cluster.

[0675] Distance channel calculation

[0676] We now discuss how to obtain appropriate position / location information (eg, xy coordinates) of the cluster center for use in calculating the distance value of the distance channel.

[0677] Reduce coordinates

[0678] Fig.80 It is a specific implementation of transforming the location / positioning information of the cluster centers identified from the output of the neural network-based template generator 1512 from the sub-pixel domain to the pixel domain.

[0679] The cluster center position / localization information is used for neural network-based base calling to at least: (i) construct input data by extracting an image patch from a sequencing image 108 that contains the center of a target cluster to be base called in its center pixel, (ii) construct a distance channel that identifies the distance of the center of a pixel of the image patch from the center of the target cluster containing its center pixel, and / or (iii) as a supplemental input 7916 to a neural network-based base caller 1514.

[0680] In some implementations, cluster center position / localization information is identified from the output of the neural network-based template generator 1512 at an upsampled sub-pixel resolution. However, in some implementations, the neural network-based base caller 1514 operates on image data at optical pixel resolution. Therefore, in one implementation, the cluster center position / localization information is transformed into the pixel domain by downscaling the coordinates of the cluster centers by the same upsampling factor used to upsample the image data fed as input to the neural network-based template generator 1512.

[0681] For example, consider that the image patch data fed as input to the neural network-based template generator 1512 is obtained by upsampling the image 108 by the upsampling factor f in some initial sequencing cycles. Then, in one specific implementation, the coordinates of the cluster centers 8002 generated by the neural network-based template generator 1512 and the post-processor 1814 and stored in the template / template image 8004 are divided by f (divider). These reduced cluster center coordinates are referred to herein as "reference cluster centers" 8008 and are stored in the template / template image 8004. In one specific implementation, the reduction is performed by a reducer 8006.

[0682] Coordinate transformation

[0683] Fig.81 is a specific implementation that uses a cycle-specific transform and an image channel-specific transform to derive the so-called "transformed cluster centers" 8104 from the reference cluster centers 8008. The motivation for doing so is first discussed.

[0684] The sequencing images acquired at different sequencing cycles are not aligned with respect to each other and have random translational offsets. This occurs because the motion accuracy of the motion stage of the sensor is limited, and also because the images acquired in different image / frequency channels have different optical paths and wavelengths. Therefore, there is an offset between the reference cluster center and the position / location of the cluster center in the sequencing image. This offset varies between images captured at different sequencing cycles and within images captured at the same sequencing cycle in different image channels.

[0685] To account for this shift, cycle-specific and image channel-specific transformations are applied to the reference cluster centers to produce corresponding transformed cluster centers for the image patches of each sequencing cycle. The cycle-specific and image channel-specific transformations are obtained through an image registration process that uses image correlation to determine a full six-parameter affine transformation (e.g., translation, rotation, scaling, shearing, right reflection, left reflection) or a Procrustes transformation (e.g., translation, rotation, scaling, optionally extended to aspect ratio), additional details of which can be found in Appendices 1, 2, 3, and 4.

[0686] For example, consider that the reference cluster center for the four cluster centers is (x 1 ,y 1 );(x 2 ,y 2 );(x 3 ,y 3 );(x 4 ,y 4 ), and the sequencing run uses a dual-channel chemistry where a red image and a green image are produced at each sequencing cycle. Then, for exemplary sequencing cycle 3, the cycle-specific and image-channel-specific transformations for the red image are The loop-specific and image channel-specific transforms for the green image are

[0687] Similarly, for exemplary sequencing cycle 9, the cycle-specific and image channel-specific transformations for the red image are The loop-specific and image channel-specific transforms for the green image are

[0688] Then, the transformed cluster centers of the red image of sequencing cycle 3 is by transforming Applied to the reference cluster center (x 1 ,y 1 );(x 2 ,y 2 );(x 3 ,y 3);(x 4 ,y 4 ) and the transformed cluster centers of the green image of sequencing cycle 3 is by transforming Applied to the reference cluster center (x 1 ,y 1 );(x 2 ,y 2 );(x 3 ,y 3 );(x 4 ,y 4 ) obtained.

[0689] Similarly, the transformed cluster centers of the red image of sequencing cycle 9 is by transforming Applied to the reference cluster center (x 1 ,y 1 );(x 2 ,y 2 );(x 3 ,y 3 );(x 4 ,y 4 ) and the transformed cluster centers of the green image of sequencing cycle 9 is by transforming Applied to the reference cluster center (x 1 ,y 1 );(x 2 ,y 2 );(x 3 ,y 3 );(x 4 ,y 4 ) obtained.

[0690] In one embodiment, these transformations are performed by transformer 8102.

[0691] The transformed cluster centers 8104 are stored in the template / template image 8004 and are respectively: (i) used for patch extraction from the corresponding sequencing image 108 (e.g., by a patch extractor 8106), (ii) used in the distance formula , used to calculate the distance channel for the corresponding image patch, and (iii) as a supplementary input to the neural network-based base caller 1514 for the corresponding sequencing cycle that was base called. In other specific implementations, different distance formulas can be used, such as distance squared, e^-distance, and e^-distance squared.

[0692] Image patch

[0693] Fig.82An image patch 8202 is shown that is part of the input data fed to the neural network-based base caller 1514. The input data includes a sequence of per-cycle image patch sets generated for a series of sequencing cycles of a sequencing run. Each per-cycle image patch set in the sequence has an image patch for a corresponding image channel in one or more image channels.

[0694] For example, consider that the sequencing run uses a dual-channel chemistry that produces a red image and a green image at each sequencing cycle, and that the input data includes data spanning a series of three sequencing cycles of the sequencing run: the current (time t) sequencing cycle at which a base call is to be made, the previous (time t-1) sequencing cycle, and the subsequent (time t+1) sequencing cycle.

[0695] The input data then includes the following sequence of image patch sets for each cycle: a current cycle image patch set, which has a current red image patch and a current green image patch extracted from the red sequencing image and the green sequencing image captured at the current sequencing cycle, respectively; a previous cycle image patch set, which has a previous red image patch extracted from the red sequencing image and the green sequencing image captured at the previous sequencing cycle, and a subsequent red image patch and a subsequent green image patch extracted from the red sequencing image and the green sequencing image captured at the subsequent sequencing cycle, respectively.

[0696] Each image patch may be n×n in size, where n may be any number in the range of 1 to 10,000. Each image patch may be in the optical pixel domain or in the upsampled sub-pixel domain. Fig.82 In the illustrated implementation, the extracted image patch 8202 has pixel intensity data for pixels covering / depicting the plurality of clusters 1-m and their surrounding background. Additionally, in the illustrated implementation, the image patch 8202 is extracted in such a way that the center of the target cluster being base called is contained in the center pixel of the image patch.

[0697] exist Fig.82 In , pixel centers are depicted by black rectangles and have integer position / location coordinates, and cluster centers are depicted by purple circles and have floating point position / location coordinates.

[0698] Distance calculation for a single target cluster

[0699] Fig.83One specific implementation of determining distance values ​​8302 for a distance channel when a single target cluster is being base called by the neural network based base caller 1514 is depicted. The center of the target cluster is contained in the center pixel of an image patch that is fed as input to the neural network based base caller 1514. The distance value is calculated pixel by pixel so that for each pixel the distance between its center and the center of the target cluster is determined. Thus, the distance value is calculated for each pixel in each of the image patches that are part of the input data.

[0700] Fig.83 Three distance values ​​d1, dc and dn for a particular image patch are shown. In one specific implementation, the following distance formula is used to calculate the distance value 8302: The distance formula operates based on the transformed cluster centers 8104. In other implementations, different distance formulas may be used, such as distance squared, e^-distance, and e^-distance squared.

[0701] In other embodiments, when the image patch is at an upsampled sub-pixel resolution, the distance value 8302 is calculated in the sub-pixel domain.

[0702] Therefore, in a single target cluster base calling implementation, the distance channel is calculated only relative to the target cluster being base called.

[0703] Fig.84 A specific implementation of pixel-by-pixel encoding 8402 of the distance value 8302 calculated between the pixel and the target cluster is shown. In a specific implementation, in the input data, the distance value 8302 as part of the distance channel is supplemented as "pixel distance data" for each corresponding image channel (image patch). Returning to the example of the red image and the green image generated by the sequencing cycle, the input data includes a red distance channel and a green distance channel, and the red image channel and the green image channel are supplemented as pixel distance data.

[0704] In other implementations, when the image patch is at an upsampled sub-pixel resolution, the range channel is encoded on a sub-pixel basis.

[0705] Distance calculation for multiple target clusters

[0706] Fig.85a Depicted is one specific implementation of determining distance values ​​8502 for a distance channel when a plurality of target clusters 1-m are simultaneously base called by a neural network based base caller 1514. The distance values ​​are calculated pixel by pixel such that for each pixel the distance between its center and the corresponding center of each of the plurality of clusters 1-m is determined, and the minimum distance value (in red) is assigned to the pixel.

[0707] Thus, the distance channel identifies the center-to-center distance of each pixel from the nearest cluster of the clusters, which is selected based on the center-to-center distance between the pixel and each of the clusters. In the illustrated implementation, Fig.85a The pixel center to cluster center distances for two pixels and four cluster centers are shown. Pixel 1 is closest to cluster 1, and pixel n is closest to cluster 3.

[0708] In one specific implementation, the distance value 8502 is calculated using the following distance formula: The distance formula operates based on the transformed cluster centers 8104. In other implementations, different distance formulas may be used, such as distance squared, e^-distance, and e^-distance squared.

[0709] In other implementations, when the image patch is at an upsampled sub-pixel resolution, the distance value 8502 is calculated in the sub-pixel domain.

[0710] Therefore, in multiple cluster base calling implementations, the distance channel is calculated relative to the closest cluster from the multiple clusters.

[0711] Figure 85b It is shown that for each target cluster in the target clusters 1-m, some nearest pixels are determined based on the distance 8504 (d1, d2, d23, d29, d24, d32, dn, d13, d14, etc.) from the pixel center to the nearest cluster center.

[0712] Fig.86 One implementation is shown in which the calculated minimum distance value between the pixel and the nearest cluster in the cluster is encoded pixel by pixel 8602. In other implementations, when the image patch is at an upsampled sub-pixel resolution, the distance channel is encoded on a sub-pixel basis.

[0713] Cluster shape-based distance calculation for multiple target clusters

[0714] Fig.87 One specific implementation of using pixel-to-cluster classification / attribution / categorization 8702 (referred to herein as "cluster shape data" or "cluster shape information") to determine cluster distance values ​​8802 for distance channels is illustrated when multiple target clusters 1-m are simultaneously base called by a neural network-based base caller 1514. First, the following is a brief overview of how the cluster shape data is generated.

[0715] As described above, the output of the neural network-based template generator 1512 is used to classify pixels into: background pixels, center pixels, and cluster / cluster internal pixels that depict / contribute to / belong to the same cluster. Regardless of the distance between the pixel center and the cluster center, this pixel-to-cluster classification information is used to attribute each pixel to only one cluster and is stored as cluster shape data.

[0716] exist Fig.87 In the specific implementation shown, background pixels are colored gray, pixels belonging to cluster 1 are colored yellow (cluster 1 pixels), pixels belonging to cluster 2 are colored green (cluster 2 pixels), pixels belonging to cluster 3 are colored red (cluster 3 pixels), and pixels belonging to cluster m are colored blue (cluster m pixels).

[0717] Fig.88 A specific implementation of using cluster shape data to calculate distance values ​​8802 is shown. First, the present invention explains why distance information calculated without considering cluster shape is prone to error. Then the present invention explains how cluster shape data overcomes this limitation.

[0718] In a "multiple clusters" base calling implementation that does not use cluster shape data ( Figure 85a to Figure 85b and Fig.86 ), the center-to-center distance value of a pixel is calculated relative to the nearest cluster from multiple clusters. Now, consider the case where a pixel belonging to cluster A is farther away from the center of cluster A but closer to the center of cluster B. In this case, without cluster shape data, the pixel is assigned a distance value calculated relative to cluster B (to which the pixel does not belong) rather than a distance value calculated relative to cluster A (to which the pixel does belong).

[0719] The "multi-cluster shape-based" base calling implementation avoids this by using true pixel-to-cluster mappings (as defined in the original image data and produced by the neural network-based template generator 1512).

[0720] With respect to pixels 34 and 35, a clear difference between the two implementations can be seen. Figure 85b In , the distance values ​​of pixels 34 and 35 are calculated relative to the nearest center of cluster 3, without considering the cluster shape data. However, in Fig.88 , based on the cluster shape data, the distance values ​​8802 of pixels 34 and 35 are calculated relative to cluster 2 (to which they actually belong).

[0721] exist Fig.88 In the example of FIG. 1 , cluster pixels depict cluster intensity and background pixels depict background intensity. The cluster distance value identifies the center-to-center distance of each cluster pixel from an assigned cluster in the cluster, which is selected based on classifying each cluster pixel into only one of the clusters. In some implementations, background pixels are assigned a predetermined background distance value, such as 0 or 0.1 or some other minimum value.

[0722] In one specific implementation, as described above, the cluster distance value 8802 is calculated using the following distance formula: The distance formula operates based on the transformed cluster centers 8104. In other implementations, different distance formulas may be used, such as distance squared, e^-distance, and e^-distance squared.

[0723] In other implementations, when the image patch is at an upsampled sub-pixel resolution, cluster distance values ​​8802 are calculated in the sub-pixel domain, and cluster and background attribution 8702 occurs on a sub-pixel basis.

[0724] Therefore, in a multiple cluster shape-based base calling implementation, a distance channel is calculated relative to an assigned cluster from the multiple clusters. The assigned cluster is selected based on classifying each cluster pixel into only one of the clusters according to a true pixel-to-cluster mapping defined in the original image data.

[0725] Fig.89 One implementation is shown in which the calculated distance value between the pixel and the assigned cluster is encoded pixel by pixel 8702. In other implementations, when the image patch is at an upsampled sub-pixel resolution, the distance channel is encoded on a sub-pixel basis.

[0726] Deep learning is a powerful machine learning technique that uses multi-layer neural networks. One particularly successful network architecture in the computer vision and image processing domains is the convolutional neural network (CNN), in which each layer performs a feed-forward convolution transformation from an input tensor (a multi-dimensional dense array similar to an image) to an output tensor of a different shape. CNNs are particularly well suited for image-like inputs due to the spatial coherence of images and the advent of general-purpose graphics processing units (GPUs), which train quickly on arrays up to 3D or 4D. Exploiting these image-like properties leads to superior empirical performance compared to other learning methods such as support vector machines (SVMs) or multi-layer perceptrons (MLPs).

[0727] We introduce a specialized architecture that enhances the standard CNN to process image data and supplements both distance and scale data. More details are as follows.

[0728] Specialized architecture

[0729] Fig.90 One specific implementation of a specialized architecture for a neural network-based base caller 1514 that isolates the processing of data for different sequencing cycles is shown. The motivation for using a specialized architecture is first described.

[0730] As described above, the neural network-based base caller 1514 processes data for the current sequencing cycle, one or more previous sequencing cycles, and one or more subsequent sequencing cycles. The data for the additional sequencing cycles provides a sequence-specific context. The neural network-based base caller 1514 learns the sequence-specific context during training and base calls the sequence-specific context. In addition, the data for the previous sequencing cycle and the subsequent sequencing cycle provide a second-order contribution to the pre-phasing and phasing signals for the current sequencing cycle.

[0731] Spatial Convolutional Layer

[0732] However, as described above, images captured at different sequencing cycles and in different image channels are misaligned with respect to each other and have residual registration errors. To account for this misalignment, the specialized architecture includes a spatial convolutional layer that does not mix information between sequencing cycles and only mixes information within a sequencing cycle.

[0733] The spatial convolution layer uses a so-called "isolated convolution" that achieves isolation by processing the data of each sequencing cycle in multiple sequencing cycles independently via a "dedicated non-shared" convolution sequence. The isolated convolution convolves the data and resulting feature maps of only a given sequencing cycle (i.e., within the cycle), and not the data and resulting feature maps of any other sequencing cycles.

[0734] For example, consider that the input data includes (i) current data of the current (time t) sequencing cycle for which base detection is to be performed, (ii) previous data of the previous (time t-1) sequencing cycle, and (iii) subsequent data of the previous (time t+1) sequencing cycle. The specialized architecture then initiates three separate data processing pipelines (or convolution pipelines), namely the current data processing pipeline, the previous data processing pipeline, and the subsequent data processing pipeline. The current data processing pipeline receives the current data of the current (time t) sequencing cycle as input, and independently processes the current data through multiple spatial convolution layers to produce a so-called "current spatial convolution representation" as the output of the final spatial convolution layer. The previous data processing pipeline receives the previous data of the previous (time t-1) sequencing cycle as input, and independently processes the previous data through multiple spatial convolution layers to produce a so-called "previous spatial convolution representation" as the output of the final spatial convolution layer. The subsequent data processing pipeline receives subsequent data of a subsequent (time t+1) sequencing cycle as input, and independently processes the subsequent data through multiple spatial convolutional layers to produce a so-called "subsequent spatial convolutional representation" as the output of the final spatial convolutional layer.

[0735] In some implementations, the current processing pipeline, the previous processing pipeline, and the subsequent processing pipeline are executed synchronously.

[0736] In some implementations, the spatial convolutional layer is part of a spatial convolutional network (or subnetwork) within a specialized architecture.

[0737] Temporal Convolutional Layer

[0738] The neural network-based base caller 1514 also includes a temporal convolutional layer that mixes information between sequencing cycles (i.e., inter-cycle). The temporal convolutional layer receives its input from the spatial convolutional network and operates on the spatial convolutional representation produced by the final spatial convolutional layer of the corresponding data processing pipeline.

[0739] The freedom of inter-cycle operability of temporal convolutional layers stems from the fact that the misalignment property present in the image data fed as input to the spatial convolutional network is cleaned from the spatial convolutional representation by the cascade of isolated convolutions performed by the sequence of spatial convolutional layers.

[0740] The temporal convolution layer uses so-called "combined convolutions" that convolve the input channels in subsequent inputs group by group on a sliding window basis. In one specific implementation, these subsequent inputs are subsequent outputs produced by a previous spatial convolution layer or a previous temporal convolution layer.

[0741] In some implementations, the temporal convolutional layer is part of a temporal convolutional network (or subnetwork) within a specialized architecture. The temporal convolutional network receives its input from the spatial convolutional network. In one implementation, a first temporal convolutional layer of the temporal convolutional network combines spatial convolutional representations between sequencing cycles group by group. In another implementation, a subsequent temporal convolutional layer of the temporal convolutional network combines subsequent outputs of previous temporal convolutional layers.

[0742] The output of the final temporal convolutional layer is fed to an output layer that produces outputs. The outputs are used to base call one or more clusters at one or more sequencing cycles.

[0743] Below is a more detailed discussion of isolated and combined convolutions.

[0744] Isolate Convolution

[0745] During the forward pass, the specialized architecture processes information from multiple inputs in two stages. In the first stage, isolated convolutions are used to prevent information mixing between inputs. In the second stage, combined convolutions are used to mix information between inputs. The results from the second stage are used to make a single inference for the multiple inputs.

[0746] This is different from batch mode techniques where a convolutional layer processes multiple inputs in a batch simultaneously and makes a corresponding inference for each input in the batch. In contrast, a specialized architecture maps the multiple inputs to the single inference. The single inference may include more than one prediction, such as a classification score for each of the four bases (A, C, T, and G).

[0747] In one implementation, the inputs are time ordered such that each input is generated at a different time step and has multiple input channels. For example, the multiple inputs may include the following three inputs: a current input generated by a current sequencing cycle at time step (t), a previous input generated by a previous sequencing cycle at time step (t-1), and a subsequent input generated by a subsequent sequencing cycle at time step (t+1). In another implementation, each input is derived from a current output, a previous output, and a subsequent output generated by one or more previous convolutional layers, respectively, and includes k feature maps.

[0748] In one implementation, each input may include the following five input channels: a red image channel (red), a red range channel (yellow), a green image channel (green), a green range channel (purple), and a scale channel (blue). In another implementation, each input may include k feature maps produced by a previous convolutional layer, and each feature map is considered an input channel.

[0749] Fig.91 One implementation of isolated convolution is depicted. Isolated convolution processes multiple inputs by synchronously applying a convolution filter once to each input. With isolated convolution, the convolution filters combine input channels in the same input and do not combine input channels in different inputs. In one implementation, the same convolution filter is synchronously applied to each input. In another implementation, a different convolution filter is synchronously applied to each input. In some implementations, each spatial convolution layer includes a set of k convolution filters, where each convolution filter is synchronously applied to each input.

[0750] Combined convolution

[0751] Combinatorial convolution mixes information between different inputs by grouping corresponding input channels of different inputs and applying convolution filters to each group. The grouping of these corresponding input channels and the application of convolution filters occur on a sliding window basis. In this context, the window spans two or more subsequent input channels, which represents, for example, the output of two subsequent sequencing cycles. Since the window is a sliding window, most of the input channels are used in two or more windows.

[0752] In some implementations, the different inputs originate from an output sequence produced by a previous spatial convolutional layer or a previous temporal convolutional layer. In the output sequence, these different inputs are arranged as subsequent outputs and are therefore considered as subsequent inputs by a subsequent temporal convolutional layer. Then, in the subsequent temporal convolutional layer, these combined convolutions apply convolution filters to corresponding input channel groups in these subsequent inputs.

[0753] In one implementation, these subsequent inputs have a time order such that the current input is generated by the current sequencing cycle at time step (t), the previous input is generated by the previous sequencing cycle at time step (t-1), and the subsequent input is generated by the subsequent sequencing cycle at time step (t+1). In another implementation, each subsequent input is derived from the current output, the previous output, and the subsequent output generated by one or more previous convolutional layers, respectively, and includes k feature maps.

[0754] In one implementation, each input may include the following five input channels: a red image channel (red), a red range channel (yellow), a green image channel (green), a green range channel (purple), and a scale channel (blue). In another implementation, each input may include k feature maps produced by a previous convolutional layer, and each feature map is considered an input channel.

[0755] The depth B of the convolution filter depends on the number of subsequent inputs whose corresponding input channels are convolved by the convolution filter group by group on a sliding window basis. In other words, the depth B is equal to the number of subsequent inputs and the group size in each sliding window.

[0756] exist Fig.92a In , corresponding input channels from two subsequent inputs are combined in each sliding window, and thus B=2. Figure 92b , the corresponding input channels from three subsequent inputs are combined in each sliding window, and thus B=3.

[0757] In one implementation, the sliding windows share the same convolutional filter. In another implementation, a different convolutional filter is used for each sliding window. In some implementations, each temporal convolutional layer includes a set of k convolutional filters, where each convolutional filter is applied to subsequent inputs on a sliding window basis.

[0758] Filter Bank

[0759] Fig.93 One specific implementation of the convolutional layers of a neural network based base caller 1514 is shown, where each convolutional layer has a set of convolutional filters. Fig.93In FIG. 5 , five convolutional layers are shown, each of which has a set of 64 convolutional filters. In some implementations, each spatial convolutional layer has a set of k convolutional filters, where k can be any number, such as 1, 2, 8, 64, 128, 256, etc. In some implementations, each temporal convolutional layer has a set of k convolutional filters, where k can be any number, such as 1, 2, 8, 64, 128, 256, etc.

[0760] Now let's discuss the supplementary scaling pass and how it is calculated.

[0761] Zoom Channel

[0762] Fig.94 Two configurations of a scaling channel that complements the image channel are depicted. The scaling channel is encoded pixel by pixel in the input data fed to the neural network-based base detector 1514. Different cluster sizes and uneven lighting conditions cause a wide range of cluster intensities to be extracted. The additive bias provided by the scaling channel makes the cluster intensities similar between clusters. In other specific implementations, the scaling channel is encoded subpixel by subpixel when the image patch is at an upsampled subpixel resolution.

[0763] When base calling is being performed on a single target cluster, the scaling channels assign the same scaling value to all pixels. When base calling is being performed on multiple target clusters simultaneously, the scaling channels assign different scaling values ​​to groups of pixels based on the cluster shape data.

[0764] The scaling channel 9410 has the same scaling value (s1) for all pixels. The scaling value (s1) is based on the average intensity of the center pixel containing the center of the target cluster. In one embodiment, the average intensity is calculated by averaging the intensity values ​​of the center pixel observed during two or more previous sequencing cycles, which produced A and T base calls for the target cluster.

[0765] Based on the cluster shape data, scaling channel 9408 has different scaling values ​​(s1, s2, s3, sm) for the corresponding pixel groups belonging to the corresponding cluster. Each pixel group includes a center cluster pixel, and the center cluster pixel includes the center of the corresponding cluster. The scaling value for a specific pixel group is based on the average intensity of its center cluster pixel. In a specific implementation, the average intensity is calculated by averaging the intensity values ​​of the center cluster pixels observed during two or more previous sequencing cycles, and these two or more previous sequencing cycles produce A and T base detections of the corresponding cluster.

[0766] In some implementations, background pixels are assigned a background scaling value (sb), which can be 0 or 0.1 or some other minimum value.

[0767] In one implementation, the scaled channels 9406 and their scaled values ​​are determined by an intensity scaler 9404. The intensity scaler 9404 uses cluster intensity data 9402 from previous sequencing cycles to calculate the average intensity.

[0768] In other embodiments, the supplemental scaling channel may be provided as an input in different ways (such as before or at the last layer of the neural network based base caller 1514, before or at one or more intermediate layers of the neural network based base caller 1514), and provided as a single value, rather than encoding the supplemental scaling channel pixel by pixel to match the image size.

[0769] Now let's discuss the input data fed to the neural network based base caller 1514.

[0770] Input data: image channel, distance channel and scale channel

[0771] Fig.95a A specific implementation of input data 9500 for a single sequencing cycle that produces a red image and a green image is shown. The input data 9500 includes the following:

[0772] • Red intensity data 9502 (red) for pixels in the image patch extracted from the red image. The red intensity data 9502 is encoded in the red image channel.

[0773] Red distance data 9504 (yellow) that supplements the red intensity data 9502 pixel by pixel.

[0774] Red range data 9504 is encoded in the red range channel.

[0775] Green intensity data 9506 (green) for pixels in the image patch extracted from the green image. The green intensity data 9506 is encoded in the green image channel.

[0776] Green distance data 9508 (purple) that complements the green intensity data 9506 pixel by pixel.

[0777] Green range data 9508 is encoded in the green range channel.

[0778] • Scaled data 9510 (blue) that complements the red intensity data 9502 and the green intensity data 9506 pixel by pixel. The scaled data 9510 is encoded in the scale channel.

[0779] In other implementations, the input data may include fewer or greater numbers of image channels and supplemental distance channels. In one example, for a sequencing run using four-channel chemistry, the input data includes four image channels and four supplemental distance channels for each sequencing cycle.

[0780] We now discuss how the distance channel and the scaling channel contribute to base calling accuracy.

[0781] Additive Bias

[0782] Fig.95b A specific implementation of a distance channel that provides an additive bias that is incorporated into the feature map generated from the image channel is illustrated. The additive bias contributes to the accuracy of base calling because it is based on the distance from the pixel center to the cluster center encoded pixel by pixel in the distance channel.

[0783] On average, about 3×3 pixels comprise a cluster. It is expected that the density at the center of a cluster is higher than at the edge, because the cluster grows outward from a substantially central location. Peripheral cluster pixels may contain conflicting signals from nearby clusters. Therefore, the center cluster pixels are considered to be the maximum intensity region and serve as a beacon to reliably identify the cluster.

[0784] The pixels of the image patch depict the intensity emissions of multiple clusters (e.g., 10 to 200 clusters) and their surrounding background. Additional clusters contain information from wider radii and contribute to base call predictions by discerning the underlying base whose intensity emission is depicted in the image patch. In other words, the intensity emissions from a set of clusters cumulatively create an intensity pattern that can be assigned to a discrete base (A, C, T, or G).

[0785] We observed that explicitly communicating the distance of each pixel from the cluster center in the supplementary distance channels to the convolutional filters resulted in higher base calling accuracy. The distance channels communicate to the convolutional filters which pixels contain the cluster center and which pixels are further away from the cluster center. The convolutional filters use this information to assign sequencing signals to their appropriate source clusters by focusing on (a) the center cluster pixels, their neighboring pixels, and the feature maps derived from them rather than (b) the peripheral cluster pixels, background pixels, and the feature maps derived from them. In one example of this focus, the distance channels provide a positive additive bias that is incorporated in the feature map produced by (a), but provide a negative additive bias that is incorporated in the feature map produced by (b).

[0786] These range channels have the same dimensionality as the image channels. This allows the convolutional filters to evaluate the image channels and range channels separately within the local receptive field and to combine these evaluations coherently.

[0787] When base calling is being performed on a single target cluster, these range channels identify only one center cluster pixel at the center of the image patch. When base calling is being performed on multiple target clusters simultaneously, these range channels identify multiple center cluster pixels distributed across the image patch.

[0788] The "single cluster" distance channel is applied to an image patch that contains in its center pixel the center of a single target cluster to be base called. The single cluster distance channel includes the center-to-center distance of each pixel in the image patch to the single target cluster. In this implementation, the image patch also includes additional clusters adjacent to the single target cluster, but base calling is not performed on these additional clusters.

[0789] The "multiple clusters" distance channel is applied to image patches that contain the centers of multiple target clusters to be base called in their corresponding center cluster pixels. The multiple clusters distance channel includes the center-to-center distance of each pixel in the image patch to the nearest cluster in the multiple target clusters. This has the possibility of measuring the center-to-center distance of the wrong cluster, but the probability of this is low.

[0790] The "multiple cluster shape-based" distance channel is applied to an image patch that contains the centers of multiple target clusters to be base called in its corresponding center cluster pixel and for which the pixel-to-cluster affiliation information is known. The multiple cluster distance channel includes the center-to-center distance of each cluster pixel in the image patch to the cluster of the multiple target clusters to which it belongs or is attributed. Background pixels may be marked as background instead of being given a calculated distance.

[0791] Fig.95b A specific implementation of a scale channel that provides an additive bias is also illustrated, which is incorporated into the feature map generated from the image channel. The additive bias contributes to the accuracy of base calling because it is based on the average intensity of the center cluster pixels encoded pixel by pixel in the scale channel. The discussion of additive bias in the context of the range channel is similarly applicable to the scale channel.

[0792] Example of Additive Bias

[0793] Fig.95b An example of how additive biases can be derived from the range channel and the scale channel and incorporated into the feature map generated from the image channel is also shown.

[0794] exist Fig.95b, convolution filter i 9514 evaluates a local receptive field 9512 (magenta) across two image channels 9502 and 9506, two range channels 9504 and 9508, and a scale channel 9510. Because these range channels and scale channels are encoded separately, additive biasing occurs when the intermediate outputs 9516a-9516e (plus bias 9516f) of each of the channel-specific convolution kernels (or feature detectors) 9516a-9516e are accumulated channel by channel 9518 as the final output / feature map element 9520 of the local receptive field 9512. In this example, the additive biases provided by the two range channels 9504 and 9508 are intermediate outputs 9516b and 9516d, respectively. The additive bias provided by the scale channel 9510 is intermediate output 9516e.

[0795] This additive bias guides the feature map compilation process by placing more emphasis on those features in the image channels that are considered more important and reliable for base calls (i.e., the pixel intensities of the center cluster pixels and their neighbors). During training, the weights of the convolution kernels are updated based on the back-propagation of gradients calculated from comparisons with the ground truth base calls to produce stronger activations for the center cluster pixels and their neighbors.

[0796] For example, considering that a pixel in a set of neighboring pixels covered by the local receptive field 9512 contains the cluster center, the distance channels 9504 and 9508 reflect the proximity of the pixel to the cluster center. Therefore, when the intensity intermediate outputs 9516a and 9516c are combined with the distance channel additive biases 9516b and 9516d at the channel-by-channel accumulation 9518, the result is a positive biased convolution representation 9520 of the pixel.

[0797] In contrast, if the pixels covered by the local receptive field 9512 are not close to the cluster center, the distance channels 9504 and 9508 reflect their separation from the cluster center. Therefore, when the intensity intermediate outputs 9516a and 9516c are combined with the distance channel additive biases 9516b and 9516d at the channel-by-channel accumulation 9518, the result is a negatively biased convolutional representation 9520 of the pixel.

[0798] Similarly, the scaled channel additive bias 9516e derived from the scaled channel 9510 may cause the convolution representation 9520 of the pixel to be positively or negatively biased.

[0799] For the sake of clarity, Fig.95bA single convolution filter i 9514 is shown applied to input data 9500 for a single sequencing cycle. Those skilled in the art will appreciate that the discussion can be extended to multiple convolution filters (e.g., a filter bank having k filters, where k can be 8, 16, 32, 64, 128, 256, etc.), multiple convolution layers (e.g., multiple spatial convolution layers and temporal convolution layers), and multiple sequencing cycles (e.g., t, t+1, t-1).

[0800] In other implementations, these distance channels and scaling channels are not encoded separately, but are directly applied to the image channel to generate modulated pixel multiplications, because the distance channel and the scaling channel and the image channel have the same dimensions. In another implementation, the weights of the convolution kernel are determined based on the distance channel and the image channel so as to detect the most important features in the image channel during the element-by-element multiplication. In other implementations, these distance channels and scaling channels are not fed to the first layer, but are provided as auxiliary inputs to downstream layers and / or networks (e.g., to a fully connected network or a classification layer). In yet another implementation, these distance and scaling channels are fed to the first layer and re-fed to the downstream layers and / or networks (e.g., via residual connections).

[0801] The above discussion is for 2D input data with k input channels. Those skilled in the art will be aware of the extension to 3D input. In short, the volume input is a 4D tensor with dimensions k×l×w×h, where l is an additional dimension - length. Each individual kernel is a 4D tensor, and by sweeping through the 4D tensor, a 3D tensor is obtained (the channel dimension is collapsed because this dimension is not swept).

[0802] In other implementations, when the input data 9500 is at an upsampled sub-pixel resolution, these range channels and scale channels are encoded separately on a sub-pixel basis, and additive biasing occurs at the sub-pixel level.

[0803] Base calling using specialized schema and input data

[0804] We now discuss how to use specialized architectures and input data for neural network-based base calling.

[0805] Single cluster base calling

[0806] Fig.96a , Fig.96b and Fig.96cA specific implementation of base calling for a single target cluster is depicted. The specialized architecture processes input data for three sequencing cycles (i.e., the current (time t) sequencing cycle for which base calling is to be performed, the previous (time t-1) sequencing cycle, and the subsequent (time t+1) sequencing cycle) and generates a base call for the single target cluster at the current (time t) sequencing cycle.

[0807] Fig.96a and Fig.96b A spatial convolutional layer is shown. Fig.96c Temporal convolutional layers are shown along with some other non-convolutional layers. Fig.96a and Fig.96b In the figure, the vertical dashed lines define the spatial convolutional layers according to the feature maps, and the horizontal dashed lines define the three convolutional pipelines corresponding to the three sequencing cycles.

[0808] For each sequencing cycle, the input data consists of a tensor of dimension n×n×m (e.g., Fig.95a The input tensor 9500 in , where n represents the width and height of the square tensor, and m represents the number of input channels, so that the dimensions of the input data for the three loops are n×n×m×t.

[0809] Here, each tensor for each cycle contains the center of a single target cluster in the center pixel of its image channel. Each tensor also shows the intensity emission of a single target cluster, some neighboring clusters, and its surrounding background captured in each image channel at a particular sequencing cycle. Fig.96a , two exemplary image channels are depicted, namely a red image channel and a green image channel.

[0810] Each tensor for each loop also includes distance channels that complement the corresponding image channels (e.g., red range channel and green range channel). These distance channels identify the center-to-center distance of each pixel in the corresponding image channel to a single target cluster. Each tensor for each loop also includes a scaling channel that scales the intensity values ​​in each of the image channels on a pixel-by-pixel basis.

[0811] This specialized architecture has five spatial convolutional layers and two temporal convolutional layers. Each spatial convolutional layer uses a Separated convolution is applied by using a set of k convolutional filters, where j represents the width and height of the square filter, and Denotes its depth. Each temporal convolutional layer applies combined convolution using a set of k convolutional filters of dimension j×j×α, where j denotes the width and height of the square filter and α denotes its depth.

[0812] The specialized architecture has a pre-classification layer (e.g., a flattening layer and a dense layer) and an output layer (e.g., a softmax classification layer). The pre-classification layer prepares input for the output layer. The output layer produces base calls for a single target cluster at the current (time t) sequencing cycle.

[0813] The decreasing spatial dimension

[0814] Fig.96a , Fig.96b and Fig.96c Also shown are the resulting feature maps (convolutional representations or intermediate convolutional representations or convolutional features or activation maps) produced by the convolutional filters. Starting from the tensor for each cycle, the spatial dimensions of these resulting feature maps decrease by a constant step size from one convolutional layer to the next, a concept referred to herein as "continuously decreasing spatial dimensions". Fig.96a , Fig.96b and Fig.96c In , an exemplary constant step size of 2 is used for the decreasing spatial dimension.

[0815] The decreasing spatial dimension is represented by the following formula: "Current feature map spatial dimension = previous feature map spatial dimension - convolution filter spatial dimension + 1". This decreasing spatial dimension causes the convolution filter to gradually narrow the focus on the center cluster pixel and its neighboring pixels, and generate a feature map with features that capture the local dependencies between the center cluster pixel and its neighboring pixels. This in turn helps to accurately base call the cluster whose center is contained in the center cluster pixel.

[0816] The isolated convolutions of these five spatial convolutional layers prevent information mixing between the three sequencing cycles and maintain three separate convolutional pipelines.

[0817] The combined convolution of these two temporal convolutional layers mixes the information between the three sequencing cycles. The first temporal convolutional layer convolves the subsequent and current spatial convolutional representations generated by the final spatial convolutional layer for the subsequent and current sequencing cycles, respectively. This produces a first temporal output. The first temporal convolutional layer also convolves the current and previous spatial convolutional representations generated by the final spatial convolutional layer for the current and previous sequencing cycles, respectively. This produces a second temporal output. The second temporal convolutional layer convolves the first temporal output and the second temporal output and produces a final temporal output.

[0818] In some implementations, the final temporal output is fed to a flattening layer to produce a flattened output. The flattened output is then fed to a dense layer to produce a dense output. The dense output is processed by the output layer to produce a base call for a single target cluster at the current (time t) sequencing cycle.

[0819] In some implementations, the output layer generates the likelihood (classification scores) of bases that are A, C, T, and G incorporated into a single target cluster at the current sequencing cycle, and classifies the bases as A, C, T, or G based on these probabilities (e.g., selecting the base with the greatest likelihood, such as Fig.96a In such implementations, these likelihoods are the exponentially normalized scores produced by the softmax classification layer and sum to 1.

[0820] In some implementations, the output layer exports an output pair for a single target cluster. The output pair identifies the class labels of bases that are A, C, T, or G that are incorporated into the single target cluster at the current sequencing cycle, and base calls the single target cluster based on these class labels. In one implementation, class label 1,0 identifies the A base; class label 0,1 identifies the C base; class label 1,1 identifies the T base; and class label 0,0 identifies the G base. In another implementation, class label 1,1 identifies the A base; class label 0,1 identifies the C base; class label 0.5,0.5 identifies the T base; and class label 0,0 identifies the G base. In yet another implementation, class label 1,0 identifies the A base; class label 0,1 identifies the C base; class label 0.5,0.5 identifies the T base; and class label 0,0 identifies the G base. In yet another specific implementation, class label 1,2 identifies an A base; class label 0,1 identifies a C base; class label 1,1 identifies a T base; and class label 0,0 identifies a G base.

[0821] In some implementations, the output layer derives a class label for a single target cluster, the class label identifying the base that is A, C, T, or G incorporated into the single target cluster at the current sequencing cycle, and base calling is performed on the single target cluster based on the class label. In one implementation, class label 0.33 identifies the A base; class label 0.66 identifies the C base; class label 1 identifies the T base; and class label 0 identifies the G base. In another implementation, class label 0.50 identifies the A base; class label 0.75 identifies the C base; class label 1 identifies the T base; and class label 0.25 identifies the G base.

[0822] In some implementations, the output layer derives a single output value, compares the single output value to a range of class values ​​corresponding to bases A, C, T, and G, assigns the single output value to a specific class value range based on the comparison, and base calls a single target cluster based on the assignment. In one implementation, the single output value is derived using a sigmoid function, and the single output value is in the range of 0 to 1. In another implementation, a class value range of 0 to 0.25 represents an A base, a class value range of 0.25 to 0.50 represents a C base, a class value range of 0.50 to 0.75 represents a T base, and a class value range of 0.75 to 1 represents a G base.

[0823] Those skilled in the art will appreciate that in other specific implementations, the specialized architecture can process input data for fewer or more sequencing cycles, and can include fewer or more spatial convolutional layers and temporal convolutional layers. In addition, the dimensions of the input data, the tensors for each cycle in the input data, the convolutional filters, the resulting feature maps, and the outputs can be different. In addition, the number of convolutional filters in the convolutional layer can be different. It can use various padding and stride configurations. It can use different classification functions (e.g., sigmoid or regression), and may or may not include a fully connected layer. It can use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilation or dilation convolution, transposed convolution, depth-separable convolution, point-by-point convolution, 1×1 convolution, grouped convolution, flat convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It may use one or more loss functions, such as logistic regression / logarithmic loss, multi-class cross entropy / softmax loss, binary cross entropy loss, mean square error loss, L1 loss, L2 loss, smooth L1 loss, and Huber loss. It may use any parallelism, efficiency, and compression scheme, such as TFRecords, compressed encoding (e.g., PNG), sharpening, parallel detection of map conversion, batching, prefetching, model parallelism, data parallelism, and synchronous / asynchronous SGD. It may include upsampling layers, downsampling layers, recursive connections, gate and gate memory units (such as LSTM or GRU), residual blocks, residual connections, high-speed connections, skip connections, peephole connections, activation functions (e.g., nonlinear transformation functions such as rectified linear units (ReLU), leaky ReLU, exponential lining units (ELU), sigmoid, and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout layers, pooling layers (e.g., max or average pooling), global average pooling layers, and attention mechanisms.

[0824] Having described single cluster base calling, we now discuss multiple cluster base calling.

[0825] Multiple cluster base calling

[0826] Depending on the size of the input data and the cluster density on the flow cell, anywhere between one hundred thousand to three hundred thousand clusters are base called simultaneously on a per input basis by the neural network based base caller 1514. Extending this to data parallelism and / or model parallelism strategies implemented on a parallel processor, using batches or mini-batches of size ten results in one hundred to three million clusters being base called on a per batch basis or on a per mini-batch basis.

[0827] Depending on the sequencing configuration (e.g., cluster density, number of blocks on the flow cell), a block includes 20,000 to 300,000 clusters. In another specific implementation, Illumina's NovaSeq sequencer has up to four million clusters per block. Therefore, the sequencing image of the block (block image) can depict the intensity emission from 20,000 to 300,000 clusters and their surrounding background. Therefore, in one specific implementation, using input data that includes the entire block image results in simultaneous base calling of 300,000 clusters based on each input. In another specific implementation, using image patches of size 15×15 pixels in the input data results in simultaneous base calling of less than one hundred clusters based on each input. Those skilled in the art will appreciate that these numbers can vary depending on the sequencing configuration, parallelization strategy, details of the architecture (e.g., based on optimal architecture hyperparameters), an...

Claims

1. A system comprising: at least one processor; as well as a non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the system to: identifying one or more base calls of one or more analytes based on the sequencing images captured at the one or more sequencing cycles; providing input data derived from the sequencing images to a machine learning system; as well as The input data is processed by the machine learning system to generate one or more quality predictions for the one or more base calls.

2. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to generate one or more quality predictions for the one or more base calls by determining one or more quality scores for the one or more base calls.

3. The system of claim 1, further comprising instructions that, when executed by the at least one processor, cause the system to generate one or more quality predictions for the one or more base calls by: determining a quality score for base calls of bases incorporated into the target nucleic acid cluster at one sequencing cycle in the one or more sequencing cycles; or At one sequencing cycle of the one or more sequencing cycles, a quality score for base calls of bases incorporated into the plurality of nucleic acid clusters is determined.

4. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to generate one or more quality predictions for the one or more base calls by generating a continuous value identifying the quality of the one or more base calls.

5. The system of claim 1, further comprising instructions that, when executed by the at least one processor, cause the system to: Processing the input data by the machine learning system to generate one or more predicted quality indicators for the one or more base calls; and Based on the one or more predicted quality indicators, one or more quality predictions of the one or more base calls are generated.

6. The system of claim 5, further comprising instructions that, when executed by the at least one processor, cause the system to: generating the one or more predicted quality indicators by generating quality score probabilities for base calls in one or more base calls assigned individual quality scores; and The one or more quality predictions are generated for the one or more base calls by assigning a quality score to the base call from one of the individual quality scores based on the quality score likelihoods.

7. The system of claim 5, further comprising instructions that, when executed by the at least one processor, cause the system to: generating the one or more predicted quality indications by generating a first likelihood that a base call of the one or more base calls is of high quality, a second likelihood that the base call is of medium quality, and a third likelihood that the base call is of low quality; and One or more quality predictions for the one or more base calls are determined by assigning quality scores to the base calls based on the first likelihood, the second likelihood, and the third likelihood.

8. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to provide input data derived from a sequencing image by providing the machine learning system with a sequencing image depicting intensity emissions from a target nucleic acid cluster and one or more adjacent nucleic acid clusters at one sequencing cycle of the one or more sequencing cycles, the sequencing image.

9. The system of claim 1, further comprising instructions that, when executed by the at least one processor, cause the system to provide input data derived from the sequencing image by providing the machine learning system with the following information: Supplemental distance information identifying distances between pixels within the sequencing image; or Image channel information that identifies one or more image channels corresponding to the sequencing image.

10. The system of claim 1, further comprising instructions that, when executed by the at least one processor, cause the system to: providing one or more quality predictor values ​​for the one or more base calls to the machine learning system; and The one or more quality predictor values ​​are processed by the machine learning system to generate the one or more quality predictions further based on the one or more quality predictor values.

11. The system of claim 10, wherein the one or more quality predictor values ​​comprise one or more of: online overlap, purity, phasing, start5, hexamer score, motif accumulation, end value, near homopolymer, intensity decay, penultimate purity, signal overlap with background (SOWB), purity G adjustment for offset, peak height, peak width, peak position, relative peak position, peak height ratio, peak spacing ratio, or peak correspondence.

12. A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause a system to: identifying one or more base calls of one or more analytes based on the sequencing images captured at the one or more sequencing cycles; providing the machine learning system with input data derived from the sequencing images; as well as The input data is processed by the machine learning system to generate one or more quality predictions for the one or more base calls.

13. The non-transitory computer-readable medium of claim 12, further storing instructions that, when executed by the at least one processor, cause the system to generate one or more quality predictions for the one or more base calls by determining one or more quality scores for the one or more base calls.

14. The non-transitory computer-readable medium of claim 12, further storing instructions that, when executed by the at least one processor, cause the system to generate one or more quality predictions for the one or more base calls by generating a continuous value identifying the quality of the one or more base calls.

15. The non-transitory computer readable medium of claim 12, further storing instructions that, when executed by the at least one processor, cause the system to: Processing the input data by the machine learning system to generate quality score probabilities for base calls in one or more base calls assigned individual quality scores; and The one or more quality predictions are generated for the one or more base calls by assigning a quality score to the base call from one of the individual quality scores based on the quality score likelihoods.

16. The non-transitory computer-readable medium of claim 12, further storing instructions that, when executed by the at least one processor, cause the system to provide input data derived from a sequencing image by providing the machine learning system with a sequencing image depicting intensity emissions from a target nucleic acid cluster and one or more adjacent nucleic acid clusters at one sequencing cycle of the one or more sequencing cycles.

17. A computer-implemented method comprising: identifying one or more base calls of one or more analytes based on the sequencing images captured at the one or more sequencing cycles; providing the machine learning system with input data derived from the sequencing images; as well as The input data is processed by the machine learning system to generate one or more quality predictions for the one or more base calls.

18. The computer-implemented method of claim 17, further comprising: providing one or more quality predictor values ​​for the one or more base calls to the machine learning system; and The one or more quality predictor values ​​are processed by the machine learning system to generate the one or more quality predictions further based on the one or more quality predictor values.

19. A computer-implemented method according to claim 18, wherein the one or more quality predictor values ​​include one or more of the following: online overlap, purity, phasing, start5, hexamer score, motif accumulation, end value, near homopolymer, intensity decay, second to last purity, signal overlap with background (SOWB), purity G adjustment for offset, peak height, peak width, peak position, relative peak position, peak height ratio, peak spacing ratio or peak correspondence.

20. The computer-implemented method of claim 19, further comprising determining one or more base calls of the one or more analytes based on sequencing images captured at one or more sequencing cycles.

Images