Cluster map generation

Abstract

The disclosure relates to cluster map generation that processes a first input through a first neural network and produces a first output. The first input includes first image data derived from an image of an analyte and its surrounding background captured by the sequencing system in a sequencing run. The disclosed techniques process the first output by a post-processor and generate metadata regarding the analyte and its surrounding context. The disclosed techniques process a second input through a second neural network and produce a second output. The second input includes third image data derived by modifying the second image data based on the metadata. The second image data is derived from the image of the analyte and its surrounding background. The second output identifies base detection of one or more of the analytes at one or more sequencing cycles of the sequencing run.

Description

[0001] This application is a divisional application of the application with application number 202080004529.4, application date March 22, 2020, and invention name “Artificial Intelligence-Based Sequencing”.

[0002] Priority application

[0003] This patent application claims priority or the benefit of the following patent applications:

[0004] U.S. Provisional Patent Application No. 62 / 821,602, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-1 / IP-1693-PRV);

[0005] U.S. Provisional Patent Application No. 62 / 821,618, entitled “Artificial Intelligence-Based Generation of Sequencing Metadata,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-3 / IP-1741-PRV);

[0006] U.S. Provisional Patent Application No. 62 / 821,681, entitled “Artificial Intelligence-Based Base Calling,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-4 / IP-1744-PRV);

[0007] U.S. Provisional Patent Application No. 62 / 821,724, entitled “Artificial Intelligence-Based Quality Scoring,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-7 / IP-1747-PRV);

[0008] U.S. Provisional Patent Application No. 62 / 821,766, entitled “Artificial Intelligence-Based Sequencing,” filed on March 21, 2019 (Attorney Docket No. ILLM 1008-9 / IP-1752-PRV);

[0009] Netherlands Patent Application No. 2023310, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on June 14, 2019 (Attorney Docket No. ILLM1008-11 / IP-1693-NL);

[0010] Netherlands Patent Application No. 2023311, entitled “Artificial Intelligence-Based Generation of Sequencing Metadata,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-12 / IP-1741-NL);

[0011] Netherlands Patent Application No. 2023312, entitled “Artificial Intelligence-Based Base Calling,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-13 / IP-1744-NL);

[0012] Netherlands Patent Application No. 2023314, entitled “Artificial Intelligence-Based QualityScoring,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-14 / IP-1747-NL); and

[0013] Netherlands Patent Application No. 2023316, entitled “Artificial Intelligence-Based Sequencing,” filed on June 14, 2019 (Attorney Docket No. ILLM 1008-15 / IP-1752-NL).

[0014] U.S. non-provisional patent application No. 16 / 825,987, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-16 / IP-1693-US);

[0015] U.S. non-provisional patent application No. 16 / 825,991, entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-17 / IP-1741-US);

[0016] U.S. non-provisional patent application No. 16 / 826,126, entitled “Artificial Intelligence-Based Base Calling,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-18 / IP-1744-US);

[0017] U.S. non-provisional patent application No. 16 / 826,134, entitled “Artificial Intelligence-Based Quality Scoring,” filed on March 20, 2020 (Attorney Docket No. ILLM 1008-19 / IP-1747-US);

[0018] U.S. non-provisional patent application No. 16 / 826,168, entitled “Artificial Intelligence-Based Sequencing,” filed on March 21, 2020 (Attorney Docket No. ILLM 1008-20 / IP-1752-PRV);

[0019] concurrently filed PCT Patent Application No. PCT / US2020 / 024090 (Attorney Docket No. ILLM 1008-21 / IP-1693-PCT), entitled “Training Data Generation for Artificial Intelligence-Based Sequencing,” which was subsequently published as PCT Publication No. WO2020 / 191389;

[0020] concurrently filed PCT patent application No. PCT / US2020 / 024087 (attorney docket No. ILLM 1008-22 / IP-1741-PCT), entitled “Artificial Intelligence-Based Generation of Sequencing Metadata,” which was subsequently published as PCT Publication No. WO2020 / 205296;

[0021] concurrently filed PCT Patent Application No. PCT / US2020 / 024088 (Attorney Docket No. ILLM 1008-23 / IP-1744-PCT), entitled “Artificial Intelligence-Based Base Calling,” which was subsequently published as PCT Publication No. WO2020 / 191387; and

[0022] PCT patent application No. PCT / US2020 / 024091 (attorney docket number ILLM 1008-24 / IP-1747-PCT) entitled “Artificial Intelligence-Based Quality Scoring” was filed at the same time and was subsequently published as PCT publication No. WO 2020 / 191390.

[0023] These priority applications are hereby incorporated by reference as if fully set forth herein for all purposes.

[0024] Literature Incorporation

[0025] The following documents are incorporated by reference as if fully set forth herein for all purposes:

[0026] U.S. Provisional Patent Application No. 62 / 849,091, “Systems and Devices for Characterization and Performance Analysis of Pixel-Based Sequencing,” filed on May 16, 2019 (Attorney Docket No. ILLM 1011-1 / IP-1750-PRV);

[0027] U.S. Provisional Patent Application No. 62 / 849,132, entitled “Base Calling Using Convolutions,” filed on May 16, 2019 (Attorney Docket No. ILLM 1011-2 / IP-1750-PR2);

[0028] U.S. Provisional Patent Application No. 62 / 849,133, entitled “Base Calling Using Compact Convolutions,” filed on May 16, 2019 (Attorney Docket No. ILLM 1011-3 / IP-1750-PR3);

[0029] U.S. Provisional Patent Application No. 62 / 979,384, entitled “Artificial Intelligence-Based Base Calling of Index Sequences,” filed on February 20, 2020 (Attorney Docket No. ILLM 1015-1 / IP-1857-PRV);

[0030] U.S. Provisional Patent Application No. 62 / 979,414, entitled “Artificial Intelligence-Based Many-To-ManyBase Calling,” filed on February 20, 2020 (Attorney Docket No. ILLM 1016-1 / IP-1858-PRV);

[0031] U.S. Provisional Patent Application No. 62 / 979,385, entitled “Knowledge Distillation-Based Compression of Artificial Intelligence-Based Base Caller,” filed on February 20, 2020 (Attorney Docket No. ILLM 1017-1 / IP-1859-PRV);

[0032] U.S. Provisional Patent Application No. 62 / 979,412, entitled “Multi-Cycle Cluster Based Real Time Analysis System,” filed on February 20, 2020 (Attorney Docket No. ILLM 1020-1 / IP-1866-PRV);

[0033] U.S. Provisional Patent Application No. 62 / 979,411, entitled “Data Compression for Artificial Intelligence-Based Base Calling,” filed on February 20, 2020 (Attorney Docket No. ILLM 1029-1 / IP-1964-PRV);

[0034] U.S. Provisional Patent Application No. 62 / 979,399, entitled “Squeezing Layer for Artificial Intelligence-Based Base Calling,” filed on February 20, 2020 (Attorney Docket No. ILLM 1030-1 / IP-1982-PRV);

[0035] Liu P, Hemani A, Paul K, Weis C, Jung M, Wehn ​​N, “3D-Stacked Many-Core Architecture for Biological Sequence Analysis Problems,” Int J Parallel Prog, 2017, vol. 45(no. 6): 1420–1460.

[0036] Z. Wu, K. Hammad, R. Mittmann, S. Magierowski, E. Ghafar-Zadeh, and X. Zhong, “FPGA-Based DNA Basecalling Hardware Acceleration,” in Proc. IEEE 61st Int. Midwest Symp. Circuits Syst., August 2018, pp. 1098–1101.

[0037] Z. Wu, K. Hammad, E. Ghafar-Zadeh, and S. Magierowski, “FPGA-Accelerated 3rd Generation DNA Sequencing,” IEEE Transactions on Biomedical Circuits and Systems, Vol. 14, No. 1, February 2020, pp. 65–74.

[0038] Prabhakar et al., “Plasticine: A Reconfigurable Architecture for Parallel Patterns,” ISCA 17, June 24–28, 2017, Toronto, ON, Canada;

[0039] M.Lin, Q.Chen and S.Yan, "Network in Network", Proc.of ICLR, 2014;

[0040] L.Sifre, "Rigid-motion Scattering for Image Classification", Ph.D. thesis, 2014;

[0041] L. Sifre and S. Mallat, "Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination", Proc. of CVPR, 2013;

[0042] F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions", Proc. of CVPR, 2017;

[0043] X. Zhang, X. Zhou, M. Lin and J. Sun, "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," arXiv:1707.01083, 2017;

[0044] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition", Proc. of CVPR, 2016;

[0045] S. Xie, R. Girshick, P. Dollár, Z. Tu and K. He, "Aggregated Residual Transformations for Deep Neural Networks", Proc. of CVPR, 2017;

[0046] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, "Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications", arXiv:1704.04861, 2017;

[0047] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “MobileNetV2: InvertedResiduals and Linear Bottlenecks,” arXiv:1801.04381v3, 2018;

[0048] Z. Qin, Z. Zhang, X. Chen, and Y. Peng, “FD-MobileNet: Improved Mobile Net with aFast Downsampling Strategy,” arXiv:1802.03750,2018;

[0049] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam, “Rethinking Atrous Convolution for Semantic Image Segmentation,” CoRR, abs / 1706.05587, 2017;

[0050] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, and S. Guadarrama, “Speed / accuracy trade-offs for modern convolutionalobject detectors,” arXiv preprint arXiv:1611.10012,2016;

[0051] S.Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WAVENET:A GENERATIVE MODEL FOR RAW AUDIO”, arXiv:1609.03499,2016;

[0052] S. Arik, M., Chrzanowski, A., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., Sengupta, S., & Shoeybi, M. "DEEP VOICE: REAL-TIME NEURAL TEXT-TO-SPEECH", arXiv:1702.07825, 2017;

[0053] Yu, F., & Koltun, V. "MULTI-SCALE CONTEXT AGGREGATION BY DILATED CONVOLUTIONS", arXiv:1511.07122, 2016;

[0054] He, K., Zhang, X., Ren, S., & Sun, J. "DEEP RESIDUAL LEARNING FOR IMAGE RECOGNITION", arXiv:1512.03385, 2015;

[0055] Srivastava, R. K., Greff, K., & Schmidhuber, J. "HIGHWAY NETWORKS", arXiv:1505.00387, 2015;

[0056] Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. Q. "DENSELY CONNECTED CONVOLUTIONAL NETWORKS", arXiv:1608.06993, 2017;

[0057] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. "GOING DEEPER WITH CONVOLUTIONS", arXiv:1409.4842, 2014;

[0058] Ioffe, S., & Szegedy, C. "BATCH NORMALIZATION: ACCELERATING DEEP NETWORK TRAINING BY REDUCING INTERNAL COVARIATE SHIFT", arXiv:1502.03167, 2015;

[0059] JMWolterink, T.Leiner, MAViergever and I. "DILATED CONVOLUTIONALNEURAL NETWORKS FOR CARDIOVASCULAR MR SEGMENTATION IN CONGENITAL HEARTDISEASE", arXiv:1704.03669, 2017;

[0060] L.C. Piqueras, “AUTOREGRESSIVE MODEL BASED ON A DEEP CONVOLUTIONAL NEURAL NETWORK FOR AUDIO GENERATION,” Tampere University of Technology, 2016;

[0061] J. Wu, “Introduction to Convolutional Neural Networks”, Nanjing University, 2017;

[0062] "Illumina CMOS Chip and One-Channel SBS Chemistry", Illumina, Inc, 2018, page 2;

[0063] “skikit-image / peak.py at master”, GitHub, page 5, [retrieved on November 16, 2018], retrieved from the internet <URL: https: / / github.com / scikit-image / scikit-image / blob / master / skimage / feature / peak.py#L25 >

[0064] “3.3.9.11. Watershed and random walker for segmentation”, Scipy lecturenotes, page 2, [retrieved on November 13, 2018], retrieved from the internet<URL:http: / / scipy-lectures.org / packages / scikit-image / auto_examples / plot_segmentations.html> ;

[0065] Mordvintsev, Alexander and Revision, Abid K., "Image Segmentation with Watershed Algorithm", Version 43532856, 2013, p. 6, [retrieved on November 13, 2018], retrieved from the internet <URL:https: / / opencv-python-tutroals.readthedocs.io / en / latest / py_tutorials / py_imgproc / py_watershed / py_water shed.html> ;

[0066] Mzur, “Watershed.py,” October 25, 2017, p. 3, [retrieved November 13, 2018], retrieved from the internet<URL:https: / / github.com / mzur / watershed / blob / master / Watershed.py> ;

[0067] Thakur, Pratibha, et al., “A Survey of Image Segmentation Techniques,” International Journal of Research in Computer Applications and Robotics, Vol. 2, No. 4, April 2014, pp. 158–165.

[0068] Long, Jonathan, et al., “Fully Convolutional Networks for Semantic Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, vol. 4, April 1, 2017, p. 10.

[0069] Ronneberger, Olaf, et al., “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, May 18, 2015, p. 8;

[0070] Xie, W. et al., “Microscopy cell counting and detection with fully convolutional regression networks,” Computer methods in biomechanics and biomedical engineering: Imaging & Visualization, Vol. 6 (Vol. 3), pp. 283-292, 2018.

[0071] Xie, Yuanpu et al., "Beyond classification: structured regression for robust cell detection using convolutional neural network", International Conference on Medical Image Computing and Computer-Assisted Intervention, October 12, 2015, page 12;

[0072] Snuverink, IAF, “Deep Learning for Pixelwise Classification of Hyperspectral Images,” Master of Science Thesis, Delft University of Technology, November 23, 2017, p. 19.

[0073] Shevchenko, A., "Keras weighted categorical_crossentropy", Page 1, [Retrieved January 15, 2019], Retrieved from the internet<URL:https: / / gist.github.com / skeeet / cad06d584548fb45eece1d4e28cfa98b> ;

[0074] van den Assem, DCF, “Predicting periodic and chaotic signals using Wavenets,” Master of Science thesis, Delft University of Technology, August 18, 2017, pp. 3–38;

[0075] I.J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Convolutional Networks,” in Deep Learning, MIT Press, 2016; and

[0076] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, and G. Wang, “RECENT ADVANCES IN CONVOLUTIONAL NEURAL NETWORKS,” arXiv:1512.07108, 2017. Technical Field

[0077] The technology disclosed herein relates to artificial intelligence-type computers and digital data processing systems, as well as corresponding data processing methods and products for simulating intelligence (i.e., knowledge-based systems, inference systems, and knowledge acquisition systems). The technology also includes systems for inferring uncertainty (e.g., fuzzy logic systems), adaptive systems, machine learning systems, and artificial neural networks. Specifically, the technology disclosed herein relates to the use of deep neural networks, such as deep convolutional neural networks, for analyzing data. Background Art

[0078] The subject matter discussed in this section should not be considered prior art simply because it is mentioned in this section. Similarly, problems mentioned in this section or associated with the subject matter provided as background technology should not be considered to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which themselves may also correspond to specific implementations of the claimed technology.

[0079] A deep neural network is an artificial neural network that uses multiple layers of nonlinear and complex transformations to continuously model high-level features. Deep neural networks provide feedback via backpropagation, which uses the difference between observed and predicted outputs to adjust parameters. Deep neural networks have evolved with the availability of large training datasets, the power of parallel and distributed computing, and sophisticated training algorithms. They have driven significant advances in many fields, such as computer vision, speech recognition, and natural language processing.

[0080] Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are components of deep neural networks. Convolutional neural networks are particularly successful in image recognition with an architecture comprising convolutional layers, nonlinear layers, and pooling layers. Recurrent neural networks are designed to exploit the sequential information of input data and have cyclic connections between building blocks such as perceptrons, long short-term memory units, and gated recurrent units. In addition, many other emerging deep neural networks for limited contexts have been proposed, such as deep spatiotemporal neural networks, multidimensional recurrent neural networks, and convolutional autoencoders.

[0081] The goal of training a deep neural network is to optimize the weight parameters in each layer, gradually combining simpler features into complex ones, so that the most appropriate layered representation can be learned from the data. A single cycle of the optimization process proceeds as follows. First, given a training dataset, a forward pass sequentially computes the output of each layer and propagates the function signal forward through the network. In the final output layer, the target loss function measures the error between the inferred output and the given label. To minimize the training error, a backward pass uses the chain rule to backpropagate the error signal and calculate the gradient with respect to all weights in the entire neural network. Finally, an optimization algorithm based on stochastic gradient descent is used to update the weight parameters. While batch gradient descent performs parameter updates for each complete dataset, stochastic gradient descent provides a stochastic approximation by performing updates for each small data example. Several optimization algorithms are derived from stochastic gradient descent. For example, the Adagrad and Adam training algorithms perform stochastic gradient descent while adaptively modifying the learning rate based on the update frequency and momentum of the gradient of each parameter, respectively.

[0082] Another core element in deep neural network training is regularization, which refers to strategies designed to avoid overfitting and thus achieve good generalization performance. For example, weight decay adds a penalty factor to the objective loss function so that the weight parameters converge to a small absolute value. Dropout randomly removes hidden units from a neural network during training and can be thought of as an ensemble of possible subnetworks. To enhance the capabilities of dropout, new activation functions, maximum output, and a dropout variant of recurrent neural networks (called rnnDrop) have been proposed. In addition, batch normalization provides a new regularization method by normalizing the scalar features of each activation within a mini-batch and learning each mean and variance as parameters.

[0083] Given the multidimensionality and high-dimensionality of sequence data, deep neural networks hold great promise in bioinformatics research due to their broad applicability and enhanced predictive power. Convolutional neural networks have been used to address sequence-based problems in genomics, such as motif discovery, pathogenic variant identification, and gene expression inference. Convolutional neural networks use a weight-sharing strategy that is particularly useful for studying DNA because they can capture sequence motifs, which are short, recurring local patterns in DNA that are hypothesized to have significant biological functions. The hallmark of convolutional neural networks is the use of convolutional filters.

[0084] Unlike traditional classification methods based on carefully designed and handcrafted features, convolutional filters perform adaptive learning of features, similar to the process of mapping raw input data into an information representation of knowledge. In this sense, convolutional filters act as a series of motif scanners, as a set of such filters is able to identify relevant patterns in the input and update themselves during the training process. Recurrent neural networks can capture long-range dependencies in sequence data of varying lengths, such as protein or DNA sequences.

[0085] Therefore, there is an opportunity to use a principled deep learning-based framework for template generation and base calling.

[0086] In the era of high-throughput technologies, accumulating the most interpretable data at the lowest cost per work remains a major challenge. Cluster-based nucleic acid sequencing methods, such as those that use bridge amplification to form clusters, have made important contributions to the goal of increasing nucleic acid sequencing throughput. These cluster-based methods rely on sequencing dense populations of nucleic acids immobilized on a solid support and typically involve using image analysis software to deconvolute the optical signals generated during the simultaneous sequencing of multiple clusters located at different locations on the solid support.

[0087] The present invention relates to the sequencing technology of solid-phase nucleic acid cluster.However, this type of sequencing technology based on solid-phase nucleic acid cluster is still faced with sizable obstacles, and these obstacles limit achievable flux.For example, in the sequencing method based on cluster, determining physically too close to each other and unable to spatially resolve or in fact may have an obstacle aspect the nucleic acid sequence of two or more clusters that physically overlap on solid support.For example, current image analysis software may need valuable time and computing resources to determine which cluster in two overlapping clusters has sent light signal.Therefore, for multiple detection platforms, the compromise about the quantity and / or quality of obtainable nucleic acid sequence information is inevitable.

[0088] Genomic approaches based on high-density nucleic acid clusters have also expanded into other areas of genomic analysis. For example, nucleic acid cluster-based genomics can be used for sequencing applications, diagnostics and screening, gene expression analysis, epigenetic analysis, and genetic analysis of polymorphisms. Each of these nucleic acid cluster-based genomic technologies is also limited when data generated from closely spaced or spatially overlapping nucleic acid clusters cannot be interpreted.

[0089] Clearly, there remains a need to increase the quality and quantity of nucleic acid sequencing data that can be rapidly and cost-effectively obtained for a wide variety of uses, including genomics (e.g., for genomic characterization of any and all animal, plant, microbial, or other biological species or populations), pharmacogenetics, transcriptomics, diagnostics, prognosis, biomedical risk assessment, clinical and research genetics, personalized medicine, drug efficacy and drug interaction assessment, veterinary medicine, agriculture, evolutionary and biodiversity studies, aquaculture, forestry, oceanography, ecological and environmental management, and other purposes.

[0090] The disclosed technology provides neural network-based methods and systems that address these and similar needs, including increasing throughput levels in high-throughput nucleic acid sequencing technologies, and provide other related advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] This patent or patent application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. Color drawings are also available in pairs via the Supplemental Content tab.

[0092] In the drawings, like reference characters generally refer to like parts throughout the different views. Additionally, the drawings are not necessarily drawn to scale, emphasis instead being placed on illustrating the principles of the disclosed technology. In the following description, various implementations of the disclosed technology are described with reference to the following drawings, wherein:

[0093] Figure 1 One specific implementation of a processing pipeline that uses sub-pixel base calling to determine cluster metadata is shown.

[0094] Figure 2 One specific implementation of a flow cell containing clusters within its segments is depicted.

[0095] Figure 3 An example of an Illumina GA-IIx flow cell with eight channels is shown.

[0096] Figure 4Depicted is an image set of sequencing images for a four-channel chemistry, ie, the image set has four sequencing images captured using four different wavelength bands (images / imaging channels) in the pixel domain.

[0097] Figure 5 It is a specific implementation of dividing the sequencing image into sub-pixels (or sub-pixel regions).

[0098] Figure 6 The initial center coordinates of the clusters identified by the base caller during sub-pixel base calling are shown.

[0099] Figure 7 Depicted is one implementation of merging sub-pixel base calls generated over multiple sequencing cycles to generate a so-called "cluster map" containing cluster metadata.

[0100] Figure 8a An example of a cluster map generated by merging sub-pixel base calls is shown.

[0101] Figure 8b Depicted is a specific implementation of sub-pixel base calling.

[0102] Figure 9 Another example of a cluster map identifying cluster metadata is shown.

[0103] Figure 10 It shows how to calculate the centroid (COM) of disjoint regions in a cluster map.

[0104] Figure 11 One specific implementation of computing a weighted attenuation factor based on the Euclidean distance of sub-pixels in a disjoint region from the COM of the disjoint region is depicted.

[0105] Figure 12 One specific implementation of an exemplary ground truth decay map derived from an exemplary cluster map produced by sub-pixel base calling is shown.

[0106] Figure 13 A specific implementation of deriving a ternary map from a cluster map is shown.

[0107] Figure 14 A specific implementation of deriving a binary map from a cluster map is shown.

[0108] Figure 15 is a block diagram illustrating one specific implementation of generating training data for training a neural network-based template generator and a neural network-based base caller.

[0109] Figure 16Features of the disclosed training examples used to train a neural network-based template generator and a neural network-based base caller are shown.

[0110] Figure 17 One embodiment of the disclosed neural network-based template generator processing input image data and generating output values ​​for each element in an array is shown. In one embodiment, the array is an attenuation map. In another embodiment, the array is a ternary map. In yet another embodiment, the array is a binary map.

[0111] Figure 18 One specific implementation of a post-processing technique is shown that is applied to an attenuation map, a ternary map, or a binary map produced by a neural network-based template generator to derive cluster metadata, including cluster center, cluster shape, cluster size, cluster background, and / or cluster boundaries.

[0112] Figure 19 A specific implementation for extracting cluster intensities in the pixel domain is depicted.

[0113] Figure 20 One specific implementation of extracting cluster intensities in the sub-pixel domain is shown.

[0114] Figure 21a Three different implementations of a neural network based template generator are shown.

[0115] Figure 21b Depicted is one specific implementation of input image data fed as input to a neural network-based template generator 1512. The input image data comprises a series of image sets having sequencing images generated during a certain number of initial sequencing cycles of a sequencing run.

[0116] Figure 22 Shown from Figure 21b A specific implementation of extracting patches from a series of image sets in the image data to produce a series of "reduced-size" image sets forming the input image data.

[0117] Figure 23 Describes the Figure 21b In one embodiment, a series of image sets in the image data set are upsampled to produce a series of "upsampled" image sets forming the input image data.

[0118] Figure 24 Shown from Figure 23 One specific implementation of extracting patches from a series of upsampled image sets in the image processing apparatus to produce a series of "upsampled and downsized" image sets forming the input image data.

[0119] Figure 25One specific implementation of an overall exemplary process for generating ground truth data for training a neural network-based template generator is shown.

[0120] Figure 26 A specific implementation of the regression model is shown.

[0121] Figure 27 A specific implementation of generating a ground truth decay map from a cluster map is described. The ground truth decay map is used as ground truth data for training a regression model.

[0122] Figure 28 It is a specific implementation of using backpropagation based gradient update technique to train regression models.

[0123] Figure 29 It is a specific implementation of generating templates by regression models during inference.

[0124] Figure 30 One specific implementation of subjecting the attenuation map to post-processing to identify cluster metadata is shown.

[0125] Figure 31 Depicted is a specific implementation of a watershed segmentation technique for identifying non-overlapping contiguous clusters / groups of sub-pixels within clusters that characterize clusters.

[0126] Figure 32 is a table showing an exemplary U-Net architecture for a regression model.

[0127] Figure 33 Different methods of extracting cluster intensities using cluster shape information identified in a template image are shown.

[0128] Figure 34 Different methods of using the output of a regression model for base calling are shown.

[0129] Figure 35 The difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center is shown. The results show that using the COM improves base calling.

[0130] Figure 36 An exemplary decay map produced by the regression model is shown on the left. Figure 36 Also shown on the right is an exemplary ground truth decay map that the regression model approximates during training.

[0131] Figure 37 Depicted is one specific implementation of a peak locator that identifies cluster centers in an attenuation map by detecting peaks.

[0132] Figure 38Peaks detected by the peak locator in the attenuation map produced by the regression model are compared with corresponding peaks in the ground truth attenuation map.

[0133] Figure 39 The performance of the regression model using precision and recall statistics is shown.

[0134] Figure 40 The performance of the regression model and the RTA base caller were compared for a library concentration of 20 pM (normal operation).

[0135] Figure 41 The performance of the regression model was compared to the RTA base caller for a 30 pM library concentration (dense run).

[0136] Figure 42 The number of non-duplicate correct read pairs detected by the regression model (ie, the number of paired reads where neither read aligned inward within a reasonable distance) was compared to the number of non-duplicate correct read pairs detected by the RTA base caller.

[0137] Figure 43 On the right is shown a first attenuation map resulting from the regression model. Figure 43 A second decay map resulting from the regression model is shown on the left.

[0138] Figure 44 The performance of the regression model was compared to the RTA base caller for a 40 pM library concentration (a highly dense run).

[0139] Figure 45 On the left is shown a first attenuation map resulting from the regression model. Figure 45 On the right are shown the results of the thresholding, peak location and watershed segmentation techniques applied to the first attenuation map.

[0140] Figure 46 A specific implementation of a binary classification model is shown.

[0141] Figure 47 It is a specific implementation for training a binary classification model using a backpropagation-based gradient update technique involving a softmax score.

[0142] Figure 48 is another implementation for training a binary classification model using a backpropagation-based gradient update technique involving sigmoid scores.

[0143] Figure 49 Another specific implementation of input image data fed to a binary classification model and corresponding class labels for training the binary classification model is shown.

[0144] Figure 50 It is a specific implementation of generating templates by a binary classification model during inference.

[0145] Figure 51 One specific implementation is shown where the binary map is subjected to peak detection to identify cluster centers.

[0146] Figure 52a An exemplary binary map produced by a binary classification model is shown on the left. Figure 52a Also shown on the right is an exemplary ground truth binary map that the binary classification model approaches during training.

[0147] Figure 52b The performance of the binary classification model using the precision statistic is shown.

[0148] Figure 53 is a table showing an example architecture for a binary classification model.

[0149] Figure 54 A specific implementation of the ternary classification model is shown.

[0150] Figure 55 It is a specific implementation of using back-propagation based gradient update technique to train a ternary classification model.

[0151] Figure 56 Another specific implementation of input image data fed to a ternary classification model and corresponding class labels for training the ternary classification model is shown.

[0152] Figure 57 is a table showing an example architecture for a ternary classification model.

[0153] Figure 58 It is a specific implementation of generating templates by a ternary classification model during inference.

[0154] Figure 59 The ternary map generated by the ternary classification model is shown.

[0155] Figure 60 Depicted is the cell array produced by the ternary classification model 5400, along with the cell-by-cell output values.

[0156] Figure 61 One specific implementation of post-processing the ternary map to identify cluster centers, cluster backgrounds, and cluster interiors is shown.

[0157] Figure 62a Example predictions for a ternary classification model are shown.

[0158] Figure 62b Other exemplary predictions for the ternary classification model are shown.

[0159] Figure 62c Other exemplary predictions for the ternary classification model are also shown.

[0160] Figure 63 Depicts from Figure 62a A specific implementation of deriving cluster centers and cluster shapes from the output of the ternary classification model in

[15] .

[0161] Figure 64 The base calling performance of binary classification models, regression models, and RTA base callers was compared.

[0162] Figure 65 The performance of the ternary classification model was compared with that of the RTA base caller under three contexts, five sequencing metrics, and two run densities.

[0163] Figure 66 The performance of the regression model was compared with that of the RTA base caller. Figure 65 Comparisons are made under the three contexts discussed in

[15] , five sequencing metrics, and two run densities.

[0164] Figure 67 Focusing on the penultimate layer of the neural network-based template generator.

[0165] Figure 68 Visualizes what the penultimate layer of the neural network-based template generator has learned due to backpropagation-based gradient update training. The example implementation visualizes Figure 67 24 of the 32 trained convolutional filters of the penultimate layer depicted in .

[0166] Figure 69 Cluster center predictions from the binary classification model (blue) are superimposed on those from the RTA base caller (pink).

[0167] Figure 70 Cluster center predictions (pink) made by the RTA base caller are superimposed on a visualization of the trained convolutional filters of the penultimate layer of the binary classification model.

[0168] Figure 71 One specific implementation of training data for training a neural network-based template generator is shown.

[0169] Figure 72 It is a specific implementation of image registration using beads based on cluster center prediction of a neural network-based template generator.

[0170] Figure 73 One specific implementation of cluster statistics for clusters identified by a neural network-based template generator is shown.

[0171] Figure 74 It shows how the ability of the neural network-based template generator to distinguish adjacent clusters improves when the number of initial sequencing cycles using input image data increases from five to seven.

[0172] Figure 75 Shown is the difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center.

[0173] Figure 76 The performance of the neural network-based template generator for additional detected clusters is plotted.

[0174] Figure 77 Different datasets used to train a neural network-based template generator are shown.

[0175] Figure 78 The processing stages used by an RTA base caller for base calling are shown according to one specific implementation.

[0176] Figure 79 One specific implementation of base calling using the disclosed neural network-based base caller is illustrated.

[0177] Figure 80 It is a specific implementation of transforming the location / position information of cluster centers identified from the output of a neural network-based template generator from the sub-pixel domain to the pixel domain.

[0178] Figure 81 is a specific implementation that uses a cycle-specific transformation and an image channel-specific transformation to derive so-called "transformed cluster centers" from reference cluster centers.

[0179] Figure 82 An image patch is illustrated which is part of the input data fed to a neural network based base caller.

[0180] Figure 83 Depicted is one specific implementation of determining distance values ​​for a distance channel when a single target cluster is being base called by a neural network-based base caller.

[0181] Figure 84 A specific implementation of encoding the distance value between the calculated pixel and the target cluster pixel by pixel is shown.

[0182] Figure 85a Described is one specific implementation of determining distance values ​​for a distance channel when multiple target clusters are simultaneously base called by a neural network-based base caller.

[0183] Figure 85b It is shown that for each target cluster in the target cluster, some nearest pixels are determined based on the distance from the pixel center to the nearest cluster center.

[0184] Figure 86 A specific implementation of pixel-by-pixel encoding of the calculated minimum distance value between a pixel and the nearest cluster in the cluster is shown.

[0185] Figure 87 One specific implementation of pixel-to-cluster classification / attribution / categorization using what is referred to herein as "cluster shape data" is illustrated.

[0186] Figure 88 A specific implementation of calculating distance values ​​using cluster shape data is shown.

[0187] Figure 89 One specific implementation of pixel-by-pixel encoding of the calculated distance values ​​between pixels and assigned clusters is shown.

[0188] Figure 90 One specific implementation of a specialized architecture for a neural network-based base caller that isolates the processing of data from different sequencing cycles is illustrated.

[0189] Figure 91 Depicts a specific implementation of isolated convolution.

[0190] Figure 92a Describes a specific implementation of combined convolution.

[0191] Figure 92b Depicts another implementation of combined convolution.

[0192] Figure 93 One specific implementation of the convolutional layers of a neural network-based base caller is shown, where each convolutional layer has a set of convolutional filters.

[0193] Figure 94 Depicted are two configurations of scaling channels that complement the image channels.

[0194] Figure 95a One specific implementation of input data for a single sequencing cycle that produces a red image and a green image is illustrated.

[0195] Figure 95b We illustrate one implementation of a range channel that provides an additive bias that is incorporated into the feature maps generated from the image channels.

[0196] Figure 96a 、 Figure 96b and Figure 96c Depicts a specific implementation of base calling for a single target cluster.

[0197] Figure 97 One specific implementation of simultaneous base calling for multiple target clusters is shown.

[0198] Figure 98 One specific implementation is shown of simultaneously performing base calling on multiple target clusters at multiple subsequent sequencing cycles, thereby simultaneously generating a base called sequence for each of the multiple target clusters.

[0199] Figure 99 A dimensional diagram is illustrated for a single cluster base calling implementation.

[0200] Figure 100 A dimensional diagram illustrating a multiple cluster, single sequencing cycle base calling implementation.

[0201] Figure 101 A dimensional diagram illustrating a specific implementation of base calling for multiple clusters and multiple sequencing cycles is shown.

[0202] Figure 102a Depicted is an exemplary array input configuration for multi-cycle input data.

[0203] Figure 102b An exemplary stacked input configuration for multi-cycle input data is shown.

[0204] Figure 103a One implementation is depicted in which pixels of an image patch are reconstructed so that the center of a base-called target cluster is centered in a center pixel.

[0205] Figure 103b Another exemplary reconstructed / shifted image patch is depicted, where (i) the center of the central pixel coincides with the center of the target cluster, and (ii) the non-central pixels are equidistant from the center of the target cluster.

[0206] Figure 104 One specific implementation is shown for base calling a single target cluster at the current sequencing cycle using a standard convolutional neural network and reconstructed input.

[0207] Figure 105 One specific implementation of base calling for multiple target clusters at the current sequencing cycle using a standard convolutional neural network and aligned inputs is shown.

[0208] Figure 106 One specific implementation of base calling for multiple target clusters at multiple sequencing cycles using a standard convolutional neural network and aligned inputs is shown.

[0209] Figure 107 One specific implementation of training a neural network-based base caller is shown.

[0210] Figure 108a Depicted is a specific implementation of a hybrid neural network for use as a neural network-based base caller.

[0211] Figure 108b Shown is a specific implementation of 3D convolution used by the recurrent module of the hybrid neural network to produce the current hidden state representation.

[0212] Figure 109 One specific implementation of processing per-cycle input data for a single sequencing cycle in the series of t sequencing cycles to be base called by a cascade of convolutional layers of a convolutional module is shown.

[0213] Figure 110 Depicted is one specific implementation of mixing per-cycle input data for a single sequencing cycle with a corresponding convolutional representation of the single sequencing cycle produced by a cascade of convolutional layers of a convolutional module.

[0214] Figure 111 One implementation of arranging the flattened mixed representations of subsequent sequencing cycles into a stack is shown.

[0215] Figure 112a Shows that Figure 111 The stack of undergoes recursive application of 3D convolution in the forward and backward directions and produces one specific implementation of a base call for each of the clusters at each sequencing cycle of the series of t sequencing cycles.

[0216] Figure 112b A specific implementation of processing a 3D input volume x(t) comprising a flattened set of hybrid representations is shown by applying the input gate, activation gate, forget gate, and output gate of a long short-term memory (LSTM) network using 3D convolution. The LSTM network is part of the recurrent module of the hybrid neural network.

[0217] Figure 113 One specific implementation of balancing trinucleotides (3-mers) in training data for training a neural network-based base caller is shown.

[0218] Figure 114 The base calling accuracy of the RTA base caller was compared with the base calling accuracy of the neural network-based base caller.

[0219] Figure 115 The block-to-block generalization of the RTA base caller is compared with the block-to-block generalization of the neural network-based base caller on the same block.

[0220] Figure 116The block-to-block generalization of the RTA base caller is compared with the block-to-block generalization of the neural network-based base caller on the same block and on different blocks.

[0221] Figure 117 The block-to-block generalization of the RTA base caller is also compared with that of a neural network-based base caller on different blocks.

[0222] Figure 118 Shown is how different sizes of image patches fed as input to a neural network-based base caller affect base calling accuracy.

[0223] Figure 119 、 Figure 120 、 Figure 121 and Figure 122 The neural network-based base caller is shown to generalize slot-to-slot on training data from Acinetobacter baumannii and Escherichia coli.

[0224] Figure 123 Shown above relative to Figure 119 、 Figure 120 、 Figure 121 and Figure 122 The error distribution of the slot-to-slot generalization.

[0225] Figure 124 will be Figure 123 Error distribution of the detected error sources attributed to low cluster intensity in the green channel.

[0226] Figure 125 The error distributions of the RTA base caller and the neural network-based base caller for two sequencing runs (Read 1 and Read 2) were compared.

[0227] Figure 126a Shown are the run-to-run generalization of the neural network-based base caller on four different instruments.

[0228] Figure 126b Shown is the run-to-run generalization of the neural network-based base caller across four different runs performed on the same instrument.

[0229] Figure 127 Shown are the genomic statistics of the training data used to train the neural network-based base caller.

[0230] Figure 128 Shown is the genomic context of the training data used to train a neural network-based base caller.

[0231] Figure 129The base calling accuracy of the neural network-based base caller in base-called long reads (e.g., 2×250) is shown.

[0232] Figure 130 A specific implementation of how a neural network-based base caller focuses on a central cluster of pixels and their neighboring pixels on an image patch is shown.

[0233] Figure 131 Various hardware components and configurations for training and running a neural network-based base caller according to one implementation are shown. In other implementations, different hardware components and configurations are used.

[0234] Figure 132 Various sequencing tasks that can be performed using neural network-based base callers are shown.

[0235] Figure 133 It is a scatter plot visualized by t-distributed stochastic neighbor embedding (t-SNE) and depicts the base calling results of the neural network-based base caller.

[0236] Figure 134 One specific implementation of selecting base call confidence probabilities produced by a neural network-based base caller for use in quality scoring is illustrated.

[0237] Figure 135 A specific implementation of neural network-based quality scoring is shown.

[0238] Figures 136a to 136b Depicts one specific implementation of the correspondence between quality scores and base call confidence predictions made by a neural network-based base caller.

[0239] Figure 137 One specific implementation of inferring quality scores from base call confidence predictions made by a neural network-based base caller during inference is shown.

[0240] Figure 138 One specific implementation of training a neural network-based quality scorer to process input data derived from sequencing images and directly generate quality indicators is shown.

[0241] Figure 139 One specific implementation is shown where quality indicators are generated directly during inference as output of a neural network-based quality scorer.

[0242] Figure 140 Depicted is one specific implementation of using a lossless transform to generate transformed data that can be fed as input to a neural network-based template generator, a neural network-based base caller, and a neural network-based quality scorer.

[0243] Figure 141 One specific implementation of integrating a neural network-based template generator with a neural network-based base caller using area-weighted factorization is illustrated.

[0244] Figure 142 Another implementation of integrating a neural network-based template generator with a neural network-based base caller using upsampling and background masking is illustrated.

[0245] Figure 143 An example of area-weighted factorization 14300 of the contribution from only a single cluster per pixel is depicted.

[0246] Figure 144 Depicted is an example of area-weighted factorization of contributions from multiple clusters per pixel.

[0247] Figure 145 Depicts an example of upsampling and background masking using interpolation.

[0248] Figure 146 Depicts an example of upsampling and background masking using sub-pixel count weighting.

[0249] Figure 147A and Figure 147B A specific implementation of a sequencing system is described. The sequencing system includes a configurable processor.

[0250] Figure 147C is a simplified block diagram of a system for analyzing sensor data (such as base calling sensor output) from a sequencing system.

[0251] Figure 148A is a simplified diagram illustrating aspects of base calling operations, including the functionality of a runtime program executed by a host processor.

[0252] Figure 148B Is a configurable processor such as Figure 147C A simplified diagram depicting the configuration of a configurable processor.

[0253] Figure 149 Can be Figure 147A The sequencing system is a computer system used to implement the technology disclosed in this article.

[0254] Figure 150 Different implementations of data preprocessing are shown, which may include data normalization and data augmentation.

[0255] Figure 151 shows that when a neural network-based base caller is trained on bacterial data and tested on human data, Figure 150 Our data normalization technique (DeepRTA(Normalization)) and data augmentation technique (DeepRTA(Enhancement)) reduced the base calling error percentage, where bacterial data and human data shared the same assays (e.g., both contained intron data).

[0256] Figure 152 It is shown that when a neural network-based base caller is trained on non-exonic data (e.g., intronic data) and tested on exonic data, Figure 151 The data normalization technology (DeepRTA(Normalization)) and data enhancement technology (DeepRTA(Enhancement)) of the proposed method reduce the percentage of base calling errors. DETAILED DESCRIPTION

[0257] The following discussion is presented to enable any person skilled in the art to make and use the disclosed technology, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the specific implementations shown but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0258] Introduction

[0259] Base calling from digital images is massively parallel and computationally intensive. This presents many technical challenges that we recognized before the introduction of new technologies.

[0260] As the base classification cycle progresses, the signal from the image set being evaluated becomes increasingly weak, especially as the base chains become longer. As the base classification progresses along the length of the chain, the signal-to-noise ratio decreases, and therefore the reliability decreases. When the estimated reliability of a base classification changes, an updated reliability estimate is expected.

[0261] Capture digital images from clusters of amplified sample strands. Samples are amplified by replicating strands using a variety of physical structures and chemistries. During sequencing-by-synthesis, tags are chemically attached cyclically and stimulated to emit light. A digital sensor collects photons from the tags, which are read out at the pixel, to produce an image.

[0262] Interpreting digital images to classify bases requires resolving positional uncertainty, which is hampered by limited image resolution. At resolutions greater than those collected during base calling, it is apparent that the imaged clusters have irregular shapes and uncertain center positions. Because cluster positions are not mechanically controlled, cluster centers are not aligned with pixel centers. The pixel center can be an integer coordinate assigned to a pixel. In other implementations, the pixel center can be the upper left corner of the pixel. In other implementations, the pixel center can be the center of mass or centroid of the pixel. Amplification does not produce uniform cluster shapes. Therefore, the distribution of cluster signals in the digital image is a statistical distribution rather than a regular pattern. We refer to this as positional uncertainty.

[0263] One of the signal classes may not produce a detectable signal and be classified at a specific location based on a "dark" signal. Therefore, a template is necessary for classification during dark cycles. Generating a template uses multiple imaging cycles to resolve initial positional uncertainties to avoid missing dark signals.

[0264] Trade-offs in image sensor size, magnification, and stepped design result in relatively large pixel sizes that are too large to consider cluster centers to coincide with sensor pixel centers. The present disclosure uses pixels in two senses. A physical sensor pixel is the area where an optical sensor reports detected photons. A logical pixel (referred to simply as a pixel) is the data corresponding to at least one physical pixel (i.e., the data read from a sensor pixel). A pixel can be subdivided or "upsampled" into sub-pixels, such as 4×4 sub-pixels. To account for the likelihood that all photons hit one side of a physical pixel rather than the opposite side, values ​​can be assigned to the sub-pixels by interpolation, such as bilinear interpolation or area weighting. Interpolation or bilinear interpolation is also applied when pixels are reframed by applying an affine transformation to the data from the physical pixels.

[0265] Larger physical pixels are more sensitive to weak signals than smaller pixels. While digital sensors have improved over time, the physical limitations of collector surface area are unavoidable. Given these design tradeoffs, traditional systems have been designed to collect and analyze image data from a 3×3 patch of the sensor, with the cluster center located somewhere in the center pixel of the patch.

[0266] The high-resolution sensor captures only a portion of the imaging medium at a time. The sensor is stepped across the imaging medium to cover the entire field. Thousands of digital images can be collected during a single processing cycle.

[0267] The sensor and lighting design are combined to distinguish at least four illumination response values ​​for classifying the base. If a traditional RGB camera with a Bayer color filter array is used, four sensor pixels will be combined into a single RGB value. This will reduce the effective sensor resolution to a quarter of the original. Alternatively, multiple images can be collected at a single location using different illumination wavelengths and / or different filters rotated into the position between the imaging medium and the sensor. The number of images required to distinguish the four base classifications varies between systems. Some systems use an image with four intensity levels for different classes of bases. Other systems use two images with different illumination wavelengths (e.g., red and green) and / or a filter with a truth table to classify the base. The system can also use four images with different illumination wavelengths and / or filters tuned to specific base classes.

[0268] In practice, massively parallel processing of digital images is required to align and combine relatively short chains (approximately 30 to 2000 base pairs) into longer sequences, which may be millions or even billions of bases in length. Redundant samples are desirable on the imaging medium so that a portion of the sequence can be covered by dozens of sample reads. Millions or at least hundreds of thousands of sample clusters can be imaged from a single imaging medium. Massively parallel processing of so many clusters increases sequencing capacity while reducing costs.

[0269] Sequencing capacity is growing at a rate comparable to Moore’s Law. Although the first sequencing efforts cost billions of dollars, in 2018, service providers such as Illumina TM The price of delivering sequencing results is in the hundreds of dollars. As sequencing becomes mainstream and unit prices fall, less computing power is available for classification, increasing the challenge of near-real-time classification. Given these technical challenges, we turned to the disclosed technology.

[0270] The disclosed techniques improve the processing during template generation to resolve positional uncertainty and during base classification of clusters at resolved positions. Application of the disclosed techniques allows the use of less expensive hardware, reducing machine costs. Near-real-time analysis can become cost-effective, reducing the lag between image collection and base classification.

[0271] The disclosed technology uses an upsampled image generated by interpolating sensor pixels into sub-pixels and then creating a template that resolves positional uncertainty. The resulting sub-pixels are submitted to a base caller for classification, which considers the sub-pixels to be located at cluster centers. Clusters are determined from groups of adjacent sub-pixels that repeatedly receive the same base classification. This aspect of the technology leverages existing base calling technology to determine cluster shape and super-locate cluster centers with sub-pixel resolution.

[0272] Another aspect of the disclosed technology is the creation of a ground truth training dataset that pairs images with confidently determined cluster centers and / or cluster shapes. Deep learning systems and other machine learning methods require large training sets. Human-curated data is expensive to compile. The disclosed technology can be used to generate large, confidently classified training datasets using existing classifiers in non-standard operating modes without intervention or labor. In non-standard operating modes, such as in a CNN-based deep learning system, the training data associates the original image with cluster centers and / or cluster shapes that can be obtained from an existing classifier, which can then process the image sequence directly. A training image can be rotated and reflected to produce additional equally valid examples. The training examples can focus on areas of predetermined size within the entire image. The context evaluated during base calling determines the size of the exemplary training area, rather than determining the size of the image from the entire imaging medium or the size of the entire imaging medium.

[0273] The disclosed technology can generate different types of maps that can be used as training data or templates for base classification and associate cluster centers and / or cluster shapes with digital images. First, subpixels can be classified as cluster centers, thereby locating the cluster center within a physical sensor pixel. Second, the cluster center can be calculated as the center of mass of the cluster shape. This location can be reported with a selected digital precision. Third, the cluster center with surrounding subpixels can be reported in an attenuation map at subpixel or pixel resolution. As the separation between the region and the cluster center increases, the attenuation map reduces the weight of photons detected in the region, thereby weakening the signal from more distant locations. Fourth, binary or ternary classification can be applied to subpixels or pixels in clusters of adjacent regions. In binary classification, regions are classified as belonging to the cluster center or as background. In ternary classification, the third type is the region that contains the cluster interior but does not contain the cluster center. Subpixel classification for cluster center locations can replace the real-valued cluster center coordinates within the larger optical pixel.

[0274] Alternatively, maps can be generated initially as ground truth datasets, or neural networks can be used to generate these maps through training. For example, clusters can be depicted as disjoint regions of appropriately classified adjacent sub-pixels. If the centers have not yet been determined, the intensity map clusters from the neural network can be post-processed with a peak detector filter to calculate the cluster centers. Using so-called watershed analysis, adjacent regions can be assigned to individual clusters. When generated by a neural network inference engine, the maps can be used as templates for evaluating digital image sequences and classifying bases during the base calling cycle.

[0275] When classifying bases in a sequence of digital images, the neural network processes multiple image channels in the current cycle, as well as image channels in past and future cycles. Within a cluster, some chains may extend before or after the main process of synthesis, where out-of-phase markers are called prephasing or phasing. Given the empirically observed low rates of prephasing and postphasing, almost all noise in the signal generated by prephasing and postphasing can be processed by the neural network, which processes digital images in the current, past, and future cycles (only three cycles).

[0276] In the digital image pipeline within the current cycle, careful registration to align images within the cycle significantly contributes to accurate base classification. Combinations of wavelength and misaligned illumination sources, as well as other sources of error, produce small, correctable differences in the measured cluster center positions. A general affine transformation with translation, rotation, and scaling capabilities can be applied to precisely align cluster centers across the entire image patch. Affine transformations can be used to reconstruct image data and account for shifts in cluster centers.

[0277] Reconstructing image data means interpolating the image data, typically by applying an affine transformation. Reconstruction can place the center of the cluster of interest in the middle of the center pixel of a pixel patch. Alternatively, the reconstruction can align the image with a template to overcome jitter and other differences during image collection. Reconstruction involves adjusting the intensity values ​​of all pixels in the pixel patch. Bilinear and bicubic interpolation methods and weighted area adjustment are alternative strategies.

[0278] In some implementations, cluster center coordinates can be fed to the neural network as an additional image channel.

[0279] Distance signals can also contribute to base classification. Several types of distance signals reflect the separation of regions from cluster centers. The strongest light signal is considered to coincide with the cluster center. Light signals along the periphery of a cluster sometimes include stray signals from nearby clusters. It has been observed that classification is more accurate when the contribution of the signal component decays according to its separation from the cluster center. The distance signals that play a role include a single cluster distance channel, multiple cluster distance channels, and multiple cluster shape-based distance channels. A single cluster distance channel is applied to a patch with a cluster center in the center pixel. Therefore, the distance of all regions in the patch is the distance to the cluster center in the center pixel. Pixels that do not belong to the same cluster as the center pixel can be marked as background rather than given a calculated distance. Multiple cluster distance channels precalculate the distance from each region to the nearest cluster center. This has the possibility of connecting a region to the wrong cluster center, but this possibility is low. Multiple cluster shape-based distance channels associate regions (sub-pixels or pixels) to the pixel center that produces the same base classification through adjacent regions. Through calculation, the possibility of measuring the distance to the wrong pixel is avoided. Multiple clusters and multiple cluster shape-based distance signal methods have the advantage of being subject to pre-computation and working with multiple clusters in an image.

[0280] Neural networks can use shape information to separate signal from noise to improve the signal-to-noise ratio. In the above discussion, several methods for region classification and providing distance channel information were identified. In any of these methods, regions can be marked as background and not as part of a cluster to define cluster edges. Neural networks can be trained to utilize the resulting information about irregular cluster shapes. Distance information and background classification can be used in combination or separately. As cluster density increases, separating signals from adjacent clusters becomes increasingly important.

[0281] One approach to increasing parallel processing scale is to increase cluster density on the imaging medium. Increasing density has the disadvantage of increasing background noise when reading clusters with adjacent neighbors. For example, using shape data rather than arbitrary patches (e.g., 3×3 pixel patches) can help maintain signal separation as cluster density increases.

[0282] In one aspect of the disclosed technology, base classification scores can also be used to predict quality. The disclosed technology includes correlating classification scores directly or through a prediction model with traditional Sanger or Phred quality Q scores. Scores such as Q20, Q30, or Q40 are calculated by Q=-10log 10P is logarithmically related to the probability of base sorting error. The correlation between class score and Q score can be performed using a multi-output neural network or multivariate regression analysis. During base sorting, the advantage of real-time calculation of quality score is that defective sequencing can be terminated early. Applicants have found that it is possible to occasionally (rarely) decide to terminate the run at one-eighth to one-quarter of the way through the entire analysis sequence. The decision to terminate can be made after 50 cycles or after 25 to 75 cycles. In the order-checking process that would originally have run 300 to 1000 cycles, early termination results in significant resource savings.

[0283] Specialized convolutional neural network (CNN) architectures can be used to classify bases within multiple cycles. One specialization involves isolation between digital image channels during processing of the initial layers. The convolution filter stack can be constructed to isolate processing between cycles, thereby preventing crosstalk between digital image sets from different cycles. The motivation for isolating processing between cycles is that images acquired at different cycles have residual registration errors and are therefore misaligned with respect to each other and have random translational offsets. This occurs because the motion accuracy of the sensor's motion stage is limited, and also because images acquired in different frequency channels have different optical paths and wavelengths.

[0284] The motivation for using image sets from subsequent cycles is that the contributions to pre-phasing and post-phasing the signal in a particular cycle are second-order contributions. Therefore, it can help the convolutional neural network to structurally isolate the lower-level convolutions of the digital image sets between image collection cycles.

[0285] The convolutional neural network architecture can also be specifically designed to process information about clustering. Templates of cluster centers and / or shapes provide additional information that the convolutional neural network combines with digital image data. Classification of cluster centers and distance data can be applied repeatedly across cycles.

[0286] The convolutional neural network can be configured to classify multiple clusters in an image field. When classifying multiple clusters, the distance channel of a pixel or sub-pixel can more compactly contain distance information relative to the nearest cluster center or the center of an adjacent cluster to which the pixel or sub-pixel belongs. Alternatively, a large distance vector can be provided for each pixel or sub-pixel, or at least for each pixel or sub-pixel that contains a cluster center, which gives complete distance information from the cluster center to all other pixels that are contextual to the given pixel.

[0287] Some combinations of template generation and base calling can use area-weighted variations instead of distance channels. We now discuss how to use the output of the template generator directly instead of the distance channel.

[0288] We discuss three considerations that influence the direct application of a template image to modify pixel values: whether the image set is processed in the pixel domain or the sub-pixel domain; how the area weights are computed in either domain; and, in the sub-pixel domain, the application of the template image as a mask to modify the interpolated intensity values.

[0289] Performing base classification in the pixel domain has the advantage of not requiring an increase in computation due to upsampling (e.g., 16x). In the pixel domain, even the top convolution layer can have sufficient cluster density to justify performing computations that would otherwise be lost, rather than adding logic to cancel unnecessary computations. We begin with an example using template image data directly in the pixel domain without a range channel.

[0290] In some implementations, classification is focused on specific clusters. In these cases, pixels on the periphery of a cluster may have different modified intensity values, depending on which adjacent cluster is the focus of the classification. A template image in the sub-pixel domain may indicate that overlapping pixels contribute to the intensity values ​​of two different clusters. When two or more adjacent or contiguous clusters both overlap a pixel, we refer to an optical pixel as an "overlapping pixel"; both of these two or more adjacent or contiguous clusters contribute to the intensity reading from the optical pixel. Watershed analysis (named for the division of rainwater flows into different water bodies at ridge lines) can be applied to separate even adjacent clusters. When data is received for cluster-by-cluster classification, a template image can be used to modify the intensity data of overlapping pixels along the periphery of the cluster. The overlapping pixels may have different modified intensities, depending on which cluster is the focus of the classification.

[0291] The modified intensity of a pixel can be reduced based on the sub-pixel contribution of the overlapping pixel to the primary cluster (i.e., the cluster to which the pixel belongs or the cluster whose intensity emission is primarily depicted by the pixel) rather than the distant cluster (i.e., the non-primary cluster whose intensity emission is depicted by the pixel). Assume that 5 sub-pixels are part of the primary cluster and 2 sub-pixels are part of the distant cluster. Then, 7 sub-pixels contribute to the intensity of the primary cluster or the distant cluster. During focus on the primary cluster, in one embodiment, the intensity of the overlapping pixel is reduced by 7 / 16 because 7 of the 16 sub-pixels contribute to the intensity of the primary cluster or the distant cluster. In another embodiment, the intensity is reduced by 5 / 16 based on the area of ​​the sub-pixels contributing to the primary cluster divided by the total number of sub-pixels. In a third embodiment, the intensity is reduced by 5 / 7 based on the area of ​​the sub-pixels contributing to the primary cluster divided by the total area of ​​the contributing sub-pixels. When the focus shifts to the distant cluster, the latter two calculations change, resulting in a fraction with a numerator of "2".

[0292] Of course, further reduction in intensity can be applied if the distance channel is considered together with the sub-pixel map of the cluster shape.

[0293] Once the pixel intensities of the clusters that are the focus of classification have been modified using the template image, the modified pixel values ​​are convolved through the layers of a neural network-based classifier to produce modified images. These modified images are used to classify bases in subsequent sequencing cycles.

[0294] Alternatively, classification in the pixel domain can be performed simultaneously for all pixels or all clusters in an image block. In this case, only one modification of the pixel value can be applied to ensure reusability of intermediate calculations. Any of the scores given above can be used to modify the pixel intensity, depending on whether a smaller or larger intensity attenuation is required.

[0295] Once the pixel intensities of an image patch have been modified using the template image, the pixels and surrounding context are convolved through layers of a neural network-based classifier to produce a modified image. Performing convolution on the image patch allows for the reuse of intermediate computations between pixels that already share context. These modified images are then used to classify bases in subsequent sequencing cycles.

[0296] This description can be used in parallel to apply area weights in the sub-pixel domain. Parallel means that the weights can be calculated for individual sub-pixels. The weights can, but do not have to, be the same for different sub-pixel portions of an optical pixel. Repeating the above scenario with a main cluster of 5 and a distant cluster of 2 sub-pixels in overlapping pixels, respectively, the intensity distribution of the sub-pixels belonging to the main cluster can be 7 / 16, 5 / 16, or 5 / 7 of the pixel intensity. Similarly, if the range channel is considered together with the sub-pixel map of the cluster shape, further intensity reduction can be applied.

[0297] Once the pixel intensities of an image patch have been modified using the template image, the subpixels and surrounding context are convolved through layers of a neural network-based classifier to produce a modified image. Performing convolution on the image patch allows for the reuse of intermediate computations between subpixels that share context. These modified images are then used to classify bases in subsequent sequencing cycles.

[0298] Another alternative is to apply a template image as a binary mask in the sub-pixel domain to the image data interpolated into the sub-pixel domain. The template image can be arranged to require background pixels between clusters or to allow sub-pixels from different clusters to be adjacent. The template image can be used as a mask. If an interpolated pixel is classified as background in the template image, the mask determines whether the interpolated pixel retains the value assigned by the interpolation method or receives the background value (e.g., zero).

[0299] Similarly, once the pixel intensities of an image patch have been masked using a template image, the subpixels and surrounding context are convolved through layers of a neural network-based classifier to produce a modified image. Performing convolution on the image patch allows for the reuse of intermediate computations between subpixels that share context. These modified images are then used to classify bases in subsequent sequencing cycles.

[0300] Features of the disclosed techniques can be combined to classify any number of clusters within a shared context, thereby reusing intermediate computations. In one implementation, at optical pixel resolution, approximately ten percent of the pixels hold cluster centers to be classified. In conventional systems, 3×3 optical pixel groups are grouped as potential signal contributors to cluster centers for analysis, given the observation of irregularly shaped clusters. Even with a 3×3 filter away from the top convolutional layer, the cluster density may roll up into pixels at the cluster center based on light signals from substantially more than half of the optical pixels. Only at the supersampled resolution does the cluster center density at the top convolutional layer drop to less than one percent.

[0301] In some implementations, the shared context is substantial. For example, a 15×15 optical pixel context can contribute to accurate base classification. The equivalent context for a 4x upsampling would be 60×60 sub-pixels. This level of context helps the neural network recognize the effects of uneven lighting and background during imaging.

[0302] The disclosed technique uses small filters at lower convolutional layers to combine cluster boundaries in the template input with boundaries detected in the digital image input. The cluster boundaries help the neural network separate the signal from the background conditions and normalize the image processing relative to the background.

[0303] The disclosed technique essentially reuses intermediate computations. Assume that 20 to 25 cluster centers appear within a context area of ​​15×15 optical pixels. The first layer of convolution is then reused 20 to 25 times in a block-by-block rollup. The reuse factor decreases layer by layer until the penultimate layer, where the reuse factor at optical resolution first drops to less than 1x.

[0304] Block-by-block roll-up training and inference from multiple convolutional layers applies subsequent roll-ups to blocks of pixels or sub-pixel blocks. There is an overlap region around the perimeter of the block where data used during the roll-up of a first data block overlaps with and can be reused for a second rolled-up block. Within the block, in a central region surrounded by this overlap region are pixel values ​​and intermediate computations that can be rolled up and reused. Using the overlap region, convolution results that gradually reduce the size of the context field (e.g., from 15×15 to 13×13 by applying a 3×3 filter) can be written to the same storage block that holds the convolved values, saving memory without compromising reuse of underlying computations within the block. For larger blocks, sharing intermediate computations in the overlap region requires fewer resources. For smaller blocks, it may be possible to compute multiple blocks simultaneously to share intermediate computations in the overlap region.

[0305] Larger filters and dilation reduce the number of convolutional layers after lower convolutional layers react to cluster boundaries in the template and / or digital image data, which speeds up computation without harming classification.

[0306] The input channels for the template data can be selected so that the template structure is consistent with the classification of multiple cluster centers in the digital image field. The two alternatives described above do not meet this consistency criterion: reconstruction in the entire context and distance mapping. Reconstruction places the center of only one cluster at the center of the optical pixel. For classification of multiple clusters, it is better to provide a center offset for pixels classified as holding a cluster center.

[0307] Unless each pixel has its own distance map across the entire context, it is difficult to perform distance mapping across the entire context area (if provided). Simpler distance maps provide useful consistency for classifying multiple clusters from a digital image input patch.

[0308] The neural network can learn from the classification in the template of pixels or sub-pixels at the cluster boundaries, so the distance channel can be replaced by a template that provides binary or ternary classification and is accompanied by a cluster center offset channel. When used, the distance map can give the distance of the pixel from the center of the cluster to which the pixel (or sub-pixel) belongs. Or the distance map can give the distance to the nearest cluster center. The distance map can encode the binary classification with the label value assigned to the background pixels, or it can be a separate channel for pixel classification. Combined with the cluster center offset, the distance map can encode the ternary classification. In some specific implementations, especially in specific implementations where the pixel classification is encoded with one or two bits, it may be desirable to use separate channels for pixel classification and distance, at least during development.

[0309] The disclosed techniques may include reducing computations to save some computational resources in upper layers. A cluster center offset channel or a ternary classification map may be used to identify the centers of pixel convolutions that do not contribute to the final classification of the pixel center. In many hardware / software implementations, performing lookups and skipped convolution rollups during inference may be more efficient than performing even nine multiplications and eight additions to apply a 3×3 filter. In custom hardware that routes computations for parallel execution, each pixel may be classified within the pipeline. The cluster center map may then be used after the final convolution to obtain results for only the pixels coinciding with the cluster center, since only these pixels require final classification. Similarly, in the optical pixel domain, at the currently observed cluster density, rollup computations will be obtained for approximately ten percent of the pixels. In a 4x upsampled domain, on some hardware, more layers may benefit from skipped convolutions since less than one percent of sub-pixel classifications will be obtained in the top layer.

[0310] Template generation based on neural network

[0311] The first step in template generation is to determine cluster metadata. Cluster metadata identifies the spatial distribution of clusters and identifies the center, shape, size, background and / or boundaries of these clusters.

[0312] Determine cluster metadata

[0313] Figure 1 One specific implementation of a processing pipeline that uses sub-pixel base calling to determine cluster metadata is shown.

[0314] Figure 2 Depicts one specific implementation of a flow cell containing clusters within its blocks. The flow cell is divided into channels. These channels are further divided into non-overlapping regions called "blocks." During sequencing, clusters within the blocks and their surrounding background are imaged.

[0315] Figure 3 An exemplary Illumina GA-IIx with eight channels is shown. TM Circulation pool. Figure 3 Also shown is a zoomed-in view of one block and its cluster and its surrounding background.

[0316] Figure 4Depicts an image set of sequencing images for four-channel chemistry, i.e., the image set has four sequencing images captured using four different wavelength bands (images / imaging channels) in the pixel domain. Each image in the image set covers a block of a circulation cell and depicts the intensity emission captured for a specific imaging channel in a specific sequencing cycle of a plurality of sequencing cycles of a sequencing run performed on the circulation cell by a cluster on the block and its surrounding background. In one specific implementation, each imaging channel corresponds to a filter wavelength band in a plurality of filter wavelength bands. In another specific implementation, each imaging channel corresponds to an imaging event in a plurality of imaging events in a sequencing cycle. In yet another specific implementation, each imaging channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter. The intensity emission of the cluster includes signals detected from the analyte, which can be used to classify the bases associated with the analyte. For example, the intensity emission can be a signal indicating the photons emitted by a label, which is chemically attached to the analyte during the cycle of stimulating the label and can be detected by one or more digital sensors, as described above.

[0317] Figure 5 It is a specific implementation of dividing the sequencing image into sub-pixels (or sub-pixel regions). In the specific implementation of the example, a sub-pixel of one quarter (0.25) is used, which results in each pixel in the sequencing image being divided into sixteen sub-pixels. In view of the resolution of the sequencing image shown is 20x20 pixels (i.e., 400 pixels), the division produces 6400 sub-pixels. Each sub-pixel in the sub-pixel is regarded as the regional center of the sub-pixel base call by the base caller. In some specific implementations, the base caller does not use a neural network-based process. In other specific implementations, the base caller is a neural network-based base caller.

[0318] For a given sequencing cycle and a specific sub-pixel, the base caller is configured with logic to generate a base call for the given sequencing cycle and a specific sub-pixel by performing an image processing step and extracting the sub-pixel intensity data from the corresponding image set of the sequencing cycle. This is done for each sub-pixel in the sub-pixel and for each sequencing cycle in multiple sequencing cycles. Experiments have also been conducted with a quarter-subpixel division of a block image of an Illumina MiSeq sequencer with a resolution of 1800x1800 pixels. Sub-pixel base calling was performed for 50 sequencing cycles and ten blocks of one channel.

[0319] Figure 6 The initial center coordinates of the clusters identified by the base caller during sub-pixel base calling are shown. Figure 6 Also shown is an "origin sub-pixel" or "center sub-pixel" containing the initial center coordinates.

[0320] Figure 7 An example of merging sub-pixel base calls generated over multiple sequencing cycles to generate a so-called "cluster map" containing cluster metadata is depicted. In the illustrated implementation, a breadth-first search approach is used to merge sub-pixel base calls.

[0321] Figure 8a An example of a cluster map generated by merging sub-pixel base calls is shown. Figure 8b Depicted is an example of sub-pixel base calling. Figure 8b Also shown is one implementation of analyzing the sub-pixel base call sequence generated by sub-pixel base calling to generate a cluster map.

[0322] Sequencing images

[0323] Cluster metadata determination involves analyzing image data generated by sequencing instruments 102 (e.g., Illumina's iSeq, HiSeqX, HiSeq3000, HiSeq4000, HiSeq2500, NovaSeq6000, NextSeq, NextSeqDx, MiSeq, and MiSeqDx). According to one specific implementation, the following discussion outlines how image data is generated and what it depicts.

[0324] Base calling is the process by which the raw signal from the sequencing instrument 102 (i.e., intensity data extracted from the image) is decoded into a DNA sequence and a quality score. In one embodiment, the Illumina platform uses cyclic reversible termination (CRT) chemistry for base calling. The process relies on growing a nascent DNA chain complementary to a template DNA chain with modified nucleotides while tracking the emission signal of each newly added nucleotide. The modified nucleotides have a 3' removable block that anchors the fluorophore signal of the nucleotide type.

[0325] Sequencing is performed in repeated cycles, each of which consists of three steps: (a) extending the nascent chain by adding modified nucleotides; (b) exciting the fluorophore using one or more lasers of the optical system 104 and imaging it through different filters of the optical system 104, thereby generating a sequencing image 108; and (c) cleaving the fluorophore and removing the 3' block in preparation for the next sequencing cycle. The binding and imaging cycles are repeated until a specified number of sequencing cycles is reached, thereby defining the read length of all clusters. Using this method, each cycle interrogates a new position along the template chain.

[0326] The huge capacity of Illumina platform comes from its ability to perform and sense millions or even billions of clusters that experience CRT reactions simultaneously. The sequencing process occurs in a circulation cell 202, which is a small glass slide that keeps input DNA fragments during sequencing. The circulation cell 202 is connected to a high-throughput optical system 104, which includes microscope imaging, excitation lasers, and fluorescence filters. The circulation cell 202 includes a plurality of chambers referred to as grooves 204. The grooves 204 are physically separated from each other and can include different tag sequencing libraries that can be distinguished when there is no sample cross-contamination. Imaging device 106 (e.g., a solid-state imager, such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor) takes snapshots at multiple positions along the grooves 204, in a series of non-overlapping areas (referred to as blocks 206).

[0327] For example, each channel in the Illumina Genome Analyzer II has one hundred blocks, and each channel in the Illumina HiSeq 2000 has sixty-eight blocks. Block 206 accommodates hundreds of thousands to millions of clusters. An image generated from a block with clusters shown as bright spots is shown at 208. Cluster 302 includes approximately one thousand identical copies of the template molecule, but the clusters are of different sizes and shapes. Before the sequencing run, clusters are generated from the template molecules by bridge amplification of the input library. The purpose of amplification and cluster growth is to increase the intensity of the emitted signal because the imaging device 106 cannot reliably sense a single fluorophore. However, the physical distance of the DNA fragments within the cluster 302 is small, so the imaging device 106 perceives the cluster of fragments as a single point 302.

[0328] The output of the sequencing run is sequencing images 108 , each of which depicts the intensity emission of clusters on the block in the pixel domain for a particular combination of lane, block, sequencing cycle, and fluorophore ( 208A, 208C, 208T, 208G).

[0329] In one embodiment, the biosensor includes an array of photosensors. The photosensors are configured to sense information from corresponding pixel areas (e.g., reaction sites / wells / nanopores) on the detection surface of the biosensor. An analyte disposed in a pixel area is said to be associated with the pixel area, i.e., an associated analyte. In a sequencing cycle, the photosensors corresponding to the pixel areas are configured to detect / capture / sense the emission / photons from the associated analytes, and in response, generate pixel signals for each imaging channel. In one embodiment, each imaging channel corresponds to a filter wavelength band in a plurality of filter wavelength bands. In another embodiment, each imaging channel corresponds to an imaging event in a plurality of imaging events in a sequencing cycle. In yet another embodiment, each imaging channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter.

[0330] The pixel signals from the photosensors are transmitted to a signal processor coupled to the biosensor (e.g., via a communication port). For each sequencing cycle and each imaging channel, the signal processor generates an image whose pixels each depict / contain / indicate / represent / characterize the pixel signals obtained from the corresponding photosensors. Thus, a pixel in the image corresponds to: (i) a photosensor of the biosensor that generates the pixel signal depicted by the pixel, (ii) an associated analyte whose emissions are detected by the corresponding photosensor and converted into a pixel signal, and (iii) a pixel region on the detection surface of the biosensor that holds the associated analyte.

[0331] For example, consider a sequencing run using two different imaging channels (i.e., a red channel and a green channel). Then, in each sequencing cycle, the signal processor generates a red image and a green image. Thus, for a series of k sequencing cycles in a sequencing run, a sequence with k pairs of red and green images is generated as output.

[0332] Pixels in the red and green images (i.e., different imaging channels) correspond one-to-one within a sequencing cycle. This means that corresponding pixels in a pair of red and green images depict intensity data for the same associated analyte, despite being in different imaging channels. Similarly, pixels on a pair of red and green images correspond one-to-one between sequencing cycles. This means that corresponding pixels in different pairs of red and green images depict intensity data for the same associated analyte, despite being acquired for different acquisition events / time steps (sequencing cycles) of a sequencing run.

[0333] Corresponding pixels in the red and green images (i.e., different imaging channels) can be considered as pixels of a "per-cycle image" expressing intensity data in the first red channel and the second green channel. A per-cycle image whose pixels depict the pixel signals of a subset of a pixel region (i.e., an area (patches) of the detection surface of the biosensor) is referred to as a "per-cycle patch image." Patches extracted from the per-cycle patch image are referred to as "per-cycle image patches." In one embodiment, patch extraction is performed by an input preparer.

[0334] The image data includes a sequence of image patches for each cycle generated for a series of k sequencing cycles of a sequencing run. The pixels in the image patch for each cycle contain intensity data for the associated analyte, and intensity data for one or more imaging channels (e.g., a red channel and a green channel) are obtained by corresponding light sensors that are configured to detect emission from the associated analyte. In one embodiment, when base calling is to be performed on a single target cluster, the image patch for each cycle is centered on a center pixel containing intensity data for the target associated analyte, and the non-center pixels in the image patch for each cycle contain intensity data for associated analytes adjacent to the target associated analyte. In one embodiment, the image data is prepared by an input preparer.

[0335] Sub-pixel base calling

[0336] The disclosed technology accesses a series of image sets generated during a sequencing run. The image set includes sequencing images 108. Each image set in the series of image sets is captured during a corresponding sequencing cycle of the sequencing run. Each image (or sequencing image) in the series of image sets captures a cluster on a block of the flow cell and its surrounding background.

[0337] In one implementation, the sequencing run utilizes a four-channel chemistry, and each image set has four images. In another implementation, the sequencing run utilizes a two-channel chemistry, and each image set has two images. In yet another implementation, the sequencing run utilizes a single-channel chemistry, and each image set has two images. In other implementations, each image set has only one image.

[0338] The sequencing image 108 in the pixel domain is first converted to the sub-pixel domain by the sub-pixel addresser 110 to generate a sequencing image 112 in the sub-pixel domain. In one embodiment, each pixel in the sequencing image 108 is divided into sixteen sub-pixels 502. Therefore, in one embodiment, the sub-pixels 502 are quarter sub-pixels. In another embodiment, the sub-pixels 502 are half sub-pixels. Therefore, each sequencing image 112 in the sub-pixel domain has a plurality of sub-pixels 502.

[0339] The sub-pixels are then fed individually as input to a base caller 114 to obtain a base call from the base caller 114 that classifies each of the sub-pixels into one of the four bases (A, C, T, and G). This produces a sequence of base calls 116 for each of the sub-pixels in the multiple sequencing cycles of the sequencing run. In one embodiment, the sub-pixels 502 are identified by the base caller 114 based on their integer coordinates or non-integer coordinates. By tracking the emission signal from the sub-pixels 502 across the set of images generated during the multiple sequencing cycles, the base caller 114 recovers the underlying DNA sequence for each sub-pixel. An example of this is shown in FIG. Figure 8b Shown in.

[0340] In other implementations, the disclosed techniques obtain base calls from base caller 114 that classify each of the sub-pixels as one of five bases (A, C, T, G, and N). In such implementations, an N base call indicates an undecided base call, which is typically due to a low extracted intensity level.

[0341] Some examples of base callers 114 include non-neural network-based Illumina products such as RTA (Real-Time Analysis), the Firecrest program in the Genome Analyzer analysis pipeline, the IPAR (Integrated Primary Analysis and Reporting) machine, and the OLB (Offline Base Caller). For example, base callers 114 generate base call sequences by interpolating sub-pixel intensities using intensity extraction methods including nearest neighbor intensity extraction, Gaussian-based intensity extraction, intensity extraction based on the average of a 2×2 sub-pixel area, intensity extraction based on the brightest point in a 2×2 sub-pixel area, intensity extraction based on the average of a 3×3 sub-pixel area, bilinear intensity extraction, bicubic intensity extraction, and / or weighted area coverage-based intensity extraction. These techniques are described in detail in the Appendix entitled "Intensity Extraction Methods."

[0342] In other implementations, the base caller 114 can be a neural network-based base caller, such as the neural network-based base caller 1514 disclosed herein.

[0343] The sub-pixel-by-sub-pixel base call sequence 116 is then fed as input to a searcher 118. The searcher 118 searches for substantially matching base call sequences of contiguous sub-pixels. A base call sequence of contiguous sub-pixels is "substantially matched" when a predetermined portion of the base calls match in positional order (e.g., there are ≥41 matches in 45 cycles, ≤4 mismatches in 45 cycles, ≤4 mismatches in 50 cycles, or ≤2 mismatches in 34 cycles).

[0344] The searcher 118 then generates a cluster map 802 that identifies clusters as disjoint regions of contiguous sub-pixels that share a substantially matching base call sequence, such as 804a-d. The terms "disjoint" and "non-overlapping" are used interchangeably herein. The search involves base calling sub-pixels that comprise part of a cluster to allow the called sub-pixels to be linked to the contiguous sub-pixels with which they share a substantially matching base call sequence. In some implementations, the searcher 118 requires that at least some of the disjoint regions have a predetermined minimum number of sub-pixels (e.g., more than 4, 6, or 10 sub-pixels) to be processed as a cluster.

[0345] In some implementations, base caller 114 also identifies the initial center coordinates of the cluster. The sub-pixel containing the initial center coordinates is called the origin sub-pixel. Figure 6 , some exemplary initial center coordinates (604a-c) and corresponding origin sub-pixels (606a-c) identified by base caller 114 are shown in FIG. However, identification of the origin sub-pixel (the initial center coordinate of the cluster) is not required, as described below. In some implementations, searcher 118 performs a breadth-first search to identify a substantially matching base call sequence of sub-pixels, starting with origin sub-pixels 606a-c and continuing with consecutively adjacent non-origin sub-pixels 702a-c. This is also optional, as described below.

[0346] Cluster Map

[0347] Figure 8a An example of a cluster map 802 generated by merging sub-pixel base calls is shown. The cluster map identifies multiple non-intersecting regions ( Figure 8a Each disjoint region comprises a set of non-overlapping adjacent sub-pixels representing a corresponding cluster on the tile (via sub-pixel base calling from its sequencing image and generation of a cluster map for it). The region between the disjoint regions represents the background on the tile. The sub-pixels in the background region are referred to as "background sub-pixels". The sub-pixels in the disjoint regions are referred to as "cluster sub-pixels" or "cluster interior sub-pixels". In this discussion, the origin sub-pixels are those sub-pixels where the initial center cluster coordinates determined by RTA or another base caller are located.

[0348] The origin sub-pixel contains the initial center cluster coordinates. This means that the area covered by the origin sub-pixel includes coordinate positions that coincide with the initial center cluster coordinate positions. Since cluster map 802 is a logical sub-pixel image, the origin sub-pixels are some sub-pixels in the cluster map.

[0349] The search to identify clusters with substantially matching base call sequences for sub-pixels does not need to begin with identifying the origin sub-pixel (the initial center coordinate of the cluster), as the search can be performed on all sub-pixels and can begin at any sub-pixel (e.g., the 0,0 sub-pixel or any random sub-pixel). Therefore, since each sub-pixel is evaluated to determine whether it shares a substantially matching base call sequence with another adjacent sub-pixel, the search is not dependent on the origin sub-pixel; the search can begin at any sub-pixel.

[0350] Regardless of whether the origin sub-pixel is used, some clusters are identified that do not contain the origin sub-pixel (the initial center coordinate of the cluster) predicted by the base caller 114. Some examples of clusters identified by merging sub-pixel base calling and that do not contain the origin sub-pixel are Figure 8a 812c, 812d, and 812e in FIG. Thus, the disclosed technique identifies additional or additional clusters whose centers have not yet been identified by base caller 114. Thus, using base caller 114 to identify the origin sub-pixel (the initial center coordinates of the cluster) is optional and not required for searching adjacent sub-pixels for substantially matching base call sequences.

[0351] In one specific implementation, the origin sub-pixel identified by base caller 114 (the initial center coordinates of the cluster) is first used to identify a first set of clusters (by identifying substantially matching base call sequences of adjacent sub-pixels). Then, sub-pixels that are not part of the first set of clusters are used to identify a second set of clusters (by identifying substantially matching base call sequences of adjacent sub-pixels). This allows the disclosed technology to identify additional or extra clusters whose centers have not yet been identified by base caller 114. Finally, sub-pixels that are not part of the first and second sets of clusters are identified as background sub-pixels.

[0352] Figure 8b Depicted is an example of sub-pixel base calling. Figure 8b In , each sequencing cycle has an image set with four different images (ie, A, C, T, G images) captured using four different wavelength bands (images / imaging channels) and four different fluorescent dyes (one for each base).

[0353] In this example, the pixels in the image are divided into sixteen sub-pixels. The sub-pixels are then independently base called by base caller 114 in each sequencing cycle. To base call a given sub-pixel in a particular sequencing cycle, base caller 114 uses the intensity of the given sub-pixel in each of the A, C, T, G images. For example, the intensity in the image area covered by sub-pixel 1 in each of the A, C, T, G images for cycle 1 is used to base call sub-pixel 1 in cycle 1. For sub-pixel 1, these image areas include the upper-left one-sixteenth of the area of ​​the corresponding upper-left pixel in each of the A, C, T, G images for cycle 1. Similarly, the intensity in the image area covered by sub-pixel m in each of the A, C, T, G images for cycle n is used to base call sub-pixel m in cycle n. For sub-pixel m, these image areas include the lower-right one-sixteenth of the area of ​​the corresponding lower-right pixel in each of the A, C, T, G images for cycle 1.

[0354] This process generates a subpixel-by-subpixel base call sequence 116 across multiple sequencing cycles. A searcher 118 then evaluates pairs of adjacent subpixels to determine whether they have substantially matching base call sequences. If so, the pair of subpixels is stored in a cluster map 802 as being part of the same cluster in a disjoint region. If not, the pair of subpixels is stored in a cluster map 802 as not being part of the same disjoint region. Thus, the cluster map 802 identifies groups of adjacent subpixels for which the base calls for the subpixels substantially match across multiple cycles. The cluster map 802 thus uses information from multiple cycles to provide a plurality of clusters with high confidence, wherein each of the plurality of clusters provides sequence data for a single DNA strand.

[0355] The cluster metadata generator 122 then processes the cluster map 802 to determine cluster metadata, including determining the spatial distribution of clusters based on disjoint regions, and determining the center (810a), shape, size, background, and / or boundaries ( Figure 9 ).

[0356] In some implementations, the cluster metadata generator 122 identifies as background those sub-pixels in the cluster map 802 that do not belong to any of the disjoint regions and therefore do not contribute to any cluster. Such sub-pixels are referred to as background sub-pixels 806a-c.

[0357] In some implementations, the cluster map 802 identifies cluster boundary portions 808a-c between two adjacent sub-pixels whose base call sequences do not substantially match.

[0358] The cluster map is stored in memory (e.g., cluster map data storage 120) to serve as ground truth for training classifiers such as neural network-based template generator 1512 and neural network-based base caller 1514. Cluster metadata may also be stored in memory (e.g., cluster metadata storage 124).

[0359] Figure 9 Another example of a cluster map identifying cluster metadata including the spatial distribution of clusters, as well as cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries is shown.

[0360] Center of mass (COM)

[0361] Figure 10 How to calculate the center of mass (COM) of disjoint areas in a cluster map is shown. The COM can be used as a "modified" or "improved" center of the corresponding cluster in downstream processing.

[0362] In some implementations, the centroid generator 1004 determines the cluster's hyperlocalization center coordinates 1006 on a cluster-by-cluster basis by calculating the centroid of the disjoint regions of the cluster map as the average of the coordinates of the corresponding adjacent sub-pixels that form the disjoint regions. The centroid generator then stores the cluster's hyperlocalization center coordinates cluster-by-cluster in a memory to be used as ground truth for training the classifier.

[0363] In some implementations, the sub-pixel classifier identifies the centroid sub-pixel 1008 in the disjoint regions 804a-d of the cluster map 802 on a cluster-by-cluster basis at the hyper-located center coordinates 1006 of the cluster.

[0364] In other specific implementations, the cluster map is upsampled using interpolation, and the upsampled cluster map is stored in a memory to be used as a ground truth for training the classifier.

[0365] Attenuation Factor and Attenuation Map

[0366] Figure 11One embodiment of a method for calculating a weighted attenuation factor for a subpixel based on the Euclidean distance of the subpixel from the centroid (COM) of the disjoint region to which the subpixel belongs is depicted. In the illustrated embodiment, the weighted attenuation factor provides the highest value to the subpixel containing the COM, and the value obtained for the subpixel farther from the COM is smaller. The weighted attenuation factor is used to derive a ground truth attenuation map 1204 from the cluster map generated according to the subpixel base calling described above. The ground truth attenuation map 1204 comprises an array of cells and assigns at least one output value to each cell in the array. In some embodiments, the cells are subpixels, and an output value is assigned to each subpixel based on the weighted attenuation factor. The ground truth attenuation map 1204 is then used as the ground truth for training the disclosed neural network-based template generator 1512. In some embodiments, information from the ground truth attenuation map 1204 is also used to prepare input for the disclosed neural network-based base caller 1514.

[0367] Figure 12 One implementation of an exemplary ground truth decay map 1204 derived from an exemplary cluster map generated by sub-pixel base calling as described above is shown. In some implementations, in the upsampled cluster map, a value is assigned to each adjacent sub-pixel in a disjoint region on a cluster-by-cluster basis based on a decay factor 1102 that is proportional to the distance 1106 of the adjacent sub-pixel from the centroid sub-pixel 1104 in the disjoint region to which the adjacent sub-pixel belongs.

[0368] Figure 12 A ground truth attenuation map 1204 is depicted. In one implementation, the sub-pixel value is an intensity value normalized between zero and one. In another implementation, the same predetermined value is assigned to all sub-pixels identified as background in the upsampled cluster map. In some implementations, the predetermined value is a zero intensity value.

[0369] In some implementations, a ground truth attenuation map 1204 is generated by a ground truth attenuation map generator 1202 based on the upsampled cluster map, wherein the ground truth attenuation map represents adjacent sub-pixels in a disjoint region and sub-pixels identified as background based on the assigned values ​​of the adjacent sub-pixels. The ground truth attenuation map 1204 is stored in a memory to serve as a ground truth for training a classifier. In one implementation, each sub-pixel in the ground truth attenuation map 1204 has a value normalized between zero and one.

[0370] Ternary (three-category) mapping

[0371] Figure 13A specific implementation of deriving a ground truth ternary map 1304 from a cluster map is shown. The ground truth ternary map 1304 comprises an array of cells and assigns at least one output value to each cell in the array. As the name implies, the ternary map implementation of the ground truth ternary map 1304 assigns three output values ​​to each cell in the array such that for each cell, the first output value corresponds to a classification label or score for the background class, the second output value corresponds to a classification label or score for the cluster center class, and the third output value corresponds to a classification label or score for the cluster / cluster interior class. The ground truth ternary map 1304 is used as ground truth data for training a neural network-based template generator 1512. In some specific implementations, information from the ground truth ternary map 1304 is also used to prepare input for a neural network-based base caller 1514.

[0372] Figure 13 An exemplary ground truth ternary map 1304 is depicted. In another embodiment, in the upsampled cluster map, the ground truth ternary map generator 1302 classifies adjacent sub-pixels in the disjoint region on a cluster-by-cluster basis as cluster-internal sub-pixels belonging to the same cluster, classifies centroid sub-pixels as cluster-center sub-pixels, and classifies sub-pixels that do not belong to any cluster as background sub-pixels. In some embodiments, the classifications are stored in the ground truth ternary map 1304. These classifications and the ground truth ternary map 1304 are stored in memory for use as ground truth for training the classifier.

[0373] In other embodiments, the coordinates of the cluster interior sub-pixels, cluster center sub-pixels, and background sub-pixels are stored in a memory cluster by cluster to serve as ground truth for training the classifier. The coordinates are then scaled down by a factor used to upsample the cluster map. The scaled down coordinates are then stored in a memory cluster by cluster to serve as ground truth for training the classifier.

[0374] In other embodiments, the ground truth ternary map generator 1302 uses the cluster map to generate ternary ground truth data 1304 based on the upsampled cluster map. The ternary ground truth data 1304 labels background sub-pixels as belonging to the background class, cluster center sub-pixels as belonging to the cluster center class, and cluster interior sub-pixels as belonging to the cluster interior class. In some visualization embodiments, color coding can be used to depict and distinguish different class labels. The ternary ground truth data 1304 is stored in a memory for use as ground truth for training a classifier.

[0375] Binary (two-class) mapping

[0376] Figure 14Shown is a specific implementation of deriving a reference truth binary map 1404 from the cluster map. Binary map 1404 comprises an array of cells and distributes at least one output value to each cell in the array. As the name implies, the binary map distributes two output values ​​to each cell in the array so that for each cell, the first output value corresponds to a classification mark or score for the cluster center class, and the second output value corresponds to a classification mark or score for the non-center class. The binary map is used as the reference truth data for training a template generator 1512 based on a neural network. In some specific implementations, information from the binary map is also used to prepare input for a base caller 1514 based on a neural network.

[0377] Figure 14 The ground truth binary map 1404 is depicted. The ground truth binary map generator 1402 uses cluster mapping Figure 120 Based on the upsampled cluster map, binary ground truth data 1404 is generated. The binary ground truth data 1404 labels the cluster center sub-pixels as belonging to the cluster center class and labels all other sub-pixels as belonging to the non-center class. The binary ground truth data 1404 is stored in memory to be used as ground truth for training the classifier.

[0378] In some implementations, the disclosed technology generates cluster maps for multiple blocks of a flow cell. Figure 120 , store these cluster maps in memory, and based on these cluster maps Figure 120 To determine the spatial distribution of clusters in the block, and determine the shape and size of these clusters. Then, the cluster map of the upsampled clusters in the block is Figure 120 In the disclosed technique, sub-pixels are classified into cluster-inside sub-pixels, cluster-center sub-pixels, and background sub-pixels belonging to the same cluster, cluster by cluster. The disclosed technique then stores the classification in a memory to serve as ground truth for training a classifier, and stores the coordinates of the cluster-inside sub-pixels, cluster-center sub-pixels, and background sub-pixels cluster by cluster across tiles in a memory to serve as ground truth for training a classifier. Furthermore, the disclosed technique reduces the coordinates by a factor used to upsample the cluster map, and stores the reduced coordinates in a memory cluster by cluster across tiles to serve as ground truth for training a classifier.

[0379] In some implementations, the flow cell has at least one patterned surface having an array of pores that are occupied by clusters. In such implementations, the disclosed techniques determine, based on the determined shapes and sizes of the clusters: (1) which of the pores are substantially occupied by at least one cluster, (2) which of the pores are minimally occupied, and (3) which of the pores are co-occupied by multiple clusters. This allows for the determination of corresponding metadata for multiple clusters that co-occupy the same pore, i.e., the center, shape, and size of two or more clusters that share the same pore.

[0380] In some specific implementations, the solid support on which the sample is amplified into clusters includes a patterned surface." Patterned surface" refers to the arrangement of different regions in the exposed layer of the solid support or on the exposed layer. For example, one or more regions in these regions can be features in which there are one or more amplification primers. Features can be separated by gap regions in which there are no amplification primers. In some specific implementations, the pattern can be the xy format of features in the form of rows and columns. In some specific implementations, the pattern can be the repeated arrangement of features and / or gap regions. In some specific implementations, the pattern can be the random arrangement of features and / or gap regions. Exemplary patterned surfaces that can be used in the methods and compositions described herein are described in U.S. Patent No. 8,778,849, U.S. Patent No. 9,079,148, U.S. Patent No. 8,778,848 and U.S. Patent Publication No. 2014 / 0243224, each of which is incorporated herein by reference.

[0381] In some embodiments, the solid support includes an array of holes or depressions in its surface. This can be manufactured using a variety of techniques as are generally known in the art, including but not limited to photolithography, imprinting techniques, molding techniques, and micro-etching techniques. Those skilled in the art will appreciate that the technique used will depend on the composition and shape of the array substrate.

[0382] The features in the patterned surface can be wells (e.g., micropores or nanopores) in a well array on a glass, silicon, plastic, or other suitable solid support having a patterned and covalently attached gel such as poly(N-(5-azidoacetamidopentyl)acrylamide-co-acrylamide) (PAZAM, see, e.g., U.S. Patent Publication Nos. 2013 / 184796, WO 2016 / 066586, and WO 2015-002813, each of which is incorporated herein by reference in its entirety). The method produces a gel pad for sequencing that can be stable over sequencing runs with a large number of cycles. The covalent attachment of the polymer to the wells helps maintain the gel as a structured feature during multiple uses and throughout the life of the structured substrate. However, in many implementations, the gel need not be covalently attached to the wells. For example, under some conditions, silane-free acrylamide (SFA, see, eg, US Pat. No. 8,563,477, which is incorporated herein by reference in its entirety) that is not covalently attached to any portion of the structured substrate can be used as the gel material.

[0383] In a particular embodiment, a structured substrate can be made by patterning a solid support material with pores (e.g., micropores or nanopores), coating the patterned support with a gel material (e.g., PAZAM, SFA, or a chemically modified variant thereof, such as an azidated version of SFA (azido-SFA)), and polishing the gel-coated support, for example by chemical or mechanical polishing, so that the gel remains in the pores while removing or inactivating substantially all of the gel from the interstitial regions on the surface of the structured substrate between the pores. Primer nucleic acids can be attached to the gel material. A solution of target nucleic acids (e.g., a fragmented human genome) can then be contacted with the polished substrate such that a single target nucleic acid will be seeded into a single pore through interaction with a primer attached to the gel material; however, due to the absence or inactivation of the gel material, the target nucleic acid will not occupy the interstitial regions. Amplification of the target nucleic acid will be confined to the pores because the absence or inactivation of the gel in the interstitial regions will prevent the outward migration of the growing nucleic acid colony. The process can be fabricated conveniently and scalably, and utilizes micro- or nanofabrication methods.

[0384] As used herein, term " flow cell " refers to the chamber comprising a solid surface, and one or more fluid reagents can flow through this solid surface. The example description of the flow cell and related fluid system and detection platform that can be easily used in the method for the present disclosure is such as following: Bentley et al., Nature, 456:53-59 (2008); WO 04 / 018497, US 7,057,026, WO 91 / 06678; WO 07 / 123744; US 7,329,492, US 7,211,414, US 7,315,019, US 7,405,281 and US 2008 / 0108082, each of these documents is incorporated herein by reference.

[0385] Throughout this disclosure, when referring to amplification primers, the terms "P5" and "P7" are used. It should be understood that any suitable amplification primer can be used in the method presented herein, and the use of P5 and P7 is only an exemplary embodiment. Using amplification primers such as P5 and P7 on a flow cell is known in the art, as illustrated by the disclosures of WO 2007 / 010251, WO 2006 / 064199, WO2005 / 065814, WO 2015 / 106941, WO 1998 / 044151, and WO 2000 / 018957, each of which is incorporated herein by reference in its entirety. For example, any suitable forward amplification primer, whether immobilized or in solution, can be used in the method presented herein, for hybridization with complementary sequences and amplified sequences. Similarly, any suitable reverse amplification primer, whether immobilized or in solution, can be used in the methods presented herein for hybridizing with complementary sequences and amplifying sequences. Those skilled in the art will understand how to design and use primer sequences suitable for capturing and amplifying the nucleic acids presented herein.

[0386] In some implementations, the flow cell has at least one non-patterned surface, and the clusters are non-uniformly dispersed on the non-patterned surface.

[0387] In some implementations, the density of the clusters is about 100,000 clusters / mm 2 to approximately 1,000,000 clusters / mm 2 In other embodiments, the density of the clusters is about 1,000,000 clusters / mm 2 to approximately 10,000,000 clusters / mm 2 within the range.

[0388] In one implementation, the initial center coordinates of the clusters determined by the base callers are defined in the template image of the tile. In some implementations, the pixel resolution, image coordinate system, and measurement scale of the image coordinate system are the same for both the template image and the image.

[0389] In another specific implementation, the disclosed technology involves determining metadata about clusters on a block of a flow cell. First, the disclosed technology accesses (1) a set of images of the block captured during a sequencing run and (2) initial center coordinates of the clusters determined by base callers.

[0390] Then, for each image set, the disclosed technique obtains base calls that classify the following into one of four bases: (1) the origin sub-pixel containing the initial center coordinates, and (2) a predetermined neighborhood of adjacent sub-pixels that are contiguously adjacent to a corresponding one of the origin sub-pixels. This produces a sequence of base calls for each of the origin sub-pixels and each of the predetermined neighborhoods of adjacent sub-pixels. The predetermined neighborhood of adjacent sub-pixels can be an m×n sub-pixel patch centered on the sub-pixel containing the origin sub-pixel. In one embodiment, the sub-pixel patch is a 3×3 sub-pixel. In other embodiments, the image patch can have any size, such as 5×5, 15×15, 20×20, etc. In other embodiments, the predetermined neighborhood of adjacent sub-pixels can be an n-connected sub-pixel neighborhood centered on the sub-pixel containing the origin sub-pixel.

[0391] In one specific implementation, the disclosed technology identifies those sub-pixels in the cluster map that do not belong to any of the disjoint regions as background.

[0392] The disclosed technique then generates a cluster map that identifies clusters as disjoint regions of contiguous sub-pixels that: (a) are contiguously adjacent to at least some of the corresponding ones of the origin sub-pixels, and (b) share a substantially matching base call sequence of one of the four bases with at least some of the corresponding ones of the origin sub-pixels.

[0393] The disclosed technology then stores the cluster map in a memory and determines the shape and size of the clusters based on the disjoint areas in the cluster map. In other implementations, the center of the cluster is also determined.

[0394] Generate training data for the template generator

[0395] Figure 15 is a block diagram illustrating one specific implementation of generating training data for training the neural network-based template generator 1512 and the neural network-based base caller 1514.

[0396] Figure 16Features of the disclosed training examples used to train the neural network-based template generator 1512 and the neural network-based base caller 1514 are shown. Each training example corresponds to a block and is labeled with a corresponding ground truth data representation. In some implementations, the ground truth data representation is a ground truth mask or ground truth map that identifies ground truth cluster metadata in the form of a ground truth decay map 1204, a ground truth ternary map 1304, or a ground truth binary map 1404. In some implementations, multiple training examples correspond to the same block.

[0397] In one specific implementation, the disclosed technology involves generating training data 1504 for neural network-based template generation and base calling. First, the disclosed technology accesses a plurality of images 108 of a flow cell 202 captured over multiple cycles of a sequencing run. The flow cell 202 has a plurality of tiles. In the plurality of images 108, each tile in the tile has a sequence of image sets generated over the plurality of cycles. Each image in the sequence of image sets 108 depicts the intensity emission of a cluster 302 and its surrounding background 304 at a particular tile in the tile at a particular cycle.

[0398] Then, the training set constructor 1502 constructs a training set 1504 having a plurality of training examples. Figure 16 As shown, each training example corresponds to a specific block in the block and includes image data from at least some of the image sets in the sequence of image sets 1602 for the specific block in the block. In one implementation, the image data includes images from at least some of the image sets in the sequence of image sets 1602 for the specific block in the block. For example, the images may have a resolution of 1800×1800. In other implementations, the images may have any resolution, such as 100×100, 3000×3000, 10000×10000, and so on. In other implementations, the image data includes at least one image patch from each image. In one implementation, the image patch covers a portion of the specific block in the block. In one example, the image patch may have a resolution of 20×20. In other implementations, the image patch may have any resolution, such as 50×50, 70×70, 90×90, 100×100, 3000×3000, 10000×10000, and so on.

[0399] In some implementations, the image data includes an upsampled representation of the image patch. The upsampled representation can have a resolution of, for example, 80×80. In other implementations, the upsampled representation can have any resolution, such as 50×50, 70×70, 90×90, 100×100, 3000×3000, 10000×10000, etc.

[0400] In some implementations, the plurality of training examples correspond to the same particular one of the tiles and each include, as image data, a different image patch from each image in each of at least some of the image sets in the sequence of image sets 1602 for the same particular one of the tiles. In such implementations, at least some of the different image patches overlap with each other.

[0401] Then, a ground truth generator 1506 generates at least one ground truth data representation for each of the training examples. The ground truth data representation identifies at least one of a spatial distribution of clusters and their surrounding background on a particular one of the blocks, and identifies at least one of a cluster shape, a cluster size, and / or a cluster boundary and / or a cluster center, the intensity emissions of the clusters and their surrounding background being depicted by the image data.

[0402] In one embodiment, the reference truth data represents identifying a cluster as a non-intersecting area of ​​adjacent sub-pixels, identifying the center of the cluster as a centroid sub-pixel within the corresponding non-intersecting area in the non-intersecting area, and identifying its surrounding background as sub-pixels that do not belong to any of the non-intersecting areas.

[0403] In one implementation, the ground truth data representation has an upsampled resolution of 80 × 80. In other implementations, the ground truth data representation may have any resolution, such as 50 × 50, 70 × 70, 90 × 90, 100 × 100, 3000 × 3000, 10000 × 10000, etc.

[0404] In one embodiment, the ground truth data indicates that each sub-pixel is identified as a cluster center or a non-cluster center. In another embodiment, the ground truth data indicates that each sub-pixel is identified as a cluster interior, a cluster center, or surrounding background.

[0405] In some implementations, the disclosed technology stores the training examples in the training set 1504 and the associated ground truth data 1508 in memory as training data 1504 for training the neural network-based template generator 1512 and the neural network-based base caller 1514. The training is performed by a trainer 1510.

[0406] In some implementations, the disclosed technology generates training data for a variety of flow cells, sequencing instruments, sequencing protocols, sequencing chemistries, sequencing reagents, and cluster densities.

[0407] Neural network based template generator

[0408] In an inference or production implementation, the disclosed technology uses peak detection and segmentation to determine cluster metadata. The disclosed technology processes input image data 1702 derived from a series of image sets 1602 through a neural network 1706 to generate an alternative representation 1708 of the input image data 1702. For example, an image set may be for a particular sequencing cycle and include four images (A, C, T, and G images), one image for each imaging channel. Then, for a sequencing run with fifty sequencing cycles, there will be fifty such image sets, i.e., a total of 200 images. When temporarily arranged, the fifty image sets, each of which has four images, will form a series of image sets 1602. In some implementations, an image patch of a specific size is extracted from each image in the fifty image sets, thereby forming fifty image patch sets, each of which has four image patches, and in one implementation, this is the input image data 1702. In other specific implementations, the input image data 1702 includes image patch sets where each image patch set has four image patches, and the image patch sets are used for less than fifty sequencing cycles, i.e., only one, two, three, fifteen, or twenty sequencing cycles.

[0409] Figure 17 One embodiment of processing input image data 1702 through a neural network-based template generator 1512 and generating an output value for each element in an array is shown. In one embodiment, the array is an attenuation map 1716. In another embodiment, the array is a ternary map 1718. In yet another embodiment, the array is a binary map 1720. Thus, the array can represent one or more attributes of each of the plurality of locations represented in the input image data 1702.

[0410] Unlike the template generator trained using the structure in the previous figure, the ground truth decay map 1204, the ground truth ternary map 1304, and the ground truth binary map 1404, the decay map 1716, the ternary map 1718, and / or the binary map 1720 are generated by forward propagation of the trained neural network-based template generator 1512. Forward propagation can be performed during training or during inference. During training, due to gradient updates based on backward propagation, the decay map 1716, the ternary map 1718, and the binary map 1720 (i.e., cumulatively, the output 1714) gradually match or approach the ground truth decay map 1204, the ground truth ternary map 1304, and the ground truth binary map 1404, respectively.

[0411] According to one embodiment, the size of the image array analyzed during inference depends on the size of the input image data 1702 (e.g., the same size or an enlarged or reduced version). Each cell may represent a pixel, sub-pixel, or super-pixel. The cell-by-cell output value of the array may characterize / represent / indicate the attenuation map 1716, the ternary map 1718, or the binary map 1720. In some embodiments, the input image data 1702 is also an array of cells at pixel, sub-pixel, or super-pixel resolution. In such embodiments, the neural network-based template generator 1512 uses semantic segmentation techniques to generate an output value for each cell in the input array. Additional details about the input image data 1702 can be found in Figure 21b 、 Figure 22 、 Figure 23 and Figure 24 and discussions about these figures.

[0412] In some implementations, the neural network-based template generator 1512 is a fully convolutional network, such as the fully convolutional network described in J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation" in CVPR (2015), which is incorporated herein by reference. In other implementations, the neural network-based template generator 1512 is a U-Net network with skip connections between the decoder and the encoder, such as the U-Net network described in Ronneberger O, Fischer P, Brox T., "U-net: Convolutional networks for biomedical image segmentation" in Med. Image Comput. Comput. Assist. Interv. (2015) (available from http: / / link.springer.com / chapter / 10.1007 / 978-3-319-24574-4_28), which is incorporated herein by reference. The U-Net architecture is similar to an autoencoder with two main substructures: 1) an encoder that takes an input image and reduces its spatial resolution through multiple convolutional layers to create a representation code. 2) a decoder that takes the representation code and increases the spatial resolution back to produce a reconstructed image as output. U-Net introduces two innovations to this architecture: first, setting the objective function to reconstruct the segmentation mask using a loss function; second, using skip connections to connect the convolutional layers of the encoder to the corresponding layers of the decoder with the same resolution. In another embodiment, the neural network-based template generator 1512 is a deep fully convolutional segmentation neural network with an encoder subnetwork and a corresponding decoder network. In such embodiments, the encoder subnetwork includes an encoder hierarchy, and the decoder subnetwork includes a decoder hierarchy that maps the low-resolution encoder feature map to the full-input resolution feature map. Additional details about the segmentation network can be found in the Appendix titled "Segmentation Network".

[0413] In one implementation, the neural network-based template generator 1512 is a convolutional neural network. In another implementation, the neural network-based template generator 1512 is a recurrent neural network. In yet another implementation, the neural network-based template generator 1512 is a residual neural network with residual blocks and residual connections. In another implementation, the neural network-based template generator 1512 is a combination of a convolutional neural network and a recurrent neural network.

[0414] Those skilled in the art will appreciate that the neural network-based template generator 1512 (i.e., the neural network 1706 and / or the output layer 1710) can use various padding and stride configurations. It can use different output functions (e.g., classification or regression) and may or may not include one or more fully connected layers. It can use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or dilated convolution, transposed convolution, depth-separable convolution, point-by-point convolution, 1×1 convolution, grouped convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It can use one or more loss functions, such as logistic regression / logarithmic loss function, multi-class cross entropy / softmax loss function, binary cross entropy loss function, mean squared error loss function, L1 loss function, L2 loss function, smoothed L1 loss function, and Huber loss function. It can use any parallelism, efficiency and compression scheme, such as TFRecords, compression encoding (e.g., PNG), sharpening, parallel detection of map transformations, batching, prefetching, model parallelism, data parallelism and synchronous / asynchronous SGD. It can include upsampling layers, downsampling layers, recurrent connections, gate and gate memory units (such as LSTM or GRU), residual blocks, residual connections, high-speed connections, skip connections, peephole connections, activation functions (e.g., nonlinear transformation functions such as rectified linear units (ReLU), leaky ReLU, exponential lining units (ELU), sigmoid and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout layers, pooling layers (e.g., max or average pooling), global average pooling layers and attention mechanisms.

[0415] In some implementations, each image in the sequence of image set 1602 covers a tile and depicts the intensity emission of a cluster on the tile and its surrounding background, captured for a particular imaging channel during a particular sequencing cycle among multiple sequencing cycles of a sequencing run performed on the flow cell. In one implementation, input image data 1702 includes at least one image patch from each image in the sequence of images in image set 1602. In such implementations, the image patch covers a portion of the tile. In one example, the image patch has a resolution of 20×20. In other cases, the resolution of the image patch may range from 20×20 to 10,000×10,000. In another implementation, input image data 1702 includes an upsampled sub-pixel resolution representation of the image patch from each image in the sequence of images in image set 1602. In one example, the upsampled sub-pixel representation has a resolution of 80×80. In other cases, the resolution of the upsampled sub-pixel representation may range from 80×80 to 10,000×10,000.

[0416] Input image data 1702 has a cell array 1704 depicting a cluster and its surrounding background. For example, an image set may be used for a particular sequencing cycle and include four images (A, C, T, and G images), one for each imaging channel. For a sequencing run with fifty sequencing cycles, there would then be fifty such image sets, for a total of 200 images. When temporarily arranged, these fifty image sets, each with four images, would form a series of image sets 1602. In some implementations, image patches of a specific size are extracted from each image in the fifty image sets, thereby forming fifty image patch sets, each with four image patches, and in one implementation, this is input image data 1702. In other implementations, input image data 1702 includes image patch sets, each with four image patches, for fewer than fifty sequencing cycles, i.e., only one, two, three, fifteen, or twenty sequencing cycles. Another representation is a feature map. When the neural network is a convolutional neural network, the feature map may be a convolutional feature or a convolutional representation. When the neural network is a recurrent neural network, the feature map may be a hidden state feature or a hidden state representation.

[0417] The disclosed technology then processes the alternative representation 1708 through an output layer 1710 to generate an output 1714 having an output value 1712 for each cell in the array 1704. The output layer can be a classification layer that produces a cell-by-cell output value, such as a softmax or sigmoid. In one specific implementation, the output layer is a ReLU layer or any other activation function layer that produces a cell-by-cell output value.

[0418] In one implementation, the elements in the input image data 1702 are pixels, thus generating pixel-by-pixel output values ​​1712 in the output 1714. In another implementation, the elements in the input image data 1702 are sub-pixels, thus generating sub-pixel-by-sub-pixel output values ​​1712 in the output 1714. In yet another implementation, the elements in the input image data 1702 are super-pixels, thus generating super-pixel-by-super-pixel output values ​​1712 in the output 1714.

[0419] Export cluster metadata from decay, ternary, and / or binary maps

[0420] Figure 18One embodiment of a post-processing technique is shown, which is applied to the attenuation map 1716, the ternary map 1718, or the binary map 1720 generated by the neural network-based template generator 1512 to derive cluster metadata, including cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries. In some embodiments, the post-processing technique is applied by a post-processor 1814, which also includes a thresholder 1802, a peak locator 1806, and a segmenter 1810.

[0421] The input to the thresholder 1802 is the attenuation map 1716, ternary map 1718, or binary map 1720 generated by the template generator 1512 (such as the disclosed neural network-based template generator). In one embodiment, the thresholder 1802 applies thresholding to the values ​​in the attenuation map, ternary map, or binary map to identify background cells 1804 (i.e., sub-pixels representing non-cluster background) and non-background cells. In other words, upon generating the output 1714, the thresholder 1802 thresholds the output values ​​of the cells 1712 and classifies or reclassifies a first subset of the cells 1712 as "background cells" 1804, which represent the surrounding background of the cluster, and "non-background cells," which represent cells that are likely to belong to the cluster. The thresholds applied by the thresholder 1802 can be preset.

[0422] The input to peak locator 1806 is also the attenuation map 1716, ternary map 1718, or binary map 1720 generated by neural network-based template generator 1512. In one embodiment, peak locator 1806 applies peak detection to the values ​​in attenuation map 1716, ternary map 1718, or binary map 1720 to identify center cells 1808 (i.e., the center sub-pixel representing the center of a cluster). In other words, peak locator 1806 processes the output values ​​of cells 1712 in output 1714 and classifies a second subset of cells 1712 as "center cells" 1808 containing cluster centers. In some embodiments, the cluster centers detected by peak locator 1806 are also cluster centroids. Center cells 1808 are then provided to segmenter 1810. Additional details regarding peak locator 1806 can be found in the Appendix entitled "Peak Detection."

[0423] Thresholding and peak detection can be performed in parallel or sequentially, i.e., they are independent of each other.

[0424] The input to the segmenter 1810 is also the attenuation map 1716, ternary map 1718, or binary map 1720 generated by the neural network-based template generator 1512. Additional complementary inputs to the segmenter 1810 include the threshold cells (background, non-background cells) 1804 identified by the thresholder 1802, and the center cell 1808 identified by the peak locator 1806. The segmenter 1810 uses the background, non-background cells 1804, and center cells 1808 to identify disjoint regions 1812 (i.e., non-overlapping contiguous clusters / groups of subpixels within clusters that characterize clusters). In other words, the segmenter 1810 processes the output values ​​of cells 1712 in the output 1714 and uses the background, non-background cells 1804, and center cells 1808 to determine the shape 1812 of the cluster as a non-overlapping region of contiguous cells separated by background cells 1804 and centered around center cells 1808. The output of the segmenter 1810 is cluster metadata 1812. Cluster metadata 1812 identifies cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries.

[0425] In one embodiment, the segmenter 1810 starts with the center cell 1808 and determines for each center cell a set of consecutive adjacent cells that delineate the same cluster whose centroid is contained within the center cell. In one embodiment, the segmenter 1810 uses a so-called "watershed" segmentation technique to subdivide the adjacent clusters into multiple adjacent clusters at valley intensities. Additional details on the watershed segmentation technique and other segmentation techniques can be found in the appendix entitled "Watershed Segmentation."

[0426] In one implementation, the output value of unit 1712 in output 1714 is a continuous value, such as the value encoded in the ground truth decay map 1204. In another implementation, the output value is a softmax score, such as the score encoded in the ground truth ternary map 1304 and the ground truth binary map 1404. In the ground truth decay map 1204, according to one implementation, the neighboring cells in the corresponding non-overlapping areas of the non-overlapping regions have output values ​​weighted according to the distance of the neighboring cells from the center cell in the non-overlapping area to which the neighboring cells belong. In this implementation, the center cell has the highest output value in the corresponding non-overlapping area of ​​the non-overlapping regions. As described above, during training, due to back-propagation-based gradient updates, the decay map 1716, the ternary map 1718, and the binary map 1720 (i.e., cumulatively, the output 1714) gradually match or approach the ground truth decay map 1204, the ground truth ternary map 1304, and the ground truth binary map 1404, respectively.

[0427] Pixel Domain—Intensity Extraction of Irregular Cluster Shapes

[0428] We now discuss how the cluster shape determined by the disclosed technique can be used to extract the cluster's intensity. Since clusters typically have irregular shapes and outlines, the disclosed technique can be used to identify which sub-pixels contribute to the irregularly shaped non-intersecting / non-overlapping regions that represent the cluster shape.

[0429] Figure 19 One specific implementation of extracting cluster intensities in the pixel domain is depicted. A "template image" or "template" may refer to a data structure that contains or identifies cluster metadata 1812 derived from the attenuation map 1716, the ternary map 1718, and / or the binary map 1718. The cluster metadata 1812 identifies cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries.

[0430] In some implementations, the template image is in the upsampled sub-pixel domain to distinguish cluster boundaries at a fine-grained level. However, the sequencing image 108 containing cluster and background intensity data is typically in the pixel domain. Therefore, the disclosed technology proposes two methods to use the cluster shape information encoded in the upsampled sub-pixel resolution template image to extract the intensity of irregularly shaped clusters from the optical pixel resolution sequencing image. In the first method, as shown in FIG. Figure 19 As depicted, non-overlapping groups of adjacent sub-pixels identified in the template image are located in the pixel-resolution sequencing image, and the intensities of these non-overlapping groups of adjacent sub-pixels are extracted by interpolation. Additional details on this intensity extraction technique can be found in Figure 33 and a discussion about the diagram.

[0431] In one implementation, when the non-overlapping regions have irregular contours and the units are sub-pixels, the cluster intensity 1912 of a given cluster is determined by the intensity extractor 1902 as follows.

[0432] First, sub-pixel locator 1904 identifies sub-pixels that contribute to the cluster intensity of a given cluster based on corresponding non-overlapping areas of adjacent sub-pixels that identify the shape of the given cluster.

[0433] The sub-pixel locator 1904 then locates the identified sub-pixels in one or more optical pixel resolution images 1918 generated for one or more imaging channels in the current sequencing cycle. In one embodiment, integer or non-integer coordinates (e.g., floating point) are located in the optical pixel resolution images after being downscaled based on a downscaling factor that matches the upsampling factor used to create the sub-pixel domain.

[0434] The interpolator and sub-pixel intensity combiner 1906 then performs the following operations: interpolates the intensities of the sub-pixels identified in the processed image, combines the interpolated intensities, and normalizes the combined interpolated intensities to produce a per-image cluster intensity for a given cluster in each image in the image. Normalization is performed by a normalizer 1908 and is based on a normalization factor. In one embodiment, the normalization factor is the number of sub-pixels identified. This is done to normalize / account for different cluster sizes and the uneven illumination received by the clusters (depending on their position in the flow cell).

[0435] Finally, a cross-channel sub-pixel intensity accumulator 1910 combines the per-image cluster intensities for each image in the image to determine a cluster intensity 1912 for a given cluster in the current sequencing cycle.

[0436] The given cluster is then base called based on the cluster intensity 1912 in the current sequencing cycle by any of the base callers discussed in this application, resulting in a base call 1916.

[0437] However, in some implementations, when the cluster size is large enough, the outputs of the neural network-based base caller 1514 (i.e., the attenuation map 1716, the ternary map 1718, and the binary map 1720) are in the optical pixel domain. Therefore, in such implementations, the template image is also in the optical pixel domain.

[0438] Sub-pixel domain—intensity extraction of irregular cluster shapes

[0439] Figure 20 A second method for extracting cluster intensities in the sub-pixel domain is described. In this second method, the sequencing image at optical pixel resolution is upsampled to sub-pixel resolution. This results in a correspondence between the "cluster shapes depicted in sub-pixels" in the template image and the "cluster intensities depicted in sub-pixels" in the upsampled sequencing image. Cluster intensities are then extracted based on the correspondence. Additional details on this intensity extraction technique can be found in Figure 33 and a discussion about the diagram.

[0440] In one implementation, when the non-overlapping regions have irregular contours and the units are sub-pixels, the cluster intensity 2012 of a given cluster is determined by the intensity extractor 2002 as follows.

[0441] First, the sub-pixel locator 2004 identifies the sub-pixels that contribute to the cluster intensity of a given cluster based on corresponding non-overlapping areas of adjacent sub-pixels that identify the shape of the given cluster.

[0442] The sub-pixel locator 2004 then locates the identified sub-pixels in one or more optical pixel resolution images 2018 that are upsampled from the corresponding optical pixel resolution images 1918 generated for one or more imaging channels in the current sequencing cycle. Upsampling can be performed using the following methods: nearest neighbor intensity extraction, Gaussian-based intensity extraction, intensity extraction based on the average of a 2×2 sub-pixel area, intensity extraction based on the brightest point in a 2×2 sub-pixel area, intensity extraction based on the average of a 3×3 sub-pixel area, bilinear intensity extraction, bicubic intensity extraction, and / or weighted area coverage-based intensity extraction. These techniques are described in detail in the Appendix entitled "Intensity Extraction Methods." In some implementations, a template image can be used as a mask for intensity extraction.

[0443] Then, in each of the upsampled images, a sub-pixel intensity combiner 2006 combines the intensities of the identified sub-pixels and normalizes the combined intensities to produce a per-image cluster intensity for a given cluster in each of the upsampled images. Normalization is performed by a normalizer 2008 and is based on a normalization factor. In one embodiment, the normalization factor is the number of identified sub-pixels. This is done to normalize / account for different cluster sizes and the uneven illumination received by the clusters (depending on their position in the flow cell).

[0444] Finally, a cross-channel sub-pixel intensity accumulator 2010 combines the per-image cluster intensities for each of the upsampled images to determine a cluster intensity 2012 for a given cluster in the current sequencing cycle.

[0445] The given cluster is then base called based on the cluster intensity 2012 in the current sequencing cycle by any of the base callers discussed in this application, resulting in a base call 2016.

[0446] Types of Neural Network-Based Template Generators

[0447] The details of three different implementations of the neural network based template generator 1512 are now discussed. Figure 21a , and includes: (1) a template generator 2600 based on an attenuation map (also known as a regression model), (2) a template generator 4600 based on a binary map (also known as a binary classification model), and (3) a template generator 5400 based on a ternary map (also known as a ternary classification model).

[0448] In one embodiment, the regression model 2600 is a fully convolutional network. In another embodiment, the regression model 2600 is a U-Net network with skip connections between the decoder and the encoder. In one embodiment, the binary classification model 4600 is a fully convolutional network. In another embodiment, the binary classification model 4600 is a U-Net network with skip connections between the decoder and the encoder. In one embodiment, the ternary classification model 5400 is a fully convolutional network. In another embodiment, the ternary classification model 5400 is a U-Net network with skip connections between the decoder and the encoder.

[0449] Input image data

[0450] Figure 21b One specific implementation of input image data 1702 is depicted as being fed as input to the neural network-based template generator 1512. The input image data 1702 includes a series of image sets 2100 having sequencing images 108 that are generated during a certain number of initial sequencing cycles (e.g., the first 2 to 7 sequencing cycles) of a sequencing run.

[0451] In some implementations, the intensities of the sequencing images 108 are corrected for background and / or the images are aligned with each other using an affine transform. In one implementation, the sequencing run utilizes a four-channel chemistry, and each image set has four images. In another implementation, the sequencing run utilizes a two-channel chemistry, and each image set has two images. In yet another implementation, the sequencing run utilizes a single-channel chemistry, and each image set has two images. In other implementations, each image set has only one image. These and other different implementations are described in Appendices 6 and 9.

[0452] Each image 2116 in the series of image sets 2100 covers a patch 2104 of the flow cell 2102 and depicts the intensity emission of a cluster 2106 and its surrounding background on the patch 2104, captured for a particular imaging channel during a particular sequencing cycle among a plurality of sequencing cycles of a sequencing run. In one example, for cycle t1, the image set includes four images 2112A, 2112C, 2112T, and 2112G: one image for each base A, C, T, and G, which are labeled with corresponding fluorescent dyes and imaged in corresponding wavelength bands (images / imaging channels).

[0453] For illustrative purposes, in image 2112G, Figure 21bCluster intensity emission is depicted as 2108 and background intensity emission is depicted as 2110. In another example, for cycle tn, the image set includes four images 2114A, 2114C, 2114T, and 2114G: one image for each base A, C, T, and G, which are labeled with corresponding fluorescent dyes and imaged in corresponding wavelength bands (images / imaging channels). Also for illustrative purposes, in image 2114A, Figure 21b The cluster intensity emission is depicted as 2118 , and in image 2114T, the background intensity emission is depicted as 2120 .

[0454] Non-image data

[0455] Input image data 1702 is encoded using intensity channels (also referred to as imaging channels). For each image in the c-image obtained from the sequencer for a particular sequencing cycle, its intensity signal data is encoded using a separate imaging channel. For example, considering that the sequencing run uses a dual-channel chemistry that produces a red image and a green image in each sequencing cycle. In this case, the input data 2632 includes (i) a first red imaging channel with w×h pixels that depicts the intensity emission of the one or more clusters and their surrounding background captured in the red image; and (ii) a second green imaging channel with w×h pixels that depicts the intensity emission of the one or more clusters and their surrounding background captured in the green image.

[0456] In another embodiment, image data is not used as input to the neural network-based template generator 1512 or the neural network-based base caller 1514. Instead, the input to the neural network-based template generator 1512 and the neural network-based base caller 1514 is based on pH changes caused by the release of hydrogen ions during molecular extension. The pH change is detected and converted into a voltage change that is proportional to the number of bases incorporated (e.g., in the case of Ion Torrent).

[0457] In another specific embodiment, the inputs of the neural network-based template generator 1512 and the neural network-based base detector 1514 are constructed based on nanopore sensing, which uses a biosensor to measure the interruption of electric current when an analyte passes through a nanopore or near its pore opening, and simultaneously determines the type of base. For example, Oxford Nanopore Technology (ONT) sequencing is based on the following concept: single-stranded DNA (or RNA) is passed through a membrane via a nanopore and a voltage difference is applied across the membrane. The nucleotides present in the pore will affect the resistance of the pore, so the current measurement results over time can indicate the sequence of DNA bases passing through the pore. This current signal (called a "squiggle" due to its appearance when plotted) is the raw data collected by the ONT sequencer. These measurements are stored as 16-bit integer data acquisition (DAC) values ​​acquired at a frequency of (for example) 4kHz. With a DNA chain speed of approximately 450 base pairs / second, this gives an average of approximately nine raw observations per base. This signal is then processed to identify interruptions in the pore opening signal corresponding to each reading. The greatest use of these raw signals is to perform base calling, the process of converting DAC values ​​into DNA base sequences.In some implementations, input data 2632 includes normalized or scaled DAC values.

[0458] Patch extraction

[0459] Figure 22 Shown from Figure 21b In one embodiment, patches are extracted from a series of image sets 2100 in the sequence of images to produce a series of "reduced-size" image sets that form the input image data 1702. In the illustrated embodiment, the sequencing images 108 in the series of image sets 2100 have a size of L×L (e.g., 2000×2000). In other embodiments, L is any number in the range of 1 to 10,000.

[0460] In one embodiment, the patch extractor 2202 extracts patches from the sequencing images 108 in the series of image sets 2100 and generates a series of reduced-size image sets 2206, 2208, 2210, and 2212. Each image in the series of reduced-size image sets is a patch of size M×M (e.g., 20×20) extracted from a corresponding sequencing image in the series of image sets 2100. The size of the patch can be preset. In other embodiments, M is any number in the range of 1 to 1000.

[0461] exist Figure 22, four exemplary series of reduced-size image sets are shown. A first exemplary series of reduced-size image sets 2206 is extracted from the sequence of sequencing images 108 (coordinates 0, 0 to 20, 20) in the sequence of image set 2100. A second exemplary series of reduced-size image sets 2208 is extracted from the sequence of sequencing images 108 (coordinates 20, 20 to 40, 40) in the sequence of image set 2100. A third exemplary series of reduced-size image sets 2210 is extracted from the sequence of sequencing images 108 (coordinates 40, 40 to 60, 60) in the sequence of image set 2100. A fourth exemplary series of reduced-size image sets 2212 is extracted from the sequence of sequencing images 108 (coordinates 60, 60 to 80, 80) in the sequence of image set 2100.

[0462] In some implementations, the series of reduced-size image sets forms input image data 1702, which is fed as input to the neural network-based template generator 1512. Multiple series of reduced-size image sets can be fed simultaneously as an input batch, and a separate output can be generated for each series in the input batch.

[0463] Upsampling

[0464] Figure 23 Describes the Figure 21b In one embodiment, a series of image sets 2100 in FIG. 1 are upsampled to produce a series of “upsampled” image sets 2300 that form the input image data 1702 .

[0465] In one specific implementation, the upsampler 2302 uses an interpolation method (eg, bicubic interpolation) to upsample the sequencing images 108 in the series of image sets 2100 by an upsampling factor (eg, 4 times) to generate a series of upsampled image sets 2300 .

[0466] In the illustrated implementation, the sequencing images 108 in the series of image sets 2100 are of size L×L (e.g., 2000×2000) and are upsampled by an upsampling factor of 4 to produce upsampled images of size U×U (e.g., 8000×8000) in the series of upsampled image sets 2300.

[0467] In one specific implementation, the sequencing images 108 in the series of image sets 2100 are fed directly to the neural network-based template generator 1512, and upsampling is performed by the initial layer of the neural network-based template generator 1512. That is, the upsampler 2302 is part of the neural network-based template generator 1512 and operates as its first layer, which upsamples the sequencing images 108 in the series of image sets 2100 and generates a series of upsampled image sets 2300.

[0468] In some implementations, a series of upsampled image sets 2300 form the input image data 1702, which is fed as input to the neural network-based template generator 1512.

[0469] Figure 24 Shown from Figure 23 One specific implementation of extracting patches from the series of upsampled image sets 2300 to produce a series of “upsampled and downsized” image sets 2406 , 2408 , 2410 , and 2412 that form the input image data 1702 .

[0470] In one embodiment, the patch extractor 2202 extracts patches from the upsampled images in the series of upsampled image sets 2300 and generates a series of upsampled and downsized image sets 2406, 2408, 2410, and 2412. Each upsampled image in the series of upsampled and downsized image sets is a patch of size M×M (e.g., 80×80) extracted from a corresponding upsampled image in the series of upsampled image sets 2300. The size of the patch can be preset. In other embodiments, M is any number in the range of 1 to 1000.

[0471] exist Figure 24 , four exemplary series of upsampled and downsized image sets are shown. A first exemplary series of upsampled and downsized image sets 2406 is extracted from the upsampled images (coordinates 0,0 to 80,80) in the series of upsampled image sets 2300. A second exemplary series of upsampled and downsized image sets 2408 is extracted from the upsampled images (coordinates 80,80 to 160,160) in the series of upsampled image sets 2300. A third exemplary series of upsampled and downsized image sets 2410 is extracted from the upsampled images (coordinates 160,160 to 240,240) in the series of upsampled image sets 2300. A fourth exemplary series of upsampled and downsized image sets 2412 is extracted from the upsampled images (coordinates 240,240 to 320,320) in the series of upsampled image sets 2300.

[0472] In some implementations, the series of upsampled and downsized image sets forms input image data 1702, which is fed as input to the neural network-based template generator 1512. Multiple series of upsampled and downsized image sets can be fed simultaneously as an input batch, and a separate output can be generated for each series in the input batch.

[0473] Output

[0474] These three models are trained to produce different outputs. This is achieved by using different types of ground truth data representations as training labels. Regression model 2600 is trained to produce outputs that represent / indicate / indicate a so-called "attenuation map" 1716. Binary classification model 4600 is trained to produce outputs that represent / indicate / indicate a so-called "binary map" 1720. Ternary classification model 5400 is trained to produce outputs that represent / indicate / indicate a so-called "ternary map" 1718.

[0475] The output 1714 of each type of model includes an array of cells 1712. Cells 1712 can be pixels, sub-pixels, or super-pixels. The output of each type of model includes cell-by-cell output values, such that the output values ​​of the cell arrays together represent / represent / indicate an attenuation map 1716 in the case of the regression model 2600, a binary map 1720 in the case of the binary classification model 4600, and a ternary map 1718 in the case of the ternary classification model 5400. More details are provided below.

[0476] Benchmark truth data generation

[0477] Figure 25 A specific implementation of the overall exemplary process for generating ground truth data for training the neural network-based template generator 1512 is shown. For the regression model 2600, the ground truth data may be the decay map 1204. For the binary classification model 4600, the ground truth data may be the binary map 1404. For the ternary classification model 5400, the ground truth data may be the ternary map 1304. The ground truth data is generated from cluster metadata. The cluster metadata is generated by the cluster metadata generator 122. The ground truth data is generated by the ground truth data generator 1506.

[0478] In the illustrated embodiment, ground truth data is generated for block A on channel A of flow cell A. Ground truth data is generated from sequencing images 108 of block A captured during sequencing run A. Sequencing images 108 of block A are in the pixel domain. In an example involving four-channel chemistry generating four sequencing images per sequencing cycle, 200 sequencing images 108 are obtained for 50 sequencing cycles. Each of the 200 sequencing images 108 depicts the intensity emission of a cluster on block A and its surrounding background, captured in a specific imaging channel during a specific sequencing cycle.

[0479] The sub-pixel addresser 110 converts the sequenced image 108 into the sub-pixel domain (eg, by dividing each pixel into a plurality of sub-pixels) and generates a sequenced image 112 in the sub-pixel domain.

[0480] Then, a base caller 114 (e.g., RTA) processes the sequencing image 112 in the sub-pixel domain and generates a base call for each sub-pixel and each of the fifty sequencing cycles. This is referred to herein as "sub-pixel base calling."

[0481] The sub-pixel base calls are then merged 116 to generate a base call sequence for each sub-pixel across fifty sequencing cycles. The base call sequence for each sub-pixel has fifty base calls, i.e., one base call for each of the fifty sequencing cycles.

[0482] Searcher 118 evaluates the base call sequences of adjacent sub-pixels on a pair-by-pair basis. The search involves evaluating each sub-pixel to determine which of its adjacent sub-pixels it shares a substantially matching base call sequence with. The base call sequences of adjacent sub-pixels are "substantially matched" when a predetermined portion of the base calls match in positional order (e.g., ≥41 matches in 45 cycles, ≤4 mismatches in 45 cycles, ≤4 mismatches in 50 cycles, or ≤2 mismatches in 34 cycles).

[0483] In some implementations, base caller 114 also identifies the initial center coordinates of the cluster. The sub-pixel containing the initial center coordinates is referred to as the center or origin sub-pixel. Figure 6 , some exemplary initial center coordinates (604a-c) and corresponding origin sub-pixels (606a-c) identified by base caller 114 are shown in FIG. However, identification of the origin sub-pixel (the initial center coordinate of the cluster) is not required, as described below. In some implementations, searcher 118 performs a breadth-first search to identify a substantially matching base call sequence of sub-pixels, starting with origin sub-pixels 606a-c and continuing with consecutively adjacent non-origin sub-pixels 702a-c. This is also optional, as described below.

[0484] The search for a substantially matching base call sequence for a sub-pixel does not require identification of the origin sub-pixel (the initial center coordinate of the cluster), as the search can be performed on all sub-pixels, and the search does not need to begin at the origin sub-pixel, but can begin at any sub-pixel (e.g., the 0,0 sub-pixel or any random sub-pixel). Therefore, since each sub-pixel is evaluated to determine whether it shares a substantially matching base call sequence with another adjacent sub-pixel, the search does not need to utilize the origin sub-pixel and can begin at any sub-pixel.

[0485] Regardless of whether the origin sub-pixel is used, some clusters are identified that do not contain the origin sub-pixel (the initial center coordinate of the cluster) predicted by the base caller 114. Some examples of clusters identified by merging sub-pixel base calling and that do not contain the origin sub-pixel are Figure 8aThus, using base caller 114 to identify the origin sub-pixel (the initial center coordinate of the cluster) is optional and not necessary to search for a substantially matching base call sequence of sub-pixels.

[0486] Searcher 118: (1) identifies adjacent sub-pixels with substantially matching base call sequences as so-called "disjoint regions," (2) further evaluates the base call sequences of those sub-pixels that do not belong to any of the disjoint regions identified at (1) to generate additional disjoint regions, and (3) then identifies background sub-pixels as those sub-pixels that do not belong to either of the disjoint regions identified at (1) and (2). Action (2) allows the disclosed technique to identify additional or additional clusters whose centers have not yet been identified by base caller 114.

[0487] The results of the searcher 118 are encoded in a so-called "cluster map" for tile A and stored in a cluster map data storage 120. In the cluster map, each of the clusters on tile A is identified by a corresponding non-intersecting region of adjacent sub-pixels, with background sub-pixels separating the non-intersecting regions to identify the surrounding background on the tile.

[0488] A centroid (COM) calculator 1004 determines the center of each of the clusters on block A by calculating the COM of each of the disjoint regions as the average of the coordinates of the corresponding adjacent sub-pixels forming the disjoint regions. The centroid of the cluster is stored as COM data 2502.

[0489] The subpixel classifier 2504 uses the cluster map and the COM data 2502 to generate a subpixel classification 2506. The subpixel classification 2506 classifies subpixels in the cluster map into: (1) background subpixels, (2) COM subpixels (each non-intersecting region of the COM containing the corresponding non-intersecting region has one COM subpixel), and (3) cluster / cluster-internal subpixels forming the corresponding non-intersecting region. That is, each subpixel in the cluster map is assigned to one of the three categories.

[0490] Based on the sub-pixel classification 2506, in some embodiments, (i) the reference truth attenuation map 1204 is generated by the reference truth attenuation map generator 1202, (ii) the reference truth binary map 1304 is generated by the reference truth binary map generator 1302, and (iii) the reference truth ternary map 1404 is generated by the reference truth ternary map generator 1402.

[0491] 1. Regression Model

[0492] Figure 26One implementation of a regression model 2600 is shown. In the illustrated implementation, the regression model 2600 is a fully convolutional network 2602 that processes input image data 1702 via an encoder subnetwork and a corresponding decoder subnetwork. The encoder subnetwork includes an encoder hierarchy. The decoder subnetwork includes a decoder hierarchy that maps a low-resolution encoder feature map to a full-input resolution attenuation map 1716. In another implementation, the regression model 2600 is a U-Net network 2604 with skip connections between the decoder and encoder. Additional details about the segmentation network can be found in the Appendix titled "Segmentation Network."

[0493] Attenuation Map

[0494] Figure 27 A specific implementation of generating a ground truth attenuation map 1204 from a cluster map 2702 is depicted. The ground truth attenuation map 1204 is used as ground truth data for training the regression model 2600. In the ground truth attenuation map 1204, the ground truth attenuation map generator 1202 assigns a weighted attenuation value to each adjacent sub-pixel in the disjoint region based on a weighted attenuation factor. The weighted attenuation value is proportional to the Euclidean distance of the adjacent sub-pixel from the centroid (COM) sub-pixel in the disjoint region to which the adjacent sub-pixel belongs, such that the weighted attenuation value is highest (e.g., 1 or 100) for the COM sub-pixel and lower for sub-pixels farther from the COM sub-pixel. In some specific implementations, the weighted attenuation value is multiplied by a preset factor, such as 100.

[0495] Furthermore, the ground truth attenuation map generator 1202 assigns the same predetermined value (eg, a floor background value) to all background sub-pixels.

[0496] The ground truth attenuation map 1204 represents adjacent sub-pixels and background sub-pixels in the disjoint region based on assigned values. The ground truth attenuation map 1204 also stores the assigned values ​​in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0497] train

[0498] Figure 28 It is a specific implementation of training 2800 the regression model 2600 using a back-propagation based gradient update technique, which modifies the parameters of the regression model 2600 until the training output of the regression model 2600 during training 2800 gradually approaches or matches the reference true value attenuation map 1204 to produce an attenuation map 1716.

[0499] Training 2800 includes iteratively optimizing a loss function that minimizes an error 2806 between the attenuation map 1716 and the ground truth attenuation map 1204, and updating parameters of the regression model 2600 based on the error 2806. In one embodiment, the loss function is a mean squared error, and the error is minimized on a sub-pixel-by-sub-pixel basis between the weighted attenuation values ​​of corresponding sub-pixels in the attenuation map 1716 and the ground truth attenuation map 1204.

[0500] Training 2800 includes hundreds, thousands, and / or millions of iterations of forward propagation 2808 and backward propagation 2810, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 is performed by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0501] infer

[0502] Figure 29 A specific implementation of generating a template by the regression model 2600 during inference 2900, wherein the regression model 2600 generates an attenuation map 1716 as an inference output during inference 2900. An example of the attenuation map 1716 is disclosed in the appendix entitled "Regression_Model_Sample_Ouput." The appendix includes unit-by-unit weighted attenuation output values ​​2910 that together represent the attenuation map 1716.

[0503] Inference 2900 includes hundreds, thousands, and / or millions of iterations of forward propagation 2904, including parallelization techniques such as batch processing. Inference 2900 is performed on inference data 2908, which includes a series of upsampled and downsized image sets as input image data 1702. Inference 2900 is operated by a tester 2906.

[0504] Watershed segmentation

[0505] Figure 30 One embodiment of attenuation map 1716 is shown as being subjected to (i) thresholding to identify background sub-pixels representing the background of a cluster and (ii) peak detection to identify center sub-pixels representing the center of a cluster. Thresholding is performed by thresholder 1802, which uses a local threshold binary value to produce a binarized output. Peak detection is performed by peak locator 1806 to identify the center of the cluster. Additional details regarding the peak locator can be found in the Appendix entitled "Peak Detection."

[0506] Figure 31A specific implementation of a watershed segmentation technique is depicted, which takes as input the background sub-pixels identified by thresholder 1802 and the center sub-pixels identified by peak locator 1806, finds the valley intensities between adjacent clusters, and outputs a set of non-overlapping adjacent clusters / cluster interior sub-pixels representing the clusters. Additional details on the watershed segmentation technique can be found in the appendix entitled "Watershed Segmentation."

[0507] In one embodiment, the watershed segmenter 3102 takes as input the following data: (1) the negative output values ​​2910 in the attenuation map 1716, (2) the binarized output of the thresholder 1802, and (3) the cluster centers identified by the peak locator 1806. The watershed segmenter 3102 then generates an output 3104 based on the inputs. In the output 3104, each cluster center is identified as a set of unique sub-pixels that belong to the cluster center (as long as the sub-pixel is "1" in the binarized output, i.e., it is not a background sub-pixel, it belongs to the cluster center). In addition, the clusters are filtered based on containing at least four sub-pixels. The watershed segmenter 3102 can be part of the segmenter 1810, which in turn can be part of the post-processor 1814.

[0508] Network Architecture

[0509] Figure 32 is a table showing an exemplary U-network architecture for the regression model 2600, along with details of the layers of the regression model 2600, the dimensions of the layer outputs, the magnitudes of the model parameters, and the interconnections between the layers. Similar details are disclosed in a file titled “Regression_Model_Example_Architecture,” which is filed as an appendix to this application.

[0510] Cluster intensity extraction

[0511] Figure 33 Different methods of extracting cluster intensities using cluster shape information identified in a template image are shown. As described above, the template image identifies cluster shape information at upsampled sub-pixel resolution. However, the cluster intensity information is in the sequencing images 108, which typically have optical pixel resolution.

[0512] According to a first approach, the coordinates of the sub-pixels are located in the sequencing image 108 and their respective intensities are extracted using bilinear interpolation and normalized based on the count of sub-pixels contributing to the cluster.

[0513] The second method uses a weighted area cover technique to modulate the intensity of a pixel according to the number of sub-pixels contributing to the pixel. Here again, the modulated pixel intensity is normalized by the sub-pixel count parameter.

[0514] The third method upsamples the sequenced image into the sub-pixel domain using bicubic interpolation, sums the intensities of the upsampled pixels belonging to the cluster, and normalizes the summed intensities based on the counts of the upsampled pixels belonging to the cluster.

[0515] Experimental results and observations

[0516] Figure 34 Different methods of base calling using the output of the regression model 2600 are shown. In the first method, cluster centers identified from the output of the neural network-based template generator 1512 in the template image are fed to a base caller (e.g., Illumina's Real-Time Analysis software, referred to herein as "RTA Base Caller") for base calling.

[0517] In the second method, instead of cluster centers, cluster intensities extracted from sequencing images based on cluster shape information in the template image are fed to the RTA base caller for base calling.

[0518] Figure 35 The difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center is shown. The results show that using the COM improves base calling.

[0519] Example model output

[0520] Figure 36 An exemplary attenuation map 1716 produced by the regression model 2600 is shown on the left. Figure 36 Also shown on the right is an exemplary ground truth decay map 1204 that the regression model 2600 approximates during training.

[0521] Both the attenuation map 1716 and the ground truth attenuation map 1204 depict clusters as non-intersecting regions of adjacent sub-pixels, the center of the cluster as the center sub-pixel at the centroid of the corresponding non-intersecting regions in the non-intersecting regions, and its surrounding background as background sub-pixels that do not belong to any of the non-intersecting regions.

[0522] Additionally, adjacent sub-pixels in corresponding non-intersecting regions of the non-intersecting regions have values ​​weighted according to the distance of the adjacent sub-pixels from the center sub-pixel in the non-intersecting regions to which the adjacent sub-pixels belong. In one embodiment, the center sub-pixel has the highest value in the corresponding non-intersecting regions of the non-intersecting regions. In one embodiment, all background sub-pixels have the same minimum background value within the attenuation map.

[0523] Figure 37One specific implementation of peak locator 1806 is depicted for identifying cluster centers in the attenuation map by detecting peaks 3702. Additional details regarding the peak locator can be found in the appendix entitled "Peak Detection."

[0524] Figure 38 The peaks detected by peak locator 1806 in the attenuation map 1716 produced by regression model 2600 are compared to the corresponding peaks in ground truth attenuation map 1204. The red markers are peaks predicted by regression model 2600 to be cluster centers, and the green markers are the ground truth centroids of the clusters.

[0525] More experimental results and observations

[0526] Figure 39 The performance of the regression model 2600 using precision and recall statistics is shown. The precision and recall statistics indicate that the regression model 2600 is suitable for recovering all identified cluster centers.

[0527] Figure 40 The performance of regression model 2600 was compared to that of the RTA base caller for a library concentration of 20 pM (normal operation). Regression model 2600 outperformed the RTA base caller, identifying 34,323 more (4.46%) clusters (i.e., 988,884 clusters) in a higher cluster density environment.

[0528] Figure 40 Results for other sequencing metrics are also shown, such as the number of clusters passing the purity filter ("% PF" (passed filter)), the number of aligned reads ("% aligned"), the number of duplicate reads ("% duplicates"), the number of reads mismatched with the reference sequence among all reads aligned to the reference sequence ("% mismatches"), the number of bases detected with a quality score of 30 and above ("% Q30 bases"), and so on.

[0529] Figure 41 The performance of regression model 2600 was compared to that of the RTA base caller for a 30 pM library concentration (dense run). Regression model 2600 outperformed the RTA base caller, identifying 34,323 more (6.27%) clusters (i.e., 1,351,588 clusters) in a much higher cluster density environment.

[0530] Figure 41 Results for other sequencing metrics are also shown, such as the number of clusters passing the purity filter ("% PF" (passed filter)), the number of aligned reads ("% aligned"), the number of duplicate reads ("% duplicates"), the number of reads mismatched with the reference sequence among all reads aligned to the reference sequence ("% mismatches"), the number of bases detected with a quality score of 30 and above ("% Q30 bases"), and so on.

[0531] Figure 42 The number of non-duplicate (unique or deduplicated) correct read pairs detected by the regression model 2600 (i.e., the number of paired reads where both reads aligned within a reasonable distance) was compared to the number of non-duplicate correct read pairs detected by the RTA base caller. The comparison was performed for both the 20 pM normal run and the 30 pM dense run.

[0532] More importantly, Figure 42 The results show that the disclosed neural network-based template generator is able to detect more clusters in fewer sequencing cycles with template generation input than the RTA base caller. In only four sequencing cycles, the regression model 2600 identified 11% more non-duplicate correct read pairs than the RTA base caller during a 20 pM normal run and 33% more non-duplicate correct read pairs during a 30 pM intensive run. In only seven sequencing cycles, the regression model 2600 identified 4.5% more non-duplicate correct read pairs than the RTA base caller during a 20 pM normal run and 6.3% more non-duplicate correct read pairs during a 30 pM intensive run.

[0533] Figure 43 On the right is shown a first attenuation map generated by the regression model 2600. The first attenuation map identifies clusters imaged during normal operation at 20 pM and their surrounding background, and its spatial distribution depicting cluster shape, cluster size, and cluster centers.

[0534] Figure 43 Shown on the left is a second attenuation map generated by the regression model 2600. The second attenuation map identifies clusters imaged during the 30 pM intensive run and their surrounding background, and its spatial distribution depicting cluster shape, cluster size, and cluster centers.

[0535] Figure 44 The performance of regression model 2600 was compared to that of the RTA base caller for a 40 pM library concentration (a highly dense run). Regression model 2600 generated 89,441,688 more aligned bases than the RTA base caller in a much higher cluster density environment (i.e., 1,509,395 clusters).

[0536] Figure 44 Results for other sequencing metrics are also shown, such as the number of clusters passing the purity filter ("% PF" (passed filter)), the number of aligned reads ("% aligned"), the number of duplicate reads ("% duplicates"), the number of reads mismatched with the reference sequence among all reads aligned to the reference sequence ("% mismatches"), the number of bases detected with a quality score of 30 and above ("% Q30 bases"), and so on.

[0537] More example model outputs

[0538] Figure 45 Shown on the left is a first attenuation map generated by the regression model 2600. The first attenuation map identifies clusters imaged during normal operation at 40 pM and their surrounding background, and it depicts the spatial distribution of cluster shape, cluster size, and cluster centers.

[0539] Figure 45 On the right side, the results of the following operations are shown: thresholding and peak location are applied to the first attenuation map to distinguish the corresponding clusters from each other and from the background, and to identify their corresponding cluster centers. In some implementations, the intensity of the corresponding clusters is identified, and a purity filter (or filtering) is applied to reduce the mismatch rate.

[0540] 2. Binary Classification Model

[0541] Figure 46 One implementation of a binary classification model 4600 is shown. In the illustrated implementation, the binary classification model 4600 is a deep fully convolutional segmentation neural network that processes input image data 1702 via an encoder subnetwork and a corresponding decoder subnetwork. The encoder subnetwork includes an encoder hierarchy. The decoder subnetwork includes a decoder hierarchy that maps a low-resolution encoder feature map to a full-input resolution binary map 1720. In another implementation, the binary classification model 4600 is a U-Net network with skip connections between the decoder and encoder. Additional details about the segmentation network can be found in the appendix titled "Segmentation Network."

[0542] Binary Map

[0543] The final output layer of the binary classification model 4600 is a cell-by-cell classification layer that produces a classification label for each cell in the output array. In some implementations, the cell-by-cell classification layer is a sub-pixel classification layer that produces a softmax classification score distribution across two classes (i.e., cluster center class and non-cluster class) for each sub-pixel in the binary map 1720, and the classification label of a given sub-pixel is determined by the corresponding softmax classification score distribution.

[0544] In other embodiments, the unit-by-unit classification layer is a sub-pixel-by-sub-pixel classification layer that produces a sigmoid classification score for each sub-pixel in the binary map 1720, such that the activation of a unit is interpreted as the probability that the unit belongs to the first class, and conversely, 1 minus the activation gives the probability that it belongs to the second class.

[0545] The binary map 1720 represents each sub-pixel based on the predicted classification score. The binary map 1720 also stores the predicted classification scores in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0546] train

[0547] Figure 47 It is a specific implementation of training 4700 a binary classification model 4600 using a back-propagation based gradient update technique, which modifies the parameters of the binary classification model 4600 until the binary map 1720 of the binary classification model 4600 gradually approaches or matches the reference true binary map 1404.

[0548] In the illustrated implementation, the final output layer of the binary classification model 4600 is a softmax-based sub-pixel classification layer. In the softmax implementation, the ground truth binary map generator 1402 assigns each ground truth sub-pixel to either (i) a cluster center value pair (e.g., [1, 0]) or (ii) a non-center value pair (e.g., [0, 1]).

[0549] In the cluster center value pair [1,0], the first value [1] represents the cluster center class label, and the second value [0] represents the non-center class label. In the non-center value pair [0,1], the first value [0] represents the cluster center class label, and the second value [1] represents the non-center class label.

[0550] The ground truth binary map 1404 represents each sub-pixel based on an assigned value pair / value. The ground truth binary map 1404 also stores the assigned value pairs / values ​​in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0551] Training includes iteratively optimizing a loss function that minimizes an error 4706 (eg, a softmax error) between the binary map 1720 and the ground truth binary map 1404 , and updating parameters of the binary classification model 4600 based on the error 4706 .

[0552] In one embodiment, the loss function is a custom weighted binary cross entropy loss, and the error 4706 is minimized on a sub-pixel-by-sub-pixel basis between the predicted classification scores (e.g., softmax scores) and the labeled class scores (e.g., softmax scores) of the corresponding sub-pixels in the binary map 1720 and the ground truth binary map 1404, as Figure 47 shown.

[0553] The custom weighted loss function assigns more weight to the COM sub-pixels, so that whenever a COM sub-pixel is misclassified, the cross-entropy loss is multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Additional details about the custom weighted loss function can be found in the appendix titled "Custom Weighted Loss Function".

[0554] Training 4700 includes hundreds, thousands, and / or millions of iterations of forward propagation 4708 and backward propagation 4710, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 is performed by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0555] Figure 48 is another specific implementation of training 4800 the binary classification model 4600, wherein the final output layer of the binary classification model 4600 is a sigmoid-based sub-pixel classification layer.

[0556] In a sigmoid implementation, the ground truth binary map generator 1302 assigns each ground truth sub-pixel either (i) a cluster center value (e.g., [1]) or (ii) a non-center value (e.g., [0]). The COM sub-pixel is assigned a cluster center value pair / value, and all other sub-pixels are assigned a non-center value pair / value.

[0557] For cluster center values, values ​​above a threshold midway between 0 and 1 (e.g., values ​​above 0.5) represent the center class label. For non-center values, values ​​below a threshold midway between 0 and 1 (e.g., values ​​below 0.5) represent the non-center class label.

[0558] The ground truth binary map 1404 represents each sub-pixel based on an assigned value pair / value. The ground truth binary map 1404 also stores the assigned value pairs / values ​​in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0559] Training includes iteratively optimizing a loss function that minimizes an error 4806 (eg, a sigmoid error) between the binary map 1720 and the ground truth binary map 1404 , and updating parameters of the binary classification model 4600 based on the error 4806 .

[0560] In one embodiment, the loss function is a custom weighted binary cross entropy loss, and the error 4806 is minimized on a sub-pixel-by-sub-pixel basis between the predicted scores (e.g., sigmoid scores) and the labeled scores (e.g., sigmoid scores) for the corresponding sub-pixels in the binary map 1720 and the ground truth binary map 1404, as Figure 48 shown.

[0561] The custom weighted loss function assigns more weight to the COM sub-pixels, so that whenever a COM sub-pixel is misclassified, the cross-entropy loss is multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Additional details about the custom weighted loss function can be found in the appendix titled "Custom Weighted Loss Function".

[0562] Training 4800 includes hundreds, thousands, and / or millions of iterations of forward propagation 4808 and backward propagation 4810, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 is performed by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0563] Figure 49 Another specific implementation of input image data 1702 fed to a binary classification model 4600 and corresponding class labels 4904 used to train the binary classification model 4600 is shown.

[0564] In the illustrated embodiment, the input image data 1702 includes a series of upsampled and downsized image sets 4902. The class labels 4904 include two classes: (1) "no cluster center" and (2) "cluster center", which are distinguished by different output values. That is, (1) light green cells / sub-pixels 4906 represent sub-pixels predicted by the binary classification model 4600 as not containing a cluster center, and (2) dark green sub-pixels 4908 represent cells / sub-pixels predicted by the binary classification model 4600 as containing a cluster center.

[0565] infer

[0566] Figure 505000, wherein the binary map 1720 is generated by the binary classification model 4600 as an inference output during the inference 5000. An example of the binary map 1720 includes unit-by-unit binary classification scores 5010 that together represent the binary map 1720. In a softmax application, the binary map 1720 has a first array 5002a of unit-by-unit classification scores for the non-center class and a second array 5002b of unit-by-unit classification scores for the cluster center class.

[0567] Inference 5000 includes hundreds, thousands, and / or millions of iterations of forward propagation 5004, including parallelization techniques such as batch processing. Inference 5000 is performed on inference data 2908, which includes a series of upsampled and downsized image sets as input image data 1702. Inference 5000 is operated by tester 2906.

[0568] In some implementations, the binary map 1720 is subjected to the post-processing techniques discussed above, such as thresholding, peak detection, and / or watershed segmentation, to generate cluster metadata.

[0569] Peak detection

[0570] Figure 51 A specific implementation of peak detection for identifying cluster centers on a binary map 1720 is depicted. As described above, the binary map 1720 is an array of cells that classify each sub-pixel based on a predicted classification score, where each cell in the array represents a corresponding sub-pixel in the input. The classification score can be a softmax score or a sigmoid score.

[0571] In the softmax application, the binary map 1720 includes two arrays: (1) a first array 5002a of cell-by-cell classification scores for the non-center class and (2) a second array 5002b of cell-by-cell classification scores for the cluster center class. In both arrays, each cell represents a corresponding sub-pixel in the input.

[0572] To determine which sub-pixels in the input contain cluster centers and which do not, peak locator 1806 applies peak detection to the cells in binary map 1720. Peak detection identifies those cells whose classification scores (e.g., softmax / sigmoid scores) are above a preset threshold. The identified cells are inferred to be cluster centers, and the corresponding sub-pixels of these cells in the input are determined to contain cluster centers and stored as cluster center sub-pixels in sub-pixel classification data storage 5102. Additional details regarding peak locator 1806 can be found in the Appendix entitled "Peak Detection."

[0573] The remaining cells and their corresponding sub-pixels in the input are determined to not contain a cluster center and are stored as non-center sub-pixels in the sub-pixel classification data storage device 5102.

[0574] In some implementations, before applying peak detection, those cells whose classification scores are below a certain background threshold (e.g., 0.3) are set to zero. In some implementations, such cells and their corresponding sub-pixels in the input are inferred to represent the background surrounding the cluster and are stored as background sub-pixels in the sub-pixel classification data storage 5102. In other implementations, such cells can be considered noise and ignored.

[0575] Example model output

[0576] Figure 52a An exemplary binary map produced by the binary classification model 4600 is shown on the left. Figure 52a Also shown on the right is an exemplary ground truth binary map approximated by the binary classification model 4600 during training. The binary map has multiple sub-pixels, and each sub-pixel is classified as a cluster center or non-center. Similarly, the ground truth binary map has multiple sub-pixels, and each sub-pixel is classified as a cluster center or non-center.

[0577] Experimental results and observations

[0578] Figure 52b The performance of the binary classification model 4600 using recall and precision statistics is shown. Applying these statistics, the binary classification model 4600 outperforms the RTA base caller.

[0579] Network Architecture

[0580] Figure 53 is a table showing an example architecture of a binary classification model 4600, along with details of the layers, dimensions of the layers' outputs, magnitudes of the model parameters, and interconnections between the layers. Similar details are disclosed in the appendix titled "Binary_Classification_Model_Example_Architecture."

[0581] 3. Ternary (three-category) classification model

[0582] Figure 54One implementation of a ternary classification model 5400 is shown. In the illustrated implementation, the ternary classification model 5400 is a deep fully convolutional segmentation neural network that processes input image data 1702 via an encoder subnetwork and a corresponding decoder subnetwork. The encoder subnetwork includes an encoder hierarchy. The decoder subnetwork includes a decoder hierarchy that maps the low-resolution encoder feature map to the full-input resolution ternary map 1718. In another implementation, the ternary classification model 5400 is a U-Net network with skip connections between the decoder and encoder. Additional details about the segmentation network can be found in the appendix entitled "Segmentation Network."

[0583] Ternary Map

[0584] The final output layer of the ternary classification model 5400 is a unit-by-unit classification layer that produces a classification label for each unit in the output array. In some specific implementations, the unit-by-unit classification layer is a sub-pixel classification layer that produces a softmax classification score distribution across three classes (i.e., background class, cluster center class, and cluster / cluster interior class) for each sub-pixel in the ternary map 1718, and the classification label of a given sub-pixel is determined by the corresponding softmax classification score distribution.

[0585] The ternary map 1718 represents each sub-pixel based on the predicted classification score. The ternary map 1718 also stores the predicted classification scores in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0586] train

[0587] Figure 55 It is a specific implementation of training 5500 the ternary classification model 5400 using a back-propagation based gradient update technique, which modifies the parameters of the ternary classification model 5400 until the ternary map 1718 of the ternary classification model 5400 gradually approaches or matches the reference true value ternary map 1304.

[0588] In the illustrated implementation, the final output layer of the ternary classification model 5400 is a softmax-based sub-pixel classification layer. In the softmax implementation, the ground truth ternary map generator 1402 assigns each ground truth sub-pixel to (i) a background value triplet (e.g., [1, 0, 0]), (ii) a cluster center value triplet (e.g., [0, 1, 0]), or (iii) a cluster / cluster interior value triplet (e.g., [0, 0, 1]).

[0589] Background sub-pixels are assigned a background value triplet. Centroid (COM) sub-pixels are assigned a cluster center value triplet. Cluster / cluster interior sub-pixels are assigned a cluster / cluster interior value triplet.

[0590] In the background value triplet [1,0,0], the first value [1] represents the background class label, the second value [0] represents the cluster center label, and the third value [0] represents the cluster / cluster interior class label.

[0591] In the cluster center value triplet [0, 1, 0], the first value [0] represents the background class label, the second value [1] represents the cluster center label, and the third value [0] represents the cluster / cluster interior class label.

[0592] In the cluster / cluster-inside value triplet [0, 0, 1], the first value [0] represents the background class label, the second value [0] represents the cluster center label, and the third value [1] represents the cluster / cluster-inside class label.

[0593] The ground truth ternary map 1304 represents each sub-pixel based on an assigned value triplet. The ground truth ternary map 1304 also stores the assigned triplet in a cell array, where each cell in the array represents a corresponding sub-pixel in the input.

[0594] Training includes iteratively optimizing a loss function that minimizes an error 5506 (eg, a softmax error) between the ternary map 1718 and the ground truth ternary map 1304 , and updating parameters of the ternary classification model 5400 based on the error 5506 .

[0595] In one embodiment, the loss function is a custom weighted categorical cross entropy loss, and the error 5506 is minimized on a sub-pixel-by-sub-pixel basis between the predicted classification scores (e.g., softmax scores) and the labeled class scores (e.g., softmax scores) for the corresponding sub-pixels in the ternary map 1718 and the ground truth ternary map 1304, as Figure 54 shown.

[0596] The custom weighted loss function assigns more weight to the COM sub-pixels, so that whenever a COM sub-pixel is misclassified, the cross-entropy loss is multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Additional details about the custom weighted loss function can be found in the appendix titled "Custom Weighted Loss Function".

[0597] Training 5500 includes hundreds, thousands, and / or millions of iterations of forward propagation 5508 and backward propagation 5510, including parallelization techniques such as batch processing. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 5500 is performed by trainer 1510 using a stochastic gradient update algorithm such as ADAM.

[0598] Figure 56 One specific implementation of the input image data 1702 fed to the ternary classification model 5400 and the corresponding class labels used to train the ternary classification model 5400 is shown.

[0599] In the illustrated embodiment, the input image data 1702 includes a series of upsampled and downsized image sets 5602. The class labels 5604 include three classes: (1) "background class", (2) "cluster center class", and (3) "cluster interior class", which are distinguished using different output values. For example, some of these different output values ​​can be visually represented as follows: (1) gray cells / sub-pixels 5606 represent sub-pixels predicted by the ternary classification model 5400 to be background, (2) dark green cells / sub-pixels 5608 represent sub-pixels predicted by the ternary classification model 5400 to contain cluster centers, and (3) light green sub-pixels 5610 represent sub-pixels predicted by the ternary classification model 5400 to contain cluster interiors.

[0600] Network Architecture

[0601] Figure 57 is a table showing an example architecture of the ternary classification model 5400, as well as details of the layers of the ternary classification model 5400, the dimensions of the outputs of the layers, the magnitudes of the model parameters, and the interconnections between the layers. Similar details are disclosed in the appendix titled "Ternary_Classification_Model_Example_Architecture".

[0602] infer

[0603] Figure 58 5800 , wherein the ternary map 1718 is generated by the ternary classification model 5400 as an inference output during inference 5800. An example of the ternary map 1718 is disclosed in the appendix entitled "Ternary_Classification_Model_Sample_Output." The appendix includes unit-by-unit binary classification scores 5810 that together represent the ternary map 1718. In the softmax application, the appendix has a first array 5802a of unit-by-unit classification scores for the background class, a second array 5802b of unit-by-unit classification scores for the cluster center class, and a third array 5802c of unit-by-unit classification scores for the cluster / cluster interior class.

[0604] Inference 5800 includes hundreds, thousands, and / or millions of iterations of forward propagation 5804, including parallelization techniques such as batch processing. Inference 5800 is performed on inference data 2908, which includes a series of upsampled and downsized image sets as input image data 1702. Inference 5000 is performed by tester 2906.

[0605] In some implementations, the ternary map 1718 is generated by the ternary classification model 5400 using the post-processing techniques discussed above, such as thresholding, peak detection, and / or watershed segmentation.

[0606] Figure 59 The ternary map 1718 produced by the ternary classification model 5400 is graphically depicted, where each sub-pixel has a three-way softmax classification score distribution for three corresponding classes (i.e., background class 5906, cluster center class 5902, and cluster / cluster interior class 5904).

[0607] Figure 60 5904 ).

[0608] Peak detection and watershed segmentation

[0609] Figure 61 A specific implementation of post-processing the ternary map 1718 to identify cluster centers, cluster backgrounds, and cluster interiors is shown. As described above, the ternary map 1718 is an array of cells that classify each sub-pixel based on a predicted classification score, where each cell in the array represents a corresponding sub-pixel in the input. The classification score can be a softmax score.

[0610] In the softmax application, the ternary map 1718 includes three arrays: (1) a first array 5802a of unit-by-unit classification scores for the background class, (2) a second array 5802b of unit-by-unit classification scores for the cluster center class, and (3) a third array 5802c of unit-by-unit classification scores for the cluster interior class. In all three arrays, each cell represents a corresponding sub-pixel in the input.

[0611] To determine which sub-pixels in the input contain cluster centers, which sub-pixels contain the interior of the cluster, and which sub-pixels contain the background, peak locator 1806 applies peak detection to the softmax values ​​in ternary map 1718 for cluster center class 5802b. Peak detection identifies those cells whose classification scores (e.g., softmax scores) are above a preset threshold. The identified cells are inferred to be cluster centers, and the corresponding sub-pixels of these cells in the input are determined to contain cluster centers and are stored as cluster center sub-pixels in sub-pixel classification and segmentation data storage 6102. Additional details regarding peak locator 1806 can be found in the Appendix entitled "Peak Detection."

[0612] In some implementations, before applying peak detection, those cells whose classification scores are below a certain noise threshold (e.g., 0.3) are set to zero. Such cells can be considered noise and ignored.

[0613] In addition, cells with classification scores for the background class 5802a above a certain background threshold (e.g., equal to or greater than 0.5) and their corresponding sub-pixels in the input are inferred to represent the background surrounding the cluster and are stored as background sub-pixels in the sub-pixel classification and segmentation data storage device 6102.

[0614] Then, a watershed segmentation algorithm operated by watershed segmenter 3102 is used to determine the shape of the clusters. In some implementations, background cells / sub-pixels are used as masks by the watershed segmentation algorithm. The classification scores of cells / sub-pixels inferred as cluster centers and cluster interiors are summed to produce a so-called "cluster marker." The cluster centers serve as watershed markers for separation by intensity valleys by the watershed segmentation algorithm.

[0615] In one embodiment, the negative feature cluster markers are provided as an input image to a watershed segmenter 3102, which performs segmentation and generates cluster shapes as disjoint regions of adjacent cluster interior sub-pixels separated by background sub-pixels. In addition, each disjoint region includes a corresponding cluster center sub-pixel. In some embodiments, the corresponding cluster center sub-pixel is the center of the disjoint region to which it belongs. In other embodiments, the center of mass (COM) of the disjoint region is calculated based on the base position coordinates and stored as the new center of the cluster.

[0616] The output of the watershed segmenter 3102 is stored in the sub-pixel classification and segmentation data storage 6102. Additional details regarding the watershed segmentation algorithm and other segmentation algorithms can be found in the Appendix entitled "Watershed Segmentation."

[0617] Example outputs of the peak locator 1806 and watershed segmenter 3102 are shown in Figure 62a 、 Figure 62b 、 Figure 63 and Figure 64 Shown in.

[0618] Example model output

[0619] Figure 62a Example predictions for the ternary classification model 5400 are shown. Figure 62a Four maps are shown, each with an array of cells. The first map 6202 (leftmost) shows the output value for each cell of the cluster center class 5802b. The second map 6204 shows the output value for each cell of the cluster / cluster interior class 5802c. The third map 6206 (rightmost) shows the output value for each cell of the background class 5802a. The fourth map 6208 (bottom) is a binary mask of the ground truth ternary map 6008, which assigns the class label with the highest output value to each cell.

[0620] Figure 62b Other exemplary predictions of the ternary classification model 5400 are shown. Figure 62b Four maps are shown, each with an array of cells. The first map 6212 (bottom left) shows the output value of each cell of the cluster / cluster interior class. The second map 6214 shows the output value of each cell of the cluster center class. The third map 6216 (bottom right) shows the output value of each cell of the background class. The fourth map (top) 6210 is the ground truth ternary map, which assigns the class label with the highest output value to each cell.

[0621] Figure 62c Other exemplary predictions of the ternary classification model 5400 are also shown. Figure 64 Four maps are shown, each with an array of cells. The first map 6220 (bottom left) shows the output value of each cell of the cluster / cluster interior class. The second map 6222 shows the output value of each cell of the cluster center class. The third map 6224 (bottom right) shows the output value of each cell of the background class. The fourth map 6218 (top) is the ground truth ternary map, which assigns the class label with the highest output value to each cell.

[0622] Figure 63 Describes the post-processing of the output from Figure 62a The output of the ternary classification model 5400 in

[00155] derives one specific implementation of cluster centers and cluster shapes. Post-processing (e.g., peak location, watershed segmentation) generates cluster shape data and other metadata identified in the cluster map 6310.

[0623] Experimental results and observations

[0624] Figure 64The performance of the binary classification model 4600, the regression model 2600, and the RTA base caller were compared. Performance was evaluated using a variety of sequencing metrics. One metric was the total number of clusters detected ("# clusters"), which can be measured by the number of unique cluster centers detected. Another metric was the number of clusters detected that passed the purity filter ("% PF" (passed filter)). During cycles 1-25 of the sequencing run, the purity filter removed the least reliable clusters from the image extraction results. A cluster "passed the filter" if no more than one base call had a purity value below 0.6 in the first 25 cycles. Purity is defined as the ratio of the brightest base signal intensity divided by the sum of the brightest and second brightest base signal intensities. This metric goes beyond the number of clusters detected and also conveys their quality, i.e., how many of the detected clusters can be used for accurate base calling and downstream binary and ternary analyses (such as variant calling and variant pathogenicity annotation).

[0625] Other metrics for measuring how well a detected cluster performs for downstream analysis include the number of aligned reads generated by the detected cluster ("% aligned"), the number of duplicate reads generated by the detected cluster ("% duplicates"), the number of reads generated by the detected cluster that mismatch the reference sequence among all reads aligned to the reference sequence ("% mismatches"), the number of reads generated by the detected cluster whose portions did not match the reference sequence on either side well and were therefore ignored in the alignment ("% soft clipping"), the number of bases detected for the detected cluster with a quality score of 30 or above ("% Q30 bases"), the number of paired reads generated by the detected cluster in which both reads are aligned within a reasonable distance ("total correct read pairs"), and the number of unique or deduplicated correct read pairs generated by the detected cluster ("non-duplicate correct read pairs").

[0626] like Figure 64 As shown, in template generation, the binary classification model 4600 and the regression model 2600 outperform the RTA base caller in most metrics.

[0627] Figure 65 The performance of the ternary classification model 5400 was compared to that of the RTA base caller under three contexts, five sequencing metrics, and two run densities.

[0628] In a first context, referred to as "RTA," cluster centers are detected by the RTA base caller, intensity extraction of clusters is performed by the RTA base caller, and clusters are also base called using the RTA base caller. In a second context, referred to as "RTA IE," cluster centers are detected by the ternary classification model 5400; however, intensity extraction of clusters is performed by the RTA base caller, and clusters are also base called using the RTA base caller. In a third context, referred to as "Self IE," cluster centers are detected by the ternary classification model 5400, and intensity extraction of clusters is performed using the cluster shape-based intensity extraction technique disclosed herein (note that the cluster shape information is generated by the ternary classification model 5400); however, clusters are base called using the RTA base caller.

[0629] Performance was compared between the ternary classification model 5400 and the RTA base caller based on five metrics: (1) the total number of clusters detected ("#clusters"), (2) the number of clusters detected that passed the purity filter ("#PF"), (3) the number of unique or deduplicated correct read pairs generated by the detected clusters ("#non-duplicate correct read pairs"), (4) the mismatch rate between the sequence reads generated by the detected clusters and the reference sequence after alignment ("%Mismatch Rate"), and (5) the bases called with a quality score of 30 or above for the detected clusters ("%Q30").

[0630] The performance between the ternary classification model 5400 and the RTA base caller was compared under five metrics in three contexts and two types of sequencing runs: (1) normal runs with a 20 pM library concentration and (2) dense runs with a 30 pM library concentration.

[0631] like Figure 65 As shown, the ternary classification model 5400 outperforms the RTA base caller in all metrics.

[0632] Under the same three contexts, five indicators and two run densities, Figure 66 It is shown that regression model 2600 outperforms the RTA base caller in all metrics.

[0633] Figure 67 Focus on the penultimate layer 6702 of the neural network based template generator 1512.

[0634] Figure 68 The example implementation visualizes 24 of the 32 convolutional filters of the penultimate layer 6702 superimposed on the ground truth cluster shapes. Figure 68As shown, the penultimate layer 6702 has learned cluster metadata, including the spatial distribution of clusters, such as cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and cluster boundaries.

[0635] Figure 69 Cluster center predictions (blue) from the binary classification model 4600 are superimposed on those from the RTA base caller (pink). Predictions were made on sequencing image data from an Illumina NextSeq sequencer.

[0636] Figure 70 Cluster center predictions (pink) made by the RTA base caller are superimposed on a visualization of the trained convolutional filters of the penultimate layer of the binary classification model 4600. These convolutional filters were learned as a result of training on sequencing image data from an Illumina NextSeq sequencer.

[0637] Figure 71 One embodiment of training data for training a neural network-based template generator 1512 is shown. In this embodiment, the training data is obtained from a dense flow cell that produces data with STORM probe images. In another embodiment, the training data is obtained from a dense flow cell that produces data with fewer bridge amplification cycles.

[0638] Figure 72 It is a specific implementation of image registration using beads based on cluster center prediction of the neural network based template generator 1512.

[0639] Figure 73 One specific implementation of cluster statistics is shown for clusters identified by the neural network based template generator 1512. The cluster statistics include cluster size based on the number of contributing sub-pixels and GC content.

[0640] Figure 74 17 shows how the ability of the neural network-based template generator 1512 to distinguish adjacent clusters improves when the number of initial sequencing cycles using the input image data 1702 increases from five to seven. For five sequencing cycles, a single cluster is identified by a single non-intersecting region of adjacent sub-pixels. For seven sequencing cycles, the single cluster is split into two adjacent clusters, each with its own non-intersecting region of adjacent sub-pixels.

[0641] Figure 75 Shown is the difference in base calling performance when the RTA base caller uses the ground truth centroid (COM) position as the cluster center instead of using a non-COM position as the cluster center.

[0642] Figure 76The performance of the neural network based template generator 1512 for additional detected clusters is depicted.

[0643] Figure 77 Different data sets used to train the neural network based template generator 1512 are shown.

[0644] Figure 78 The processing stages used by an RTA base caller for base calling are shown according to one specific implementation. Figure 78 Also shown are the processing stages used by the disclosed neural network-based base caller for base calling according to two specific implementations. Figure 78 As shown, the neural network-based base caller 1514 can simplify the base calling process by eliminating many of the processing stages used by the RTA base caller. This simplification improves the accuracy and scale of base calling. In a first embodiment of the neural network-based base caller 1514, the neural network-based base caller uses the position / location information of the cluster center identified from the output of the neural network-based template generator 1512 to perform base calling. In a second embodiment, the neural network-based base caller 1514 does not use the position / location information of the cluster center to perform base calling. When a patterned flow cell is designed for cluster generation, the second embodiment is used. The patterned flow cell includes a nanowell that is precisely positioned relative to a known reference position and provides a pre-arranged cluster distribution on the patterned flow cell. In other embodiments, the neural network-based base caller 1514 performs base calling on the clusters generated on the random flow cell.

[0645] Neural network-based base calling

[0646] We will now discuss neural network-based base calling, where a neural network is trained to map sequence images to base calls. This discussion will proceed as follows. First, we will describe the inputs to the neural network. Next, we will describe the structure and form of the neural network. Finally, we will describe the outputs of the neural network.

[0647] enter

[0648] Figure 79 One specific implementation of base calling using neural network 7906 is illustrated.

[0649] Main input: image channel

[0650] The primary input to the neural network 7906 is image data 7902. The image data 7902 originates from the sequencing image 108 generated by the sequencer 102 during a sequencing run. In one embodiment, the image data 7902 comprises n×n image patches extracted from the sequencing image 102, where n is any number in the range of 1 to 10,000. The sequencing run generates m images per sequencing cycle for the corresponding m image channels, and an image patch is extracted from each of the m images to prepare image data for the particular sequencing cycle. In different embodiments, such as four-channel chemistry, two-channel chemistry, and single-channel chemistry, m is 4 or 2. In other embodiments, m is 1, 3, or greater than 4. In some embodiments, the image data 7902 is in the optical pixel domain, and in other embodiments, in the upsampled sub-pixel domain.

[0651] Image data 7902 includes data for multiple sequencing cycles (e.g., a current sequencing cycle, one or more previous sequencing cycles, and one or more subsequent sequencing cycles). In one implementation, image data 7902 includes data for three sequencing cycles, such that the data for the pending base call for the current (time t) sequencing cycle is accompanied by (i) data for the left flank / context / previous / previous / previous (time t-1) sequencing cycle and (ii) data for the right flank / context / next / subsequent / after (time t+1) sequencing cycle. In other implementations, image data 7902 includes data for a single sequencing cycle.

[0652] Image data 7902 depicts the intensity emission of one or more clusters and their surrounding background. In one embodiment, when base calling is to be performed on a single target cluster, image patches are extracted from sequencing image 108 in such a way that each image patch contains the center of the target cluster in its center pixel, a concept referred to herein as "target cluster-centered patch extraction."

[0653] Image data 7902 is encoded in input data 7904 using intensity channels (also referred to as image channels). For each of the m images obtained from the sequencer 102 for a particular sequencing cycle, its intensity data is encoded using a separate image channel. For example, consider a sequencing run using a dual-channel chemistry that produces a red image and a green image in each sequencing cycle. Then, the input data 7904 includes (i) a first red image channel having n×n pixels, which depicts the intensity emission of the one or more clusters and their surrounding background captured in the red image; and (ii) a second green imaging channel having n×n pixels, which depicts the intensity emission of the one or more clusters and their surrounding background captured in the green image.

[0654] Supplementary input: distance channel

[0655] Image data 7902 is accompanied by supplementary distance data (also referred to as a distance channel). The distance channel provides an additive bias that is incorporated into the feature map generated from the image channel. This additive bias contributes to the accuracy of base calling because it is based on the distance from the pixel center to the cluster center, which is encoded pixel by pixel in the distance channel.

[0656] In a "single target cluster" base calling implementation, for each image channel (image patch) in input data 7904, a supplementary distance channel identifies the distance of the center of its pixels from the center of the target cluster containing its center pixel and for which base calling is to be performed. The distance channel thus indicates the respective distances of the pixels of the image patch from the center pixel of the image patch.

[0657] In a "multi-cluster" base calling implementation, for each image channel (image patch) in the input data 7904, a supplemental distance channel identifies the center-to-center distance of each pixel from the nearest one of these clusters, where the nearest cluster is selected based on the center-to-center distance between the pixel and each of these clusters.

[0658] In a "multi-cluster shape-based" base calling implementation, for each image channel (image patch) in the input data 7904, a supplemental distance channel identifies the center-to-center distance of each cluster pixel from an assigned cluster, which is selected based on classifying each cluster pixel into only one cluster.

[0659] Supplementary input: scaling channel

[0660] The image data 7902 is accompanied by supplemental scaling data (also referred to as a scaling channel) that accounts for varying cluster sizes and non-uniform lighting conditions. The scaling channel also provides an additive bias that is incorporated into the feature map generated from the image channel. This additive bias contributes to the accuracy of base calling because it is based on the average intensity of the center cluster pixels encoded pixel by pixel in the scaling channel.

[0661] Supplementary input: cluster center coordinates

[0662] In some implementations, location / position information 7916 (e.g., xy coordinates) of cluster centers identified from the output of the neural network-based template generator 1512 is fed to the neural network 7906 as a supplemental input.

[0663] Supplementary input: cluster affiliation information

[0664] In some implementations, the neural network 7906 receives as a supplemental input cluster affiliation information that classifies pixels or subpixels as background pixels or subpixels, cluster center pixels or subpixels, and cluster / cluster interior pixels or subpixels that depict / contribute to / belong to the same cluster. In other implementations, an attenuation map, a binary map, and / or a ternary map, or variations thereof, are fed as supplemental inputs to the neural network 7906.

[0665] Preprocessing: Intensity Modification

[0666] In some implementations, the input data 7904 does not include a range channel, and instead the neural network 7906 receives as input modified image data that has been modified according to the output (i.e., attenuation map, binary map, and / or ternary map) of the neural network-based template generator 1512. In such implementations, the intensity of the image data 7902 is modified to account for the absence of the range channel.

[0667] In other specific implementations, the image data 7902 is subjected to one or more lossless transform operations (e.g., convolution, deconvolution, Fourier transform), and the resulting modified image data is fed as input to the neural network 7906.

[0668] Network structure and form

[0669] Neural network 7906 is also referred to herein as "neural network-based base caller" 1514. In one implementation, neural network-based base caller 1514 is a multilayer perceptron. In another implementation, neural network-based base caller 1514 is a feedforward neural network. In yet another implementation, neural network-based base caller 1514 is a fully connected neural network. In another implementation, neural network-based base caller 1514 is a fully convolutional neural network. In yet another implementation, neural network-based base caller 1514 is a semantic segmentation neural network.

[0670] In one implementation, the neural network-based base caller 1514 is a convolutional neural network (CNN) having multiple convolutional layers. In another implementation, the neural network-based base caller is a recurrent neural network (RNN), such as a long short-term memory network (LSTM), a bidirectional LSTM (Bi-LSTM), or a gated recurrent unit (GRU). In yet another implementation, the neural network-based base caller includes both a CNN and an RNN.

[0671] In other embodiments, the neural network-based base caller 1514 may use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or atrous convolution, transposed convolution, depthwise separable convolution, pointwise convolution, 1×1 convolution, grouped convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It may use one or more loss functions, such as logistic regression / logarithmic loss, multi-class cross entropy / softmax loss, binary cross entropy loss, mean squared error loss, L1 loss, L2 loss, smoothed L1 loss, and Huber loss. It may use any parallelism, efficiency, and compression scheme, such as TFRecords, compressed encoding (e.g., PNG), sharpening, parallel calling of map transformations, batching, prefetching, model parallelism, data parallelism, and synchronous / asynchronous SGD. It may include upsampling layers, downsampling layers, recurrent connections, gate and gate memory units (such as LSTM or GRU), residual blocks, residual connections, high-speed connections, skip connections, peephole connections, activation functions (for example, nonlinear transformation functions such as rectified linear unit (ReLU), leaky ReLU, exponential lining unit (ELU), sigmoid and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout layers, pooling layers (for example, maximum or average pooling), global average pooling layers and attention mechanisms.

[0672] The neural network-based base caller 1514 processes the input data 7904 and generates an alternative representation 7908 of the input data 7904. The alternative representation 7908 is a convolutional representation in some implementations and a hidden representation in other implementations. The alternative representation 7908 is then processed by the output layer 7910 to generate an output 7912. The output 7912 is used to generate a base call, as described below.

[0673] Output

[0674] In one implementation, the neural network-based base caller 1514 outputs base calls for a single target cluster for a particular sequencing cycle. In another implementation, the neural network-based base caller outputs base calls for each target cluster in a plurality of target clusters for a particular sequencing cycle. In yet another implementation, the neural network-based base caller outputs base calls for each target cluster in a plurality of target clusters for each sequencing cycle in a plurality of sequencing cycles, thereby generating a base call sequence for each target cluster.

[0675] Distance channel calculation

[0676] We now discuss how to obtain appropriate position / location information (eg, xy coordinates) of the cluster centers for use in calculating the distance values ​​of the distance channel.

[0677] Reduce coordinates

[0678] Figure 80 It is a specific implementation of transforming the location / position information of the cluster centers identified from the output of the neural network-based template generator 1512 from the sub-pixel domain to the pixel domain.

[0679] The cluster center position / localization information is used for neural network-based base calling to at least: (i) construct input data by extracting an image patch from the sequencing image 108 that contains the center of the target cluster to be base called in its center pixel, (ii) construct a distance channel that identifies the distance of the center of the pixel of the image patch from the center of the target cluster containing its center pixel, and / or (iii) as a supplemental input 7916 to the neural network-based base caller 1514.

[0680] In some implementations, cluster center position / localization information is identified from the output of the neural network-based template generator 1512 at an upsampled sub-pixel resolution. However, in some implementations, the neural network-based base caller 1514 operates on image data at optical pixel resolution. Therefore, in one implementation, the cluster center position / localization information is transformed into the pixel domain by scaling down the coordinates of the cluster centers by the same upsampling factor used to upsample the image data fed as input to the neural network-based template generator 1512.

[0681] For example, consider that the image patch data fed as input to the neural network-based template generator 1512 is obtained by upsampling the image 108 by the above sampling factor f in some initial sequencing cycles. Then, in one embodiment, the coordinates of the cluster centers 8002 generated by the neural network-based template generator 1512 and the post-processor 1814 and stored in the template / template image 8004 are divided by f (divider). These reduced cluster center coordinates are referred to herein as "reference cluster centers" 8008 and are stored in the template / template image 8004. In one embodiment, this reduction is performed by a reducer 8006.

[0682] Coordinate transformation

[0683] Figure 81 is one specific implementation that uses a cycle-specific transform and an image channel-specific transform to derive the so-called "transformed cluster centers" 8104 from the reference cluster centers 8008. The motivation for doing so is first discussed.

[0684] Sequencing images acquired at different sequencing cycles are misaligned relative to each other and have random translational offsets. This occurs due to the limited motion accuracy of the sensor's motion stage and also because images acquired in different image / frequency channels have different optical paths and wavelengths. As a result, there is an offset between the reference cluster center and the position / location of the cluster center in the sequencing image. This offset varies between images captured at different sequencing cycles and within images captured at the same sequencing cycle in different image channels.

[0685] To account for this shift, cycle-specific and image channel-specific transformations are applied to the reference cluster centers to produce corresponding transformed cluster centers for the image patches of each sequencing cycle. The cycle-specific and image channel-specific transformations are obtained through an image registration process that uses image correlation to determine a full six-parameter affine transformation (e.g., translation, rotation, scaling, shearing, right reflection, left reflection) or a Procrustes transformation (e.g., translation, rotation, scaling, optionally extended to aspect ratio), further details of which can be found in Appendices 1, 2, 3, and 4.

[0686] For example, consider that the reference cluster centers for the four cluster centers are (x1, y1); (x2, y2); (x3, y3); (x4, y4), and the sequencing run uses a dual-channel chemistry where a red image and a green image are produced at each sequencing cycle. Then, for exemplary sequencing cycle 3, the cycle-specific and image-channel-specific transformations for the red image are The cycle-specific and image channel-specific transforms for the green image are

[0687] Similarly, for exemplary sequencing cycle 9, the cycle-specific and image channel-specific transformations for the red image are The cycle-specific and image channel-specific transforms for the green image are

[0688] Then, the transformed cluster centers of the red image of sequencing cycle 3 is by transforming Transformed cluster centers of the green image obtained by applying the transformation to the reference cluster centers (x1, y1); (x2, y2); (x3, y3); (x4, y4) and sequencing cycle 3 is by transforming Applied to the reference cluster centers (x1,y1); (x2,y2); (x3,y3); (x4,y4).

[0689] Similarly, the transformed cluster centers of the red image of sequencing cycle 9 is by transforming Transformed cluster centers of the green image at sequencing cycle 9, applied to the reference cluster centers (x1, y1); (x2, y2); (x3, y3); (x4, y4) is by transforming Applied to the reference cluster centers (x1,y1); (x2,y2); (x3,y3); (x4,y4).

[0690] In one embodiment, these transformations are performed by transformer 8102.

[0691] The transformed cluster centers 8104 are stored in the template / template image 8004 and are respectively: (i) used for patch extraction from the corresponding sequencing image 108 (e.g., by a patch extractor 8106), (ii) used in the distance formula , used to calculate the distance channel for the corresponding image patch, and (iii) as a supplementary input to the neural network-based base caller 1514 for the corresponding sequencing cycle that was base called. In other specific implementations, different distance formulas can be used, such as distance squared, e^-distance, and e^-distance squared.

[0692] Image patch

[0693] Figure 82 Image patches 8202 are shown as part of the input data fed to the neural network-based base caller 1514. The input data includes a sequence of per-cycle image patch sets generated for a series of sequencing cycles of a sequencing run. Each per-cycle image patch set in the sequence has an image patch for a corresponding image channel in one or more image channels.

[0694] For example, consider that the sequencing run uses a dual-channel chemistry that produces a red image and a green image at each sequencing cycle, and that the input data includes data spanning a series of three sequencing cycles of the sequencing run: the current (time t) sequencing cycle at which a base call is to be made, the previous (time t-1) sequencing cycle, and the subsequent (time t+1) sequencing cycle.

[0695] The input data then includes the following sequence of image patch sets for each cycle: a current cycle image patch set having a current red image patch and a current green image patch extracted from the red sequencing image and the green sequencing image captured at the current sequencing cycle, respectively; a previous cycle image patch set having a previous red image patch extracted from the red sequencing image and the green sequencing image captured at the previous sequencing cycle, and a subsequent red image patch and a subsequent green image patch extracted from the red sequencing image and the green sequencing image captured at the subsequent sequencing cycle, respectively.

[0696] Each image patch may be of size n x n, where n may be any number in the range of 1 to 10,000. Each image patch may be in the optical pixel domain or in the upsampled sub-pixel domain. Figure 82 In the illustrated implementation, the extracted image patch 8202 includes pixel intensity data for pixels covering / depicting the plurality of clusters 1-m and their surrounding background. Furthermore, in the illustrated implementation, the image patch 8202 is extracted such that the center pixel of the image patch contains the center of the base-called target cluster.

[0697] exist Figure 82 In , pixel centers are depicted by black rectangles and have integer position / location coordinates, and cluster centers are depicted by purple circles and have floating point position / location coordinates.

[0698] Distance calculation for a single target cluster

[0699] Figure 83 One specific implementation of determining distance values ​​8302 for a distance channel when a single target cluster is being base called by the neural network-based base caller 1514 is depicted. The center of the target cluster is contained in the center pixel of an image patch fed as input to the neural network-based base caller 1514. The distance value is calculated pixel by pixel such that for each pixel the distance between its center and the center of the target cluster is determined. Thus, a distance value is calculated for each pixel in each of the image patches that are part of the input data.

[0700] Figure 83 Three distance values ​​d1, dc, and dn are shown for a particular image patch. In one embodiment, the following distance formula is used to calculate the distance value 8302: The distance formula operates based on the transformed cluster centers 8104. In other implementations, different distance formulas may be used, such as distance squared, e^-distance, and e^-distance squared.

[0701] In other implementations, when the image patch is at an upsampled sub-pixel resolution, the distance value 8302 is calculated in the sub-pixel domain.

[0702] Therefore, in a single target cluster base calling implementation, the range channel is calculated only relative to the target cluster being base called.

[0703] Figure 84 A specific implementation of pixel-by-pixel encoding 8402 of the distance value 8302 calculated between the pixel and the target cluster is shown. In one specific implementation, the distance value 8302, which is part of the distance channel in the input data, is supplemented as "pixel distance data" for each corresponding image channel (image patch). Returning to the example of a red image and a green image generated by a sequencing cycle, the input data includes a red distance channel and a green distance channel, and the red image channel and the green image channel are supplemented as pixel distance data.

[0704] In other implementations, when the image patch is at an upsampled sub-pixel resolution, the range channel is encoded on a sub-pixel basis.

[0705] Distance calculation for multiple target clusters

[0706] Figure 85a Depicted is one specific implementation of determining distance values ​​8502 for a distance channel when multiple target clusters 1-m are simultaneously base called by a neural network-based base caller 1514. The distance values ​​are calculated pixel by pixel, such that for each pixel, the distance between its center and the corresponding center of each of the multiple clusters 1-m is determined, and the minimum distance value (in red) is assigned to the pixel.

[0707] Thus, the distance channel identifies the center-to-center distance of each pixel from the nearest cluster of the clusters, which is selected based on the center-to-center distance between the pixel and each of the clusters. In the illustrated implementation, Figure 85a The pixel center to cluster center distances for two pixels and four cluster centers are shown. Pixel 1 is closest to cluster 1, and pixel n is closest to cluster 3.

[0708] In one specific implementation, the distance value 8502 is calculated using the following distance formula: The distance formula operates based on the transformed cluster centers 8104. In other implementations, different distance formulas may be used, such as distance squared, e^-distance, and e^-distance squared.

[0709] In other implementations, when the image patch is at an upsampled sub-pixel resolution, the distance value 8502 is calculated in the sub-pixel domain.

[0710] Therefore, in multiple cluster base calling implementations, the distance channel is calculated relative to the nearest cluster from the multiple clusters.

[0711] Figure 85b It is shown that for each target cluster in the target clusters 1-m, some nearest pixels are determined based on the distance 8504 (d1, d2, d23, d29, d24, d32, dn, d13, d14, etc.) of the pixel center to the nearest cluster center.

[0712] Figure 86 One implementation is shown where the calculated minimum distance value between the pixel and the nearest cluster is encoded pixel by pixel 8602. In other implementations, the distance channel is encoded on a sub-pixel basis when the image patch is at an upsampled sub-pixel resolution.

[0713] Cluster shape-based distance calculation for multiple target clusters

[0714] Figure 87 One specific implementation of using pixel-to-cluster classification / attribution / categorization 8702 (referred to herein as "cluster shape data" or "cluster shape information") to determine cluster distance values ​​8802 for distance channels is illustrated when multiple target clusters 1-m are simultaneously base called by a neural network-based base caller 1514. First, the following is a brief overview of how the cluster shape data is generated.

[0715] As described above, the output of the neural network-based template generator 1512 is used to classify pixels into: background pixels, center pixels, and cluster / cluster-internal pixels that depict / contribute to / belong to the same cluster. This pixel-to-cluster classification information is used to attribute each pixel to only one cluster, regardless of the distance between the pixel center and the cluster center, and is stored as cluster shape data.

[0716] exist Figure 87 In the specific implementation shown, background pixels are colored gray, pixels belonging to cluster 1 are colored yellow (Cluster 1 pixels), pixels belonging to cluster 2 are colored green (Cluster 2 pixels), pixels belonging to cluster 3 are colored red (Cluster 3 pixels), and pixels belonging to cluster m are colored blue (Cluster m pixels).

[0717] Figure 88 A specific implementation of using cluster shape data to calculate distance values ​​8802 is shown. First, the present invention explains why distance information calculated without considering cluster shape is prone to error. Then the present invention explains how cluster shape data overcomes this limitation.

[0718] In a "multi-cluster" base calling implementation that does not use cluster shape data ( Figures 85a to 85b and Figure 86), the center-to-center distance value of a pixel is calculated relative to the nearest cluster from multiple clusters. Now, consider the case where a pixel belonging to cluster A is farther away from the center of cluster A but closer to the center of cluster B. In this case, without cluster shape data, the pixel is assigned a distance value calculated relative to cluster B (to which the pixel does not belong) rather than a distance value calculated relative to cluster A (to which the pixel does belong).

[0719] The "multi-cluster shape-based" base calling implementation avoids this by using true pixel-to-cluster mappings (as defined in the original image data and produced by the neural network-based template generator 1512).

[0720] With respect to pixels 34 and 35, a clear difference between the two implementations can be seen. Figure 85b In , the distance values ​​of pixels 34 and 35 are calculated relative to the nearest center of cluster 3, without considering the cluster shape data. However, in Figure 88 , based on the cluster shape data, the distance values ​​8802 of pixels 34 and 35 are calculated relative to cluster 2 (to which they actually belong).

[0721] exist Figure 88 In

[0015] , cluster pixels depict cluster intensity, and background pixels depict background intensity. A cluster distance value identifies the center-to-center distance of each cluster pixel from an assigned cluster in the cluster, where the assigned cluster is selected based on classifying each cluster pixel into only one of the clusters. In some implementations, background pixels are assigned a predetermined background distance value, such as 0 or 0.1 or some other minimum value.

[0722] In one specific implementation, as described above, the cluster distance value 8802 is calculated using the following distance formula: The distance formula operates based on the transformed cluster centers 8104. In other implementations, different distance formulas may be used, such as distance squared, e^-distance, and e^-distance squared.

[0723] In other implementations, when the image patch is at an upsampled sub-pixel resolution, cluster distance values ​​8802 are calculated in the sub-pixel domain, and cluster and background attribution 8702 occurs on a sub-pixel basis.

[0724] Thus, in a multiple cluster shape-based base calling implementation, a distance channel is computed relative to an assigned cluster from the multiple clusters, which is selected based on classifying each cluster pixel into only one of the clusters according to a true pixel-to-cluster mapping defined in the original image data.

[0725] Figure 89One implementation is shown where the calculated distance value between the pixel and the assigned cluster is encoded pixel by pixel 8702. In other implementations, the distance channel is encoded on a sub-pixel basis when the image patch is at an upsampled sub-pixel resolution.

[0726] Deep learning is a powerful machine learning technique that uses multi-layer neural networks. A particularly successful network architecture in the computer vision and image processing domains is the convolutional neural network (CNN), in which each layer performs a feed-forward convolution transformation from an input tensor (a multidimensional dense array similar to an image) to an output tensor of a different shape. CNNs are particularly well-suited to image-like inputs due to the spatial coherence of images and the advent of general-purpose graphics processing units (GPUs), which can quickly train arrays up to 3D or 4D. Exploiting these image-like properties leads to superior empirical performance compared to other learning methods such as support vector machines (SVMs) or multi-layer perceptrons (MLPs).

[0727] We introduce a specialized architecture that enhances the standard CNN to process image data and supplements it with both distance and scale data. More details are below.

[0728] Specialized architecture

[0729] Figure 90 One specific implementation of a specialized architecture for a neural network-based base caller 1514 is shown for isolating the processing of data from different sequencing cycles. The motivation for using a specialized architecture is first described.

[0730] As described above, the neural network-based base caller 1514 processes data for the current sequencing cycle, one or more previous sequencing cycles, and one or more subsequent sequencing cycles. The data from the additional sequencing cycles provides sequence-specific context. The neural network-based base caller 1514 learns the sequence-specific context during training and performs base calls based on this sequence-specific context. In addition, the data from the previous and next sequencing cycles provide second-order contributions to the prephasing and phasing signals for the current sequencing cycle.

[0731] Spatial convolutional layer

[0732] However, as mentioned above, images captured at different sequencing cycles and in different image channels are misaligned with respect to each other and have residual registration errors. To account for this misalignment, the specialized architecture includes spatial convolutional layers that do not mix information between sequencing cycles and only mix information within a sequencing cycle.

[0733] The spatial convolution layer uses so-called "isolated convolutions," which achieve isolation by independently processing the data for each of multiple sequencing cycles via a sequence of "dedicated, unshared" convolutions. This isolated convolution convolves the data and resulting feature maps only for a given sequencing cycle (i.e., within the cycle), and not for any other sequencing cycles.

[0734] For example, consider that the input data includes (i) current data of the current (time t) sequencing cycle for which base detection is to be performed, (ii) previous data of the previous (time t-1) sequencing cycle, and (iii) subsequent data of the previous (time t+1) sequencing cycle. The specialized architecture then initiates three separate data processing pipelines (or convolution pipelines), namely the current data processing pipeline, the previous data processing pipeline, and the subsequent data processing pipeline. The current data processing pipeline receives the current data of the current (time t) sequencing cycle as input, and independently processes the current data through multiple spatial convolution layers to produce a so-called "current spatial convolution representation" as the output of the final spatial convolution layer. The previous data processing pipeline receives the previous data of the previous (time t-1) sequencing cycle as input, and independently processes the previous data through multiple spatial convolution layers to produce a so-called "previous spatial convolution representation" as the output of the final spatial convolution layer. The subsequent data processing pipeline receives subsequent data of a subsequent (time t+1) sequencing cycle as input, and independently processes the subsequent data through multiple spatial convolutional layers to produce a so-called "subsequent spatial convolutional representation" as the output of the final spatial convolutional layer.

[0735] In some implementations, the current processing pipeline, the previous processing pipeline, and the subsequent processing pipeline are executed synchronously.

[0736] In some implementations, the spatial convolutional layer is part of a spatial convolutional network (or subnetwork) within a specialized architecture.

[0737] Temporal Convolutional Layer

[0738] The neural network-based base caller 1514 also includes a temporal convolutional layer that mixes information between sequencing cycles (i.e., inter-cycle). The temporal convolutional layer receives its input from the spatial convolutional network and operates on the spatial convolutional representation produced by the final spatial convolutional layer of the corresponding data processing pipeline.

[0739] The freedom of inter-cycle operability of temporal convolutional layers stems from the fact that the misalignment property present in the image data fed as input to the spatial convolutional network is cleaned from the spatial convolutional representation by the cascade of isolated convolutions performed by the sequence of spatial convolutional layers.

[0740] The temporal convolution layer uses so-called "combined convolutions" that convolve the input channels in subsequent inputs group by group on a sliding window basis. In one specific implementation, these subsequent inputs are the subsequent outputs produced by the previous spatial convolution layer or the previous temporal convolution layer.

[0741] In some implementations, the temporal convolutional layer is part of a temporal convolutional network (or subnetwork) within a specialized architecture. The temporal convolutional network receives its input from the spatial convolutional network. In one implementation, the first temporal convolutional layer of the temporal convolutional network combines the spatial convolutional representations between sequencing cycles on a group-by-group basis. In another implementation, subsequent temporal convolutional layers of the temporal convolutional network combine subsequent outputs of previous temporal convolutional layers.

[0742] The output of the final temporal convolutional layer is fed into an output layer that produces outputs that are used to base call one or more clusters at one or more sequencing cycles.

[0743] Below is a more detailed discussion of isolated and combined convolutions.

[0744] Isolate Convolution

[0745] During the forward pass, the specialized architecture processes information from multiple inputs in two stages. In the first stage, isolating convolutions are used to prevent information from mixing between inputs. In the second stage, combining convolutions are used to mix information between inputs. The results from the second stage are used to perform a single inference on the multiple inputs.

[0746] This differs from batch mode techniques where a convolutional layer processes multiple inputs in a batch simultaneously and performs a corresponding inference for each input in the batch. In contrast, a specialized architecture maps the multiple inputs to a single inference. The single inference can include more than one prediction, such as a classification score for each of the four bases (A, C, T, and G).

[0747] In one embodiment, the inputs are time-ordered such that each input is generated at a different time step and has multiple input channels. For example, the multiple inputs may include the following three inputs: a current input generated by a current sequencing cycle at time step (t), a previous input generated by a previous sequencing cycle at time step (t-1), and a subsequent input generated by a subsequent sequencing cycle at time step (t+1). In another embodiment, each input is derived from a current output, a previous output, and a subsequent output generated by one or more previous convolutional layers, respectively, and includes k feature maps.

[0748] In one implementation, each input may include the following five input channels: a red image channel (red), a red range channel (yellow), a green image channel (green), a green range channel (purple), and a scale channel (blue). In another implementation, each input may include k feature maps produced by a previous convolutional layer, and each feature map is considered an input channel.

[0749] Figure 91 One implementation of isolated convolution is depicted. Isolated convolution processes multiple inputs by applying a convolution filter to each input once synchronously. With isolated convolution, the convolution filters combine input channels from the same input and do not combine input channels from different inputs. In one implementation, the same convolution filter is applied synchronously to each input. In another implementation, a different convolution filter is applied synchronously to each input. In some implementations, each spatial convolution layer includes a set of k convolution filters, where each convolution filter is applied synchronously to each input.

[0750] Combined convolution

[0751] Combinatorial convolution mixes information between different inputs by grouping their corresponding input channels and applying a convolution filter to each group. The grouping of these corresponding input channels and the application of the convolution filter occur on a sliding window basis. In this context, a window spans two or more subsequent input channels, representing, for example, the output of two subsequent sequencing cycles. Since the window is a sliding window, most of the input channels are used in two or more windows.

[0752] In some implementations, the different inputs originate from a sequence of outputs generated by a previous spatial convolutional layer or a previous temporal convolutional layer. In the output sequence, these different inputs are arranged as subsequent outputs and are therefore considered subsequent inputs by a subsequent temporal convolutional layer. Then, in the subsequent temporal convolutional layer, these combined convolutions apply convolution filters to corresponding groups of input channels in these subsequent inputs.

[0753] In one embodiment, the subsequent inputs are time-ordered such that the current input is generated by the current sequencing cycle at time step (t), the previous input is generated by the previous sequencing cycle at time step (t-1), and the subsequent input is generated by the subsequent sequencing cycle at time step (t+1). In another embodiment, each subsequent input is derived from the current output, the previous output, and the subsequent output produced by one or more previous convolutional layers, respectively, and includes k feature maps.

[0754] In one implementation, each input may include the following five input channels: a red image channel (red), a red range channel (yellow), a green image channel (green), a green range channel (purple), and a scale channel (blue). In another implementation, each input may include k feature maps produced by a previous convolutional layer, and each feature map is considered an input channel.

[0755] The depth B of the convolution filter depends on the number of subsequent inputs whose corresponding input channels are convolved by the convolution filter group by group on a sliding window basis. In other words, the depth B is equal to the number of subsequent inputs and the group size in each sliding window.

[0756] exist Figure 92a In , corresponding input channels from two subsequent inputs are combined in each sliding window, and thus B=2. Figure 92b , the corresponding input channels from three subsequent inputs are combined in each sliding window, and thus B=3.

[0757] In one implementation, the sliding windows share the same convolutional filters. In another implementation, a different convolutional filter is used for each sliding window. In some implementations, each temporal convolutional layer includes a set of k convolutional filters, where each convolutional filter is applied to subsequent inputs on a sliding window basis.

[0758] filter banks

[0759] Figure 93 One specific implementation of the convolutional layers of a neural network based base caller 1514 is shown where each convolutional layer has a set of convolutional filters. Figure 93 In Figure 1, five convolutional layers are shown, each of which has a set of 64 convolutional filters. In some implementations, each spatial convolutional layer has a set of k convolutional filters, where k can be any number such as 1, 2, 8, 64, 128, 256, etc. In some implementations, each temporal convolutional layer has a set of k convolutional filters, where k can be any number such as 1, 2, 8, 64, 128, 256, etc.

[0760] Now let's discuss the supplementary scaling channel and how to calculate it.

[0761] Zoom Channel

[0762] Figure 94Two configurations of a scaling channel that complements the image channel are depicted. The scaling channel is encoded pixel by pixel in the input data fed to the neural network-based base detector 1514. Different cluster sizes and uneven lighting conditions result in a wide range of cluster intensities being extracted. The additive bias provided by the scaling channel makes the cluster intensities similar between clusters. In other specific implementations, when the image patch is at upsampled sub-pixel resolution, the scaling channel is encoded sub-pixel by sub-pixel.

[0763] When base calling a single target cluster, the scaling channels assign the same scaling value to all pixels. When base calling multiple target clusters simultaneously, the scaling channels assign different scaling values ​​to groups of pixels based on the cluster shape data.

[0764] The scaling channel 9410 has the same scaling value (s1) for all pixels. The scaling value (s1) is based on the average intensity of the center pixel containing the center of the target cluster. In one embodiment, the average intensity is calculated by averaging the intensity values ​​of the center pixel observed during two or more previous sequencing cycles, which produced A and T base calls for the target cluster.

[0765] Based on cluster shape data, scaling channel 9408 has different scaling values ​​(s1, s2, s3, sm) for the corresponding pixel groups belonging to the corresponding cluster. Each pixel group includes a center cluster pixel, and this center cluster pixel comprises the center of the corresponding cluster. The scaling value for a specific pixel group is based on the average intensity of its center cluster pixel. In a specific implementation, average intensity is calculated by averaging the intensity values ​​of the center cluster pixels observed during two or more previous sequencing cycles, and these two or more previous sequencing cycles produce A and T base detection of the corresponding cluster.

[0766] In some implementations, background pixels are assigned a background scaling value (sb), which can be 0 or 0.1 or some other minimum value.

[0767] In one implementation, the scaled channels 9406 and their scaled values ​​are determined by an intensity scaler 9404. The intensity scaler 9404 uses cluster intensity data 9402 from previous sequencing cycles to calculate the average intensity.

[0768] In other implementations, the supplemental scaling channel may be provided as an input in different ways (such as before or at the last layer of the neural network-based base caller 1514, before or at one or more intermediate layers of the neural network-based base caller 1514), and provided as a single value, rather than encoding the supplemental scaling channel pixel by pixel to match the image size.

[0769] Now let's discuss the input data fed to the neural network based base caller 1514.

[0770] Input data: image channel, distance channel and scale channel

[0771] Figure 95a One specific implementation of input data 9500 for a single sequencing cycle that produces a red image and a green image is shown. Input data 9500 includes the following:

[0772] • Red intensity data 9502 (red) for pixels in the image patch extracted from the red image. The red intensity data 9502 is encoded in the red image channel.

[0773] Red distance data 9504 (yellow) supplementing the red intensity data 9502 pixel by pixel.

[0774] Red range data 9504 is encoded in the red range channel.

[0775] • Green intensity data 9506 (green) for pixels in the image patch extracted from the green image. The green intensity data 9506 is encoded in the green image channel.

[0776] Green distance data 9508 (purple) that supplements the green intensity data 9506 pixel by pixel.

[0777] Green range data 9508 is encoded in the green range channel.

[0778] • Scaled data 9510 (blue) that complements the red intensity data 9502 and the green intensity data 9506 pixel by pixel. This scaled data 9510 is encoded in the scale channel.

[0779] In other specific implementations, the input data may include fewer or greater numbers of image channels and supplemental distance channels. In one example, for a sequencing run using four-channel chemistry, the input data includes four image channels and four supplemental distance channels for each sequencing cycle.

[0780] We now discuss how the distance channel and the scale channel contribute to base calling accuracy.

[0781] Additive bias

[0782] Figure 95b We illustrate one implementation of a distance channel that provides an additive bias that is incorporated into the feature maps generated from the image channels. This additive bias contributes to base calling accuracy because it is based on the distance from the pixel center to the cluster center, which is encoded pixel by pixel in the distance channel.

[0783] On average, approximately 3×3 pixels comprise a cluster. The density at the center of a cluster is expected to be higher than at the edges, as clusters grow outward from a substantially central location. Peripheral cluster pixels may contain conflicting signals from nearby clusters. Therefore, the center cluster pixels are considered the region of maximum intensity and serve as a beacon to reliably identify clusters.

[0784] The pixels of the image patch depict the intensity emissions of multiple clusters (e.g., 10 to 200 clusters) and their surrounding background. Additional clusters contain information from a wider radius and contribute to base call prediction by distinguishing the underlying base whose intensity emission is depicted in the image patch. In other words, the intensity emissions from a group of clusters cumulatively create an intensity pattern that can be assigned to a discrete base (A, C, T, or G).

[0785] We observed that explicitly conveying the distance of each pixel from the cluster center in the supplementary distance channels to the convolutional filters resulted in higher base calling accuracy. These distance channels convey to the convolutional filters which pixels contain the cluster center and which pixels are further away from the cluster center. These convolutional filters use this information to assign sequencing signals to their appropriate source clusters by focusing on (a) the center cluster pixels, their neighboring pixels, and the feature maps derived from them, rather than (b) the peripheral cluster pixels, background pixels, and the feature maps derived from them. In one example of this focus, these distance channels provide a positive additive bias incorporated in the feature map generated by (a), but provide a negative additive bias incorporated in the feature map generated by (b).

[0786] These range channels have the same dimensionality as the image channels. This allows the convolutional filters to evaluate the image and range channels separately within the local receptive field and coherently combine these evaluations.

[0787] When base calling a single target cluster, these range channels identify only one center cluster pixel at the center of the image patch. When base calling multiple target clusters simultaneously, these range channels identify multiple center cluster pixels distributed across the image patch.

[0788] The "single cluster" distance channel is applied to an image patch whose center pixel contains the center of a single target cluster to be base called. The single cluster distance channel includes the center-to-center distance from each pixel in the image patch to the single target cluster. In this implementation, the image patch also includes additional clusters adjacent to the single target cluster, but base calling is not performed on these additional clusters.

[0789] The "Multiple Clusters" distance channel is applied to image patches that contain the centers of multiple target clusters to be base called in their corresponding center cluster pixels. This multiple cluster distance channel includes the center-to-center distance from each pixel in the image patch to the nearest cluster in the multiple target clusters. This has the potential to measure the center-to-center distance of the wrong cluster, but this probability is low.

[0790] The "multi-cluster shape-based" distance channel is applied to an image patch that contains the centers of multiple target clusters to be base called in its corresponding center cluster pixel, and for which pixel-to-cluster assignment information is known. The multi-cluster distance channel includes the center-to-center distance of each cluster pixel in the image patch to the cluster to which it belongs or is assigned in the multiple target clusters. Background pixels may be labeled as background rather than given a calculated distance.

[0791] Figure 95b A specific implementation of a scale channel is also illustrated that provides an additive bias that is incorporated into the feature map generated from the image channel. This additive bias contributes to base calling accuracy because it is based on the average intensity of the center cluster pixels encoded pixel by pixel in the scale channel. The discussion of additive bias in the context of the range channel is similarly applicable to the scale channel.

[0792] Example of Additive Bias

[0793] Figure 95b An example is also shown of how additive biases are derived from the distance and scale channels and incorporated into the feature maps generated from the image channel.

[0794] exist Figure 95b In FIG, convolution filter i 9514 evaluates a local receptive field 9512 (magenta) across two image channels 9502 and 9506, two range channels 9504 and 9508, and a scale channel 9510. Because these range and scale channels are encoded separately, additive biasing occurs when the intermediate outputs 9516a-9516e of each of the channel-specific convolution kernels (or feature detectors) 9516a-9516e (plus bias 9516f) are accumulated channel-by-channel 9518 as the final output / feature map element 9520 of the local receptive field 9512. In this example, the additive biases provided by the two range channels 9504 and 9508 are intermediate outputs 9516b and 9516d, respectively. The additive bias provided by the scale channel 9510 is intermediate output 9516e.

[0795] This additive bias guides the feature map compilation process by placing greater emphasis on those features in the image channels that are considered more important and reliable for base calling (i.e., the pixel intensities of the center cluster pixel and its neighbors). During training, the weights of the convolutional kernels are updated by backpropagation of the gradients calculated from comparisons with the ground truth base calls to produce stronger activations for the center cluster pixel and its neighbors.

[0796] For example, considering that a pixel in a set of neighboring pixels covered by the local receptive field 9512 contains the cluster center, the distance channels 9504 and 9508 reflect the proximity of the pixel to the cluster center. Therefore, when the intensity intermediate outputs 9516a and 9516c are combined with the distance channel additive biases 9516b and 9516d at the channel-by-channel accumulation 9518, the result is a positive biased convolutional representation 9520 of the pixel.

[0797] In contrast, if the pixels covered by the local receptive field 9512 are not close to the cluster center, the distance channels 9504 and 9508 reflect their separation from the cluster center. Thus, when the intensity intermediate outputs 9516a and 9516c are combined with the distance channel additive biases 9516b and 9516d at the channel-by-channel accumulation 9518, the result is a negatively biased convolutional representation 9520 of the pixel.

[0798] Similarly, the scaled channel additive bias 9516e derived from the scaled channel 9510 may positively or negatively bias the convolution representation 9520 of the pixel.

[0799] For clarity, Figure 95b A single convolution filter i 9514 is shown applied to input data 9500 for a single sequencing cycle. Those skilled in the art will appreciate that this discussion can be extended to multiple convolution filters (e.g., a filter bank having k filters, where k can be 8, 16, 32, 64, 128, 256, etc.), multiple convolution layers (e.g., multiple spatial and temporal convolution layers), and multiple sequencing cycles (e.g., t, t+1, t-1).

[0800] In other embodiments, these distance channels and scale channels are not encoded separately, but are applied directly to the image channels to generate modulated pixel multiplications because the distance channels and scale channels have the same dimensions as the image channels. In another embodiment, the weights of the convolution kernel are determined based on the distance channels and the image channels so as to detect the most important features in the image channels during the element-wise multiplication. In other embodiments, these distance channels and scale channels are not fed to the first layer, but are provided as auxiliary inputs to downstream layers and / or networks (e.g., to a fully connected network or classification layer). In yet another embodiment, these distance and scale channels are fed to the first layer and re-fed to the downstream layers and / or networks (e.g., via residual connections).

[0801] The above discussion is for 2D input data with k input channels. Those skilled in the art will appreciate the extensions to 3D input. In short, the volumetric input is a 4D tensor with dimensions k × l × w × h, where l is an additional dimension—the length. Each individual kernel is a 4D tensor, and by sweeping through the 4D tensor, a 3D tensor is obtained (the channel dimension is collapsed because it is not swept).

[0802] In other implementations, when the input data 9500 is at an upsampled sub-pixel resolution, these range channels and scale channels are encoded separately on a sub-pixel basis, and additive biasing occurs at the sub-pixel level.

[0803] Base calling using specialized architecture and input data

[0804] We now discuss how to use specialized architectures and input data for neural network-based base calling.

[0805] Single cluster base calling

[0806] Figure 96a 、 Figure 96b and Figure 96c Depicted is a specific implementation for base calling a single target cluster. The specialized architecture processes input data from three sequencing cycles (i.e., the current (time t) sequencing cycle for which base calling is to be performed, the previous (time t-1) sequencing cycle, and the subsequent (time t+1) sequencing cycle) and generates a base call for the single target cluster at the current (time t) sequencing cycle.

[0807] Figure 96a and Figure 96b The spatial convolutional layers are shown. Figure 96c Temporal convolutional layers are shown, along with some other non-convolutional layers. Figure 96a and Figure 96b In the figure, the vertical dashed lines define the spatial convolutional layer according to the feature map, and the horizontal dashed lines define the three convolutional pipelines corresponding to the three sequencing cycles.

[0808] For each sequencing cycle, the input data consists of a tensor of dimension n×n×m (e.g., Figure 95a The input tensor 9500 in is , where n represents the width and height of the square tensor, and m represents the number of input channels, so that the dimensions of the input data for the three loops are n×n×m×t.

[0809] Here, each tensor for each cycle contains the center of a single target cluster in the center pixel of its image channel. Each tensor also shows the intensity emission of the single target cluster, some neighboring clusters, and its surrounding background captured in each of the image channels at a particular sequencing cycle. Figure 96a , two exemplary image channels are depicted, namely a red image channel and a green image channel.

[0810] Each tensor for each loop also includes distance channels that complement the corresponding image channels (e.g., red and green distance channels). These distance channels identify the center-to-center distance of each pixel in the corresponding image channel to a single target cluster. Each tensor for each loop also includes a scaling channel that scales the intensity values ​​in each of the image channels pixel by pixel.

[0811] This specialized architecture has five spatial convolutional layers and two temporal convolutional layers. Each spatial convolutional layer uses a Segregated convolution is applied using a set of k convolutional filters, where j represents the width and height of the square filter, and Denotes its depth. Each temporal convolutional layer applies combined convolution using a set of k convolutional filters of dimension j×j×α, where j denotes the width and height of the square filter and α denotes its depth.

[0812] The specialized architecture has a pre-classification layer (e.g., a flattening layer and a dense layer) and an output layer (e.g., a softmax classification layer). The pre-classification layer prepares the input for the output layer. The output layer generates a base call for a single target cluster at the current (time t) sequencing cycle.

[0813] The decreasing spatial dimension

[0814] Figure 96a 、 Figure 96b and Figure 96c Also shown are the resulting feature maps (convolutional representations or intermediate convolutional representations or convolutional features or activation maps) produced by the convolutional filters. Starting from the tensor for each cycle, the spatial dimensions of these resulting feature maps decrease by a constant step size from one convolutional layer to the next, a concept referred to herein as "continuously decreasing spatial dimensions." Figure 96a 、 Figure 96b and Figure 96c In , an exemplary constant step size of 2 is used for the decreasing spatial dimension.

[0815] The decreasing spatial dimensionality is represented by the following formula: "Current feature map spatial dimension = Previous feature map spatial dimension - Convolution filter spatial dimension + 1." This decreasing spatial dimensionality causes the convolution filter to gradually narrow its focus on the center cluster pixel and its neighboring pixels, generating a feature map with features that capture the local dependencies between the center cluster pixel and its neighboring pixels. This, in turn, facilitates accurate base calling of clusters whose centers are contained within the center cluster pixel.

[0816] The isolated convolutions of these five spatial convolutional layers prevent information mixing between the three sequencing cycles and maintain three separate convolutional pipelines.

[0817] The combined convolution of these two temporal convolutional layers mixes the information between the three sequencing cycles. The first temporal convolutional layer convolves the subsequent and current spatial convolutional representations generated by the final spatial convolutional layer for the subsequent and current sequencing cycles, respectively. This produces a first temporal output. The first temporal convolutional layer also convolves the current and previous spatial convolutional representations generated by the final spatial convolutional layer for the current and previous sequencing cycles, respectively. This produces a second temporal output. The second temporal convolutional layer convolves the first and second temporal outputs and produces a final temporal output.

[0818] In some implementations, the final temporal output is fed to a flattening layer to produce a flattened output. The flattened output is then fed to a dense layer to produce a dense output. The dense output is processed by the output layer to produce a base call for a single target cluster at the current (time t) sequencing cycle.

[0819] In some implementations, the output layer generates the likelihood (classification scores) of bases that are A, C, T, and G incorporated into a single target cluster at the current sequencing cycle, and classifies the bases as A, C, T, or G based on these probabilities (e.g., selecting the base with the greatest likelihood, such as Figure 96a In such implementations, these likelihoods are the exponentially normalized scores produced by the softmax classification layer and sum to 1.

[0820] In some implementations, the output layer exports an output pair for a single target cluster. The output pair identifies the class labels of the bases incorporated into the single target cluster at the current sequencing cycle as A, C, T, or G, and performs base calling for the single target cluster based on these class labels. In one implementation, class label 1,0 identifies an A base; class label 0,1 identifies a C base; class label 1,1 identifies a T base; and class label 0,0 identifies a G base. In another implementation, class label 1,1 identifies an A base; class label 0,1 identifies a C base; class label 0.5,0.5 identifies a T base; and class label 0,0 identifies a G base. In yet another implementation, class label 1,0 identifies an A base; class label 0,1 identifies a C base; class label 0.5,0.5 identifies a T base; and class label 0,0 identifies a G base. In yet another specific implementation, class label 1,2 identifies an A base; class label 0,1 identifies a C base; class label 1,1 identifies a T base; and class label 0,0 identifies a G base.

[0821] In some implementations, the output layer derives a class label for a single target cluster, where the class label identifies whether the base incorporated into the single target cluster at the current sequencing cycle is A, C, T, or G, and base calling is performed for the single target cluster based on the class label. In one implementation, a class label of 0.33 identifies an A base; a class label of 0.66 identifies a C base; a class label of 1 identifies a T base; and a class label of 0 identifies a G base. In another implementation, a class label of 0.50 identifies an A base; a class label of 0.75 identifies a C base; a class label of 1 identifies a T base; and a class label of 0.25 identifies a G base.

[0822] In some implementations, the output layer derives a single output value, compares the single output value to a range of class values ​​corresponding to the bases A, C, T, and G, assigns the single output value to a particular class value range based on the comparison, and base calls the single target cluster based on the assignment. In one implementation, the single output value is derived using a sigmoid function, and the single output value is in the range of 0 to 1. In another implementation, a class value range of 0 to 0.25 represents an A base, a class value range of 0.25 to 0.50 represents a C base, a class value range of 0.50 to 0.75 represents a T base, and a class value range of 0.75 to 1 represents a G base.

[0823] Those skilled in the art will appreciate that in other specific implementations, the specialized architecture can process input data for fewer or more sequencing cycles, and may include fewer or more spatial convolutional layers and temporal convolutional layers. In addition, the dimensions of the input data, the tensors for each cycle in the input data, the convolution filters, the resulting feature maps, and the outputs may be different. In addition, the number of convolutional filters in the convolutional layer may be different. It can use various padding and stride configurations. It can use different classification functions (e.g., sigmoid or regression) and may or may not include a fully connected layer. It can use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or hole convolution, transposed convolution, depth-separable convolution, point-by-point convolution, 1×1 convolution, grouped convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It can use one or more loss functions such as logistic regression / logarithmic loss function, multi-class cross entropy / softmax loss function, binary cross entropy loss function, mean squared error loss function, L1 loss function, L2 loss function, smooth L1 loss function and Huber loss function. It can use any parallelism, efficiency and compression scheme, such as TFRecords, compressed encoding (e.g., PNG), sharpening, parallel detection of map transformations, batching, prefetching, model parallelism, data parallelism and synchronous / asynchronous SGD. It can include upsampling layers, downsampling layers, recurrent connections, gate and gate memory units (such as LSTM or GRU), residual blocks, residual connections, high-speed connections, skip connections, peephole connections, activation functions (e.g., nonlinear transformation functions such as rectified linear units (ReLU), leaky ReLU, exponential lining units (ELU), sigmoid and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout layers, pooling layers (e.g., max or average pooling), global average pooling layers and attention mechanisms.

[0824] Having described single cluster base calling, we now discuss multiple cluster base calling.

[0825] Multiple cluster base calling

[0826] Depending on the size of the input data and the cluster density on the flow cell, anywhere between one hundred thousand and three hundred thousand clusters are base called simultaneously on each input by the neural network-based base caller 1514. Extending this to data parallelism and / or model parallelism strategies implemented on a parallel processor, using batches or mini-batches of size ten results in base calling between one hundred and three million clusters on a per-batch or per-mini-batch basis.

[0827] Depending on the sequencing configuration (e.g., cluster density, number of blocks on the flow cell), a block includes 20,000 to 300,000 clusters. In another embodiment, Illumina's NovaSeq sequencer has up to 4 million clusters per block. Thus, a sequencing image of a block (block image) can depict intensity emissions from 20,000 to 300,000 clusters and their surrounding background. Thus, in one embodiment, using input data that includes the entire block image results in simultaneous base calling of 300,000 clusters per input. In another embodiment, using image patches of size 15×15 pixels in the input data results in simultaneous base calling of less than one hundred clusters per input. Those skilled in the art will appreciate that these numbers can vary depending on the sequencing configuration, parallelization strategy, details of the architecture (e.g., based on optimal architecture hyperparameters), and available compute.

[0828] Figure 97 A specific implementation of simultaneous base calling for multiple target clusters is shown. The input data has three tensors for the three sequencing cycles described above. For each tensor of each cycle (e.g., Figure 95a The input tensor 9500 in FIG. 1 depicts the intensity emission captured in each of the image channels at a specific sequencing cycle for multiple target clusters to be base called and their surrounding background. In other embodiments, some additional neighboring clusters that are not base called are also included for context.

[0829] In a multiple cluster base calling implementation, each tensor for each cycle includes distance channels that complement the corresponding image channels (e.g., a red distance channel and a green distance channel). These distance channels identify the center-to-center distance of each pixel in the corresponding image channel to the nearest cluster in the multiple target clusters.

[0830] In a multiple cluster shape-based base calling implementation, each tensor for each cycle includes distance channels that complement corresponding image channels (e.g., a red distance channel and a green distance channel). These distance channels identify the center-to-center distance of each cluster pixel in the corresponding image channel to the cluster to which it belongs or is attributed in the multiple target clusters.

[0831] Each tensor for each cycle also includes a scaling channel that scales the intensity values ​​in each image channel pixel by pixel.

[0832] exist Figure 97 In the example, the spatial dimension of each tensor for each loop is greater than Figure 96a The spatial dimensions shown. That is, in Figure 96a In the specific implementation of single target cluster base calling in , the spatial dimension of each tensor for each...

Claims

1. A computer-implemented method of determining metadata about clusters on a block of a flow cell, the computer-implemented method comprising: accessing a set of tile images captured during a sequencing run and the initial center coordinates of the clusters determined by the base callers; For each image set, obtaining from a base caller a base call that classifies an origin sub-pixel and a predetermined neighborhood of neighboring sub-pixels containing an initial center coordinate as one of four bases, thereby generating a sequence of base calls for each of the origin sub-pixels and each of the predetermined neighborhoods of the neighboring sub-pixels, wherein the neighboring sub-pixels are continuously adjacent to a corresponding one of the origin sub-pixels; generating a cluster map that identifies the clusters as disjoint regions of contiguous sub-pixels that are contiguously adjacent to at least some of the corresponding origin sub-pixels and that share substantially matching base call sequences of one of four bases with at least some of the corresponding origin sub-pixels; as well as The cluster map is stored in a memory for use in determining the shape and size of the clusters.

2. The computer-implemented method of claim 1 , further comprising training a classifier based on the cluster map, the classifier being a neural network-based template generator for processing input image data to generate an attenuation map, a ternary map, or a binary map representing one or more attributes of each of a plurality of clusters represented in the input image data for use in base calling by a neural network-based base caller.

3. The computer-implemented method of claim 2, further comprising: determining the hyperlocalized center coordinates of the analyte by calculating the centroid of a non-intersecting region of the cluster map as the average of the coordinates of corresponding adjacent sub-pixels forming the non-intersecting region; and The hyperlocalized center coordinates of the clusters are stored in memory to be used as ground truth for training the classifier.

4. The computer-implemented method of claim 1 , wherein: The cluster map identifies boundaries between adjacent sub-pixels where base called sequences do not substantially match.

5. The computer-implemented method of claim 1 , wherein: Generating the cluster map includes identifying those sub-pixels that do not belong to any of the disjoint regions as background.

6. The computer-implemented method of claim 1 , wherein: The cluster map identifies a cluster boundary portion between two adjacent sub-pixels whose base call sequences do not substantially match.

7. A non-transitory computer readable memory comprising instructions that, when executed by at least one processor, cause a system to perform actions for determining metadata about clusters on a block of a flow cell, the actions comprising: accessing a set of tile images captured during a sequencing run and the initial center coordinates of the clusters determined by the base callers; For each image set, obtaining from a base caller a base call that classifies an origin sub-pixel and a predetermined neighborhood of neighboring sub-pixels containing an initial center coordinate as one of four bases, thereby generating a sequence of base calls for each of the origin sub-pixels and each of the predetermined neighborhoods of the neighboring sub-pixels, wherein the neighboring sub-pixels are continuously adjacent to a corresponding one of the origin sub-pixels; generating a cluster map that identifies the clusters as disjoint regions of contiguous sub-pixels that are contiguously adjacent to at least some of the corresponding origin sub-pixels and that share substantially matching base call sequences of one of four bases with at least some of the corresponding origin sub-pixels; as well as The cluster map is stored in a memory for use in determining the shape and size of the clusters.

8. The non-transitory computer readable memory of claim 7, further comprising instructions that, when executed by the at least one processor, cause the system to perform an action of training a classifier based on the cluster map, the classifier being a neural network-based template generator for processing input image data to generate an attenuation map, a ternary map, or a binary map representing one or more attributes of each of a plurality of clusters represented in the input image data for base calling by a neural network-based base caller.

9. The non-transitory computer readable memory of claim 7, further comprising instructions that, when executed by the at least one processor, cause the system to: determining the hyperlocalized center coordinates of the analyte by calculating the centroid of a non-intersecting region of the cluster map as the average of the coordinates of corresponding adjacent sub-pixels forming the non-intersecting region; and The hyperlocalized center coordinates of the clusters are stored in memory to be used as ground truth for training the classifier.

10. The non-transitory computer readable memory according to claim 7, wherein: The cluster map identifies boundaries between adjacent sub-pixels where base called sequences do not substantially match.

11. The non-transitory computer readable memory of claim 7, further comprising instructions which, when executed by the at least one processor, cause the system to perform actions including generating the cluster map by identifying those sub-pixels that do not belong to any of the non-intersecting regions as background.

12. The non-transitory computer readable memory according to claim 7, wherein: The cluster map identifies a cluster boundary portion between two adjacent sub-pixels whose base call sequences do not substantially match.

13. The non-transitory computer readable memory of claim 7, further comprising instructions that, when executed by the at least one processor, cause the system to perform actions comprising generating the cluster map based on: identifying an origin sub-pixel at an initial center coordinate of a cluster determined by a base caller; and A breadth-first search is performed for substantially matching base-called sequences by starting with the origin sub-pixel and continuing with consecutive adjacent non-origin sub-pixels.

14. A computer-implemented system comprising: at least one processor; as well as A non-transitory computer readable memory comprising instructions that, when executed by at least one processor, cause the computer implemented system to perform actions for determining metadata about clusters on a block of a flow cell, the actions comprising: accessing a set of tile images captured during a sequencing run and the initial center coordinates of the clusters determined by the base callers; For each image set, obtaining from a base caller a base call that classifies an origin sub-pixel and a predetermined neighborhood of neighboring sub-pixels containing an initial center coordinate as one of four bases, thereby generating a sequence of base calls for each of the origin sub-pixels and each of the predetermined neighborhoods of the neighboring sub-pixels, wherein the neighboring sub-pixels are continuously adjacent to a corresponding one of the origin sub-pixels; generating a cluster map that identifies the clusters as disjoint regions of contiguous sub-pixels that are contiguously adjacent to at least some of the corresponding origin sub-pixels and that share substantially matching base call sequences of one of four bases with at least some of the corresponding origin sub-pixels; as well as The cluster map is stored in a memory for use in determining the shape and size of the clusters.

15. The computer-implemented system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the computer-implemented system to perform an action of training a classifier based on the cluster map, the classifier being a neural network-based template generator for processing input image data to generate an attenuation map, a ternary map, or a binary map representing one or more attributes of each of a plurality of clusters represented in the input image data for use in base calling by the neural network-based base caller.

16. The computer-implemented system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the computer-implemented system to perform actions comprising: determining the hyperlocalized center coordinates of the analyte by calculating the centroid of a non-intersecting region of the cluster map as the average of the coordinates of corresponding adjacent sub-pixels forming the non-intersecting region; and The hyperlocalized center coordinates of the clusters are stored in memory to be used as ground truth for training the classifier.

17. The computer-implemented system of claim 14, wherein: The cluster map identifies boundaries between adjacent sub-pixels where base called sequences do not substantially match.

18. The computer-implemented system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the computer-implemented system to perform actions including generating the cluster map by identifying those sub-pixels that do not belong to any of the non-intersecting regions as background.

19. The computer-implemented system of claim 14, wherein: The cluster map identifies a cluster boundary portion between two adjacent sub-pixels whose base call sequences do not substantially match.

20. The computer-implemented system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the computer-implemented system to perform actions comprising generating the cluster map based on: identifying an origin sub-pixel at an initial center coordinate of a cluster determined by a base caller; and A breadth-first search is performed for substantially matching base-called sequences by starting with the origin sub-pixel and continuing with consecutive adjacent non-origin sub-pixels.

Images