Artificial intelligence-based sequencing

Abstract

To provide artificial intelligence-based sequencing methods and the like that address needs, including increasing the level of throughput in high-throughput nucleic acid sequencing technologies.SOLUTION: An artificial intelligence-based sequencing method comprises: processing a first input through a first neural network and producing a first output, where the first input is captured by a sequencing system for a sequencing run; processing the first output through a post-processor; and producing metadata about analytes and their surrounding background; and processing a second input through a second neural network and producing a second output, where the second input includes third image data derived by modifying second image data based on the metadata, where the second image data is derived from the images of the analytes and their surrounding background, and where the second output identifies base calls for one or more of the analytes at one or more sequencing cycles of the sequencing run.SELECTED DRAWING: Figure 141

Description

[Technical Field]

[0001] (Priority application) This application claims priority to or the benefit of the following applications:

[0002] U.S. Provisional Patent Application No. 62 / 821,602, entitled "Training Data Generation for Artificial Intelligence-Based Sequencing," filed March 21, 2019 (Attorney Docket No. ILLM1008-1 / IP-1693-PRV);

[0003] U.S. Provisional Patent Application No. 62 / 821,618, entitled "Artificial Intelligence-Based Generation of Sequencing Metadata," filed March 21, 2019 (Attorney Docket No. ILLM1008-3 / IP-1741-PRV);

[0004] U.S. Provisional Patent Application No. 62 / 821,681, entitled "Artificial Intelligence-Based Base Calling," filed March 21, 2019 (Attorney Docket No. ILLM1008-4 / IP-1744-PRV);

[0005] U.S. Provisional Patent Application No. 62 / 821,724, entitled "Artificial Intelligence-Based Quality Scoring," filed March 21, 2019 (Attorney Docket No. ILLM1008-7 / IP-1747-PRV);

[0006] U.S. Provisional Patent Application No. 62 / 821,766, entitled "Artificial Intelligence-Based Sequencing," filed March 21, 2019 (Attorney Docket No. ILLM1008-9 / IP-1752-PRV);

[0007] Dutch Patent Application No. 2023310 (Attorney Docket No. ILLM1008-11 / IP-1693-NL), entitled "Training Data Generation for Artificial Intelligence-Based Sequencing," filed on June 14, 2019;

[0008] Dutch Patent Application No. 2023311 (Attorney Docket No. ILLM1008-12 / IP-1741-NL), entitled "Artificial Intelligence-Based Generation of Sequencing Metadata," filed on June 14, 2019;

[0009] Dutch Patent Application No. 2023312 entitled "Artificial Intelligence-Based Base Calling" filed on June 14, 2019 (Attorney Docket No. ILLM1008-13 / IP-1744-NL);

[0010] Dutch Patent Application No. 2023314 entitled "Artificial Intelligence-Based Quality Scoring", filed on June 14, 2019 (Attorney Docket No. ILLM1008-14 / IP-1747-NL), and

[0011] Dutch Patent Application No. 2023316 (Attorney Docket No. ILLM1008-15 / IP-1752-NL), entitled "Artificial Intelligence-Based Sequencing," filed on June 14, 2019.

[0012] U.S. Patent Application No. 16 / 825,987, entitled "Training Data Generation for Artificial Intelligence-Based Sequencing," filed March 20, 2020 (Attorney Docket No. ILLM1008-16 / IP-1693-US);

[0013] U.S. Patent Application No. 16 / 825,991, entitled "Training Data Generation for Artificial Intelligence-Based Sequencing," filed March 20, 2020 (Attorney Docket No. ILLM1008-17 / IP-1741-US);

[0014] U.S. Patent Application No. 16 / 826,126, entitled "Artificial Intelligence-Based Base Calling," filed March 20, 2020 (Attorney Docket No. ILLM1008-18 / IP-1744-US);

[0015] U.S. Patent Application No. 16 / 826,134, entitled "Artificial Intelligence-Based Quality Scoring," filed March 20, 2020 (Attorney Docket No. ILLM1008-19 / IP-1747-US);

[0016] U.S. Patent Application No. 16 / 826,168, entitled "Artificial Intelligence-Based Sequencing," filed March 21, 2020 (Attorney Docket No. ILLM1008-20 / IP-1752-PRV);

[0017] PCT Patent Application No. PCT___________ entitled "Training Data Generation for Artificial Intelligence-Based Sequencing," filed concurrently herewith and subsequently published as PCT International Publication No. WO____________ (Attorney Docket No. ILLM1008-21 / IP-1693-PCT);

[0018] PCT Patent Application No. PCT__________ (Attorney Docket No. ILLM1008-22 / IP-1741-PCT), entitled "Artificial Intelligence Based Generation of Sequencing Metadata," filed concurrently herewith and subsequently published as PCT International Publication No. WO____________;

[0019] PCT Patent Application No. PCT___________ (Attorney Docket No. ILLM1008-23 / IP-1744-PCT), entitled "Artificial Intelligence-Based Base Calling," filed concurrently herewith and subsequently published as PCT International Publication No. WO____________; and

[0020] PCT Patent Application No. PCT__________ (Attorney Docket No. ILLM1008-24 / IP-1747-PCT), entitled "Artificial Intelligence-Based Quality Scoring," filed concurrently herewith and subsequently published as PCT International Publication No. WO____________.

[0021] The priority application is incorporated herein by reference for all purposes as if fully set forth herein. (built-in)

[0022] The following are incorporated by reference for all purposes as if fully set forth herein:

[0023] U.S. Provisional Patent Application No. 62 / 849,091, entitled "Systems and Devices for Characterization and Performance Analysis of Pixel-Based Sequencing," filed May 16, 2019 (Attorney Docket No. ILLM1011-1 / IP-1750-PRV);

[0024] U.S. Provisional Patent Application No. 62 / 849,132, entitled "Base Calling Using Convolutions," filed May 16, 2019 (Attorney Docket No. ILLM1011-2 / IP-1750-PR2);

[0025] U.S. Provisional Patent Application No. 62 / 849,133, entitled "Base Calling Using Compact Convolutions," filed May 16, 2019 (Attorney Docket No. ILLM1011-3 / IP-1750-PR3);

[0026] U.S. Provisional Patent Application No. 62 / 979,384, entitled "Artificial Intelligence-Based Base Calling of Index Sequences," filed February 20, 2020 (Attorney Docket No. ILLM1015-1 / IP-1857-PRV);

[0027] U.S. Provisional Patent Application No. 62 / 979,414, entitled "Artificial Intelligence-Based Many-To-Many Base Calling," filed February 20, 2020 (Attorney Docket No. ILLM1016-1 / IP-1858-PRV);

[0028] U.S. Provisional Patent Application No. 62 / 979,385, entitled "Knowledge Distillation-Based Compression of Artificial Intelligence-Based Base Caller," filed February 20, 2020 (Attorney Docket No. ILLM1017-1 / IP-1859-PRV);

[0029] U.S. Provisional Patent Application No. 62 / 979,412, entitled "Multi-Cycle Cluster Based Real Time Analysis System," filed February 20, 2020 (Attorney Docket No. ILLM1020-1 / IP-1866-PRV);

[0030] U.S. Provisional Patent Application No. 62 / 979,411, entitled "Data Compression for Artificial Intelligence-Based Base Calling," filed February 20, 2020 (Attorney Docket No. ILLM1029-1 / IP-1964-PRV);

[0031] U.S. Provisional Patent Application No. 62 / 979,399, entitled "Squeezing Layer for Artificial Intelligence-Based Base Calling," filed February 20, 2020 (Attorney Docket No. ILLM1030-1 / IP-1982-PRV);

[0032] Liu P, Hemani A, Paul K, Weis C, Jung M, Wehn ​​N. 3D-Stacked Many-Core Architecture for Biological Sequence Analysis Problems.Int J Parallel Prog.2017, 45(6):1420-60,

[0033] Z. Wu, K. Hammad, R. Mittmann, S. Magierowski, E. Ghafar-Zadeh, and X. Zhong, "FPGA-Based DNA Basecalling Hardware Acceleration", in Proc. IEEE 61st Int. Midwest Symp. Circuits Syst., Aug. 2018, pp. 1098-1101,

[0034] Z. Wu, K. Hammad, E. Ghafar-Zadeh, and S. Magierowski, "FPGA-Accelerated 3rd Generation DNA Sequencing", in IEEE Transactions on Biomedical Circuits and Systems, Volume 14, Issue 1, Feb. 2020, pp. 65-74,

[0035] Prabhakar et al.、「Plasticine:A Reconfigurable Architecture for Parallel Patterns」、ISCA’17,June 24-28,2017,Toronto,ON,Canada、

[0036] M.Lin,Q.Chen,and S.Yan、「Network in Network」、in Proc.of ICLR,2014、

[0037] L.Sifre、「Rigid-motion Scattering for Image Classification,Ph.D.thesis,2014、

[0038] L.Sifre and S.Mallat、「Rotation,Scaling and Deformation Invariant Scattering for Texture Discrimination」、in Proc.of CVPR,2013、

[0039] F.Chollet、「Xception:Deep Learning with Depthwise Separable Convolutions」、in Proc.of CVPR,2017、

[0040] X.Zhang,X.Zhou,M.Lin,and J.Sun、「ShuffleNet:An Extremely Efficient Convolutional Neural Network for Mobile Devices」、in arXiv:1707.01083,2017、

[0041] K.He,X.Zhang,S.Ren,and J.Sun、「Deep Residual Learning for Image Recognition」、in Proc.of CVPR,2016、

[0042] S.Xie,R.Girshick,P.Dollar,Z.Tu,and K.He、「Aggregated Residual Transformation For Deep NeuroNetworks」、Proc.of CVPR,2017、

[0043] A.G.Howard,M.Zhu,B.Chen,D.Kalenichenko,W.Wang,T.Weyand,M.Andreetto,and H.Adam、「Mobilenets:Efficient Convolutional Neural Networks for Mobile Vision Applications」、in arXiv:1704.04861,2017、

[0044] M.Sandler,A.Howard,M.Zhu,A.Zhmoginov,and L.Chen、「MobileNetV2:Inverted Residuals and Linear Bottlenecks」、in arXiv:1801.04381v3,2018、

[0045] Z.Qin,Z.Zhang,X.Chen and Y.Peng、「FD-MobileNet:Improved MobileNet with a Fast Downsampling Strategy」、in arXiv:1802.03750,2018、

[0046] Liang-Chieh Chen,George Papandreou,Florian Schroff,and Hartwig Adam.Rethinking atrous convolution for semantic image segmentation.CoRR、abs / 1706.05588,2017、

[0047] J.Huang,V.Rathod,C.Sun,M.Zhu,A.Korattikara,A.Fathi,I.Fischer,Z.Wojna,Y.Song,S.Guadarrama,et al.Speed / accuracy trade-offs for modern convolutional object detectors.arXiv preprint arXiv:1611.10012,2016、

[0048] S.Dieleman,H.Zen,K.Simonyan,O.Vinyals,A.Graves,N.Kalchbrenner,A.Senior,and K.Kavukcuoglu、「WAVENET:A GENERATIVE MODEL FOR RAW AUDIO」、arXiv:1609.03499,2016、

[0049] S.O.Arik,M.Chrzanowski,A.Coates,G.Diamos,A.Gibiansky,Y.Kang,X.Li,J.Miller,A.Ng,J.Raiman,S.Sengupta and M.Shoeybi、「DEEP VOICE:REAL-TIME NEURAL TEXT-TO-SPEECH」、arXiv:1702.07825,2017、

[0050] F.Yu and V.Koltun、「MULTI-SCALE CONTEXT AGGREGATION BY DILATED CONVOLUTIONS」、arXiv:1511.07122,2016、

[0051] K.He,X.Zhang,S.Ren,and J.Sun、「DEEP RESIDUAL LEARNING FOR IMAGE RECOGNITION」、arXiv:1512.03385,2015、

[0052] R.K.Srivastava,K.Greff,and J.Schmidhuber、「HIGHWAY NETWORKS」、arXiv:1505.00387,2015、

[0053] G.Huang,Z.Liu,L.van der Maaten and K.Q.Weinberger、「DENTILY CONNECTED CONVOLUTIONAL NETWORKS」、arXiv:1608.06993,2017、

[0054] C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,and A.Rabinovich、「GOING DEEPER WITH CONVOLUTIONS」、arXiv:1409.4842,2014、

[0055] S.Ioffe and C.Szegedy、「BATCH NORMALIZATION:ACCELERATING DEEP NETWORK TRAINING BY REDUCING INTERNAL COVARIATE SHIFT」、arXiv:1502.03167,2015、

[0056] J.M.Wolterink,T.Leiner,M.A.Viergever,and 1.Isgum、「DILATED CONVOLUTIONAL NEURAL NETWORKS FOR CARDIOVASCULAR MR SEGMENTATION IN CONGENITAL HEART DISEASE」、arXiv:1704.03669,2017、

[0057] L.C.Piqueras、「AUTOREGRESSIVE MODEL BASED ON A DEEP CONVOLUTIONAL NEURAL NETWORK FOR AUDIO GENERATION」、Tampere University of Technology,2016、

[0058] J. Wu, "Introduction to Convolutional Neural Networks", Nanjing University, 2017,

[0059] "Illumina CMOS Chip and One-Channel SBS Chemistry", Illumina,Inc.2018,2 pages,

[0060] "skikit-image / peak.py at master", GitHub, 5 pages, [Retrieved 2018-11-16]. Internet<URL:https: / / github.com / scikit-image / scikit-image / blob / master / skimage / feature / peak.py#L25> Search from,

[0061] "3.3.9.11. Watershed and random walker for segmentation," Scipy lecture notes, 2 pages, [Retrieved 2018-11-13]. Internet<URL:http: / / scipy-lectures.org / packages / scikit-image / auto_examples / plot_segmentations.html> Search from,

[0062] Mordvintsev, Alexander and Revision, Abid K., "Image Segmentation with Watershed Algorithm", Revision 43532856, 2013, 6 pages [Retrieved 2018-11-13]. Internet <URL:https: / / opencv-python-tutroals.readthedocs.io / en / latest / py_tutorials / py_imgproc / py_watershed / py_watershed.html> Search from,

[0063] Mzur, "Watershed.py", 25 October 2017, 3 pages, [Retrieved 2018-11-13]. Internet<URL:https: / / github.com / mzur / watershed / blob / master / Watershed.py> Search from,

[0064] Thakur, Pratibha, et.al. "A Survey of Image Segmentation Techniques", International Journal of Research in Computer Applications and Robotics, Vol. 2, Issue. 4, April 2014, Pg.: 158-165,

[0065] Long, Jonathan, et.al., “Fully Convolutional Networks for Semantic Segmentation”,: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 39, Issue 4,1 April 2017, 10 pages,

[0066] Ronneberger, Olaf, et.al., “U-net: Convolutional networks for biomedical image segmentation”. In International Conference on Medical image computing and computer-assisted intervention, 18 May 2015, 8 pages,

[0067] Xie, W., et al., "Microscopy cell counting and detection with fully convolutional regression networks", Computer methods in biomechanics and biomedical engineering: Imaging & Visualization, 6(3), pp. 283 - 292, 2018.

[0068] Xie, Yuanpu, et al., "Beyond classification: structured regression for robust cell detection using convolutional neural network", International Conference on Medical Image Computing and Computer - Assisted Intervention. October 2015, 12 pages.

[0069] Snuverink, I.A.F., "Deep Learning for Pixelwise Classification of Hyperspectral Images", Master of Science Thesis, Delft University of Technology, 23 November 2017, 19 pages.

[0070] Shevchenko, A., "Keras weighted categorical_crossentropy", 1 page. [Searched on January 15, 2019]. Retrieved from the Internet <URL:https: / / gist.github.com / skeeet / cad06d584548fb45eece1d4e28cfa98b>.

[0071] van den Assem, DCF, "Predicting periodic And chaotic signals using Wavenets", Master of Science Thesis, Delft University Of Technology, 18 August 2017, Pages 3-38,

[0072] IJ Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “CONVOLUTIONAL NETWORKS”, Deep Learning, MIT Press, 2016, and

[0073] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, and G. Wang, “RECENT ADVANCES IN CONVOLUTIONAL NEURAL NETWORKS”, arXiv:1512.07108, 2017.

[0074] FIELD OF THE INVENTION The technology of this disclosure relates to artificial intelligence computers and digital data processing systems and corresponding data processing methods and products for emulating intelligence (i.e., knowledge-based systems, inference systems, and knowledge acquisition systems), including systems for inferring using uncertainty (e.g., fuzzy logic systems), adaptive systems, machine learning systems, and artificial neural networks. Specifically, the disclosed technology relates to using deep neural networks, such as deep convolutional neural networks, to analyze data. [Background technology]

[0075] The subject matter described in this section should not be assumed to be prior art merely as a result of its mention in this section. Likewise, it should not be assumed that the problems mentioned in this section, or problems related to the subject matter provided as background, have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which themselves may also correspond to the implementation of the claimed technology.

[0076] Deep neural networks are a class of artificial neural networks that use multiple nonlinear and complex transformation layers to continuously model high-level functions. Deep neural networks provide feedback via backpropagation, which propagates the difference between observed and predicted outputs to adjust parameters. Deep neural networks have evolved with the availability of large training datasets, the power of parallel and distributed computing, and advanced training algorithms. Deep neural networks have driven major advances in many domains, such as computer vision, speech recognition, and natural language processing.

[0077] Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are components of deep neural networks. Convolutional neural networks, particularly those with structures including convolutional layers, nonlinear layers, and pooling layers, have been successful in image recognition. Recurrent neural networks are designed to exploit the continuous information in input data with periodic connections between their constituent units, such as perceptrons, long short-term memory units, and gated recurrent units. In addition, many other emerging deep neural networks have been proposed for limited situations, such as deep spatiotemporal neural networks, multidimensional recurrent neural networks, and convolutional autoencoders.

[0078] The goal of training a deep neural network is to optimize the weight parameters in each layer, gradually combining simpler features into more complex ones so that a better hierarchical representation can be learned from the data. A single cycle of the optimization process consists of the following: First, given a training dataset, a forward pass sequentially computes the outputs in each layer and propagates a functional signal forward through the network. At the final output layer, an objective loss function measures the error between the predicted output and a given label. To minimize the training error, a backward pass backpropagates the error signal using a chain rule and computes gradients for all weights throughout the neural network. Finally, stochastic parameters are updated using an optimization algorithm based on stochastic gradient descent. While batch gradient descent updates parameters for the entire dataset, stochastic gradient descent provides a stochastic approximation by performing updates for each small set of data examples. Several optimization algorithms are derived from stochastic gradient descent. For example, the Adagrad and Adam training algorithms perform stochastic gradient descent while adaptively modifying the learning rate based on the update frequency of each parameter and the momentum of the gradient, respectively.

[0079] Another core element in deep neural network training is regularization, which refers to a strategy intended to avoid overfitting and thus achieve good generalization performance. For example, weight decay adds a penalty term to the objective loss function so that weight parameters converge to smaller absolute values. Dropout randomly removes hidden units from a neural network during training, which can be viewed as a collection of possible subnetworks. To improve dropout's capabilities, a new activation function, maxout, and a variant of dropout for recurrent neural networks called rnnDrop have been proposed. Furthermore, batch normalization provides a new regularization method via scalar feature normalization for each activation within a mini-batch, learning its mean and variance as parameters.

[0080] Given the multidimensional and high-dimensional nature of sequence data, deep neural networks hold considerable promise for bioinformatics research due to their broad applicability and enhanced predictive capabilities. Convolutional neural networks have been employed to solve sequence-based problems in genomics, such as motif discovery, pathogenic variant identification, and gene expression inference. Convolutional neural networks use a weight-sharing strategy that is particularly useful for studying DNA, as it can capture short sequence motifs that reproduce local patterns in DNA that are predicted to have significant biological functions. A notable feature of convolutional neural networks is the use of convolutional filters.

[0081] Unlike traditional classification approaches based on carefully designed, manually crafted features, convolutional filters perform adaptive feature learning, similar to the process of mapping raw input data to information representations of knowledge. In this sense, convolutional filters function as a series of motif scanners, as a set of such filters can recognize relevant patterns in the input and update themselves during the training procedure. Recurrent neural networks can capture long-range dependencies in continuous data of various lengths, such as protein or DNA sequences.

[0082] Therefore, an opportunity arises to use a coherent deep learning-based framework for template generation and base calling.

[0083] In the era of high-throughput technologies, accumulating the highest yield of interpretable data at the lowest cost per effort remains a significant challenge. Cluster-based methods of nucleic acid sequencing, such as those that utilize bridge amplification for cluster formation, have made valuable contributions to the goal of increasing nucleic acid sequencing throughput. These cluster-based methods rely on sequencing dense populations of nucleic acids immobilized on a solid support and typically involve the use of image analysis software to suppress the optical signal generated during the simultaneous sequencing of multiple clusters located at distinct locations on the solid support.

[0084] However, such solid-phase nucleic acid cluster-based sequencing technology faces considerable obstacles that limit the amount of throughput that can be achieved.For example, in cluster-based sequencing methods, determining the nucleic acid sequences of two or more clusters that are too close to each other physically to be spatially resolved, or that actually physically overlap on a solid support, can pose obstacles.For example, current image analysis software may require valuable time and computational resources to determine which of two overlapping clusters an optical signal originates from.As a result, various detection platforms inevitably compromise the quantity and / or quality of nucleic acid sequence information that can be obtained.

[0085] High-density nucleic acid aggregate-based genomics methods extend to other areas of genome analysis as well. For example, nucleic acid cluster-based genomics can be used in sequencing applications, diagnostics and screening, gene expression analysis, epigenetic analysis, genetic analysis of polymorphisms, etc. Each of these nucleic acid cluster-based genomics techniques is limited by the inability to resolve data generated from closely spaced or spatially overlapping nucleic acid clusters.

[0086] Clearly, there is a need to improve the quality and quantity of nucleic acid sequence data that can be obtained rapidly and cost-effectively for a variety of applications, including genomics (e.g., for genomic characterization of any and all animal, plant, microbial, or other biological species or populations), pharmacogenomics, transcriptomics, diagnostics, prognosis, biomedical risk assessment, clinical and research genetics, personalized medicine, drug efficacy and drug interaction assessment, veterinary medicine, agricultural, evolutionary, and biological research, aquatic culture, forestry, marine research, ecological and environmental management, and other purposes.

[0087] The disclosed technology provides neural network-based methods and systems that address these and similar needs, including increasing the level of throughput in high-throughput nucleic acid sequencing technologies, and offers other related advantages.

[0088] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. Color drawings may also be available in PAIR (patent application information retrieval) via the Supplemental Content tab.

[0089] In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosed technology. In the following description, various embodiments of the disclosed technology are described with reference to the following drawings: [Brief explanation of the drawings]

[0090] [Figure 1] 1 illustrates one embodiment of a processing pipeline for determining cluster metadata using subpixel base calls. [Figure 2] 1 shows an embodiment of a flow cell containing clusters within its tiles. [Figure 3]An example of an Illumina GA-IIx flow cell with eight lanes is shown. [Figure 4] An image set of four-channel chemical sequencing images is depicted, i.e., the image set has four sequencing images captured using four different wavelength bands (image / imaging channels) in the pixel domain. [Figure 5] 1 is an embodiment of dividing a sequencing image into sub-pixels (or sub-pixel regions). [Figure 6] During subpixel base calling, preliminary center coordinates of clusters identified by the base caller are shown. [Figure 7] 1 illustrates one embodiment of merging sub-pixel base calls generated over multiple sequencing cycles to generate a so-called "cluster map" that contains cluster metadata. [Figure 8a] 1 shows an example of a cluster map generated by merging subpixel base calls. [Figure 8b] 1 illustrates one embodiment of subpixel base calling. [Figure 9] 10 illustrates another example of a cluster map that identifies cluster metadata. [Figure 10] This shows how the center of mass (COM) of discontinuous regions in a cluster map is calculated. [Figure 11] 10 illustrates one embodiment of a calculation of a weighted attenuation coefficient based on the Euclidean distance from a subpixel of a discontinuous region to a COM of the discontinuous region. [Figure 12] 10 illustrates one implementation of an exemplary ground truth attenuation map derived from an exemplary cluster map generated by sub-pixel base calling. [Figure 13] 1 illustrates one embodiment of deriving a ternary map from a cluster map. [Figure 14] 1 illustrates one embodiment of deriving a binary map from a cluster map. [Figure 15]FIG. 1 is a block diagram illustrating one embodiment for generating training data used to train a neural network-based template generator and a neural network-based base caller. [Figure 16] 1 illustrates the properties of the disclosed training examples used to train the neural network-based template generator and the neural network-based base caller. [Figure 17] 1 illustrates one embodiment of processing input image data through the disclosed neural network-based template generator to generate an output value for each unit in the array. In one embodiment, the array is an attenuation map. In another embodiment, the array is a ternary map. In yet another embodiment, the array is a binary map. [Figure 18] FIG. 1 illustrates one embodiment of a post-processing technique applied to attenuation maps, ternary maps, or binary maps generated by a neural network-based template generator to derive cluster metadata including cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries. [Figure 19] 1 illustrates one embodiment for extracting cluster intensities in the pixel domain. [Figure 20] 1 illustrates one embodiment for extracting cluster intensities in the sub-pixel domain. [Figure 21a] Three different implementations of a neural network-based template generator are presented. [Figure 21b] 15 illustrates one embodiment of input image data provided as input to neural network-based template generator 1512. The input image data includes a series of image sets having sequencing images generated during a specific number of initial sequencing cycles of a sequencing operation. [Figure 22] FIG. 21B illustrates one embodiment of extracting patches from the series of image sets of FIG. 21b to generate a series of "downsized" image sets that form the input image data. [Figure 23]21b shows one embodiment of upsampling the series of image sets of FIG. 21b to generate a series of "upsampled" image sets that form the input image data. [Figure 24] 24 illustrates one embodiment of extracting patches from the series of upsampled image sets of FIG. 23 to generate a series of "upsampled and downsized" image sets that form the input image data. [Figure 25] 1 illustrates one embodiment of an overall exemplary process for generating ground truth data for training a neural network-based template generator. [Figure 26] 1 illustrates one embodiment of a regression model. [Figure 27] 1 illustrates one embodiment of generating a ground truth attenuation map from a cluster map, which is used as ground truth data for training a regression model. [Figure 28] 1 is an implementation of training a regression model using a backpropagation-based gradient update technique. [Figure 29] 1 is an embodiment of template generation by a regression model during estimation. [Figure 30] 10 illustrates one embodiment of post-processing the attenuation map to identify cluster metadata. [Figure 31] 1 illustrates one implementation of a watershed segmentation technique that identifies non-overlapping groups of adjacent clusters / inter-cluster sub-pixels that characterize clusters. [Figure 32] 1 is a table showing an exemplary U-Net structure for a regression model. [Figure 33] We present a different approach to extract cluster intensities using cluster shape information identified in a template image. [Figure 34] 1 illustrates different approaches to base calling using the output of a regression model. [Figure 35]Figure 1 shows the difference in base calling performance when the RTA base caller uses ground truth center of mass (COM) positions as cluster centers, as opposed to using non-COM positions as cluster centers. The results show that using COM positions improves base calling. [Figure 36] On the left, we show an example attenuation map that generated the regression model. On the right, Figure 36 also shows an example ground truth attenuation map that the regression model approximates during training. [Figure 37] 1 illustrates one embodiment of a peak locator that identifies cluster centers in an attenuation map by detecting peaks. [Figure 38] Peaks detected by the peak locator in the attenuation map generated by the regression model are compared with peaks in the corresponding ground truth attenuation map. [Figure 39] Demonstrate the performance of your regression model using precision and recall statistics. [Figure 40] Compare the performance of the RTA base caller and the regression model for a library concentration of 20 pM (normal operation). [Figure 41] Compare the performance of the RTA base caller and the regression model for a library concentration of 30 pM (high density operation). [Figure 42] The number of non-overlapping, well-aligned read pairs, i.e., the number of read pairs in which neither read is aligned within a reasonable distance as detected by the regression model, is compared with those detected by the RTA base caller. [Figure 43] On the right, the first attenuation map generated by the regression model is shown. On the left, Figure 43 shows the second attenuation map generated by the regression model. [Figure 44] For 40 pM library concentration (high density operation), the performance of the RTA base caller and the regression model are compared. [Figure 45]The first attenuation map generated by the regression model is shown on the left. On the right, Figure 45 shows the results of thresholding, peak location processing, and watershed segmentation techniques applied to the first attenuation map. [Figure 46] 1 illustrates one embodiment of a binary classification model. [Figure 47] 1 is an implementation of training a binary classification model using a backpropagation-based gradient update technique with softmax scoring. [Figure 48] 1 is another implementation of training a binary classification model using a backpropagation-based gradient update technique with sigmoid scores. [Figure 49] 1 illustrates another embodiment of input image data provided to a binary classification model and corresponding class labels used to train the binary classification model. [Figure 50] 1 is an embodiment of template generation with a binary classification model during inference. [Figure 51] 1 illustrates one embodiment in which the binary map is subjected to peak detection to identify cluster centers. [Figure 52a] An example binary map generated by a binary classification model is shown on the left. Figure 52a also shows an example ground truth binary map on the right, to which the binary classification model is proximate during training. [Figure 52b] Use accuracy statistics to indicate the performance of a binary classification model. [Figure 53] 1 is a table illustrating an example structure of a binary classification model. [Figure 54] 1 illustrates one embodiment of a three-way classification model. [Figure 55] 1 is an implementation of training a ternary classification model using a backpropagation-based gradient update technique. [Figure 56] 10 illustrates another embodiment of input image data provided to a three-way classification model and the corresponding class labels used to train the three-way classification model. [Figure 57] 1 is a table illustrating an exemplary structure of a three-way classification model. [Figure 58]1 is an embodiment of template generation with a three-way classification model under inference. [Figure 59] 1 shows a ternary map generated by a ternary classification model. [Figure 60] The unit array generated by the three-way classification model 5400 is shown along with the output values ​​for each unit. [Figure 61] 1 illustrates one embodiment in which the ternary map is subjected to post-processing to identify cluster centers, cluster backgrounds, and cluster interiors. [Figure 62a] 1 shows an exemplary prediction of a three-way classification model. [Figure 62b] 10 shows another exemplary prediction of a three-way classification model. [Figure 62c] 10 shows yet another exemplary prediction of a three-way classification model. [Figure 63] FIG. 62b shows one embodiment for deriving cluster centers and cluster shapes from the output of the ternary classification model of FIG. 62a. [Figure 64] Compare the base calling performance of a binary classification model, a regression model, and an RTA base caller. [Figure 65] The performance of the ternary classification model is compared with that of the RTA-based caller under three conditions, five sequencing metrics, and two operating densities. [Figure 66] The performance of the regression model is compared to that of the RTA-based caller under three conditions, five sequencing metrics, and two operating densities considered in Figure 65. [Figure 67] We focus on the penultimate layer of the neural network-based template generator. [Figure 68] Visualize what the penultimate layer of a neural network-based template generator has learned as a result of backpropagation-based gradient update training. The illustrated embodiment visualizes 24 out of 32 trained convolution filters in the penultimate layer shown in Figure 67. [Figure 69] Cluster center predictions of the binary classification model (in blue) are overlaid on the RTA base caller (in pink). [Figure 70] Cluster center predictions produced by RTA-based color (in pink) are overlaid on a visualization of the trained convolutional filters in the penultimate layer of a binary classification model. [Figure 71] 1 illustrates one embodiment of training data used to train a neural network-based template generator. [Figure 72] 10 is an implementation of using beads for image registration based on cluster center prediction of a neural network-based template generator. [Figure 73] 10 illustrates one embodiment of cluster statistics for clusters identified by a neural network-based template generator. [Figure 74] We show how the ability of the neural network-based template generator to distinguish between adjacent clusters improves as the number of initial sequencing cycles for which the input image data is used increases from 5 to 7. [Figure 75] Figure 1 shows the difference in base calling performance when the RTA base caller uses ground truth center of mass (COM) positions as cluster centers as opposed to when non-COM positions are used as cluster centers. [Figure 76] We show the performance of the neural network-based template generator on additional detected clusters. [Figure 77] 1 shows different datasets used to train the neural network-based template generator. [Figure 78] 1 shows the processing steps used by the RTA base caller for base calling, according to one embodiment. [Figure 79] 1 shows one embodiment of base calling using the disclosed neural network-based base caller. [Figure 80]10 is one embodiment of transforming location / position information of cluster centers identified from the output of a neural network-based template generator from the sub-pixel domain to the pixel domain. [Figure 81] One implementation uses cycle-specific and image channel-specific transformations to derive so-called "transformed cluster centers" from reference cluster centers. [Figure 82] 1 shows image patches that are part of the input data fed into a neural network-based base caller. [Figure 83] 10 illustrates one embodiment of determining distance values ​​for a distance channel when a single target cluster is being base called by a neural network-based base caller. [Figure 84] 10 illustrates one implementation of per-pixel encoding of distance values ​​calculated between a pixel and a target cluster. [Figure 85a] 10 illustrates one embodiment for determining distance values ​​for a distance channel when multiple target clusters are being simultaneously base called by a neural network-based base caller. [Figure 85b] For each target cluster, we show a number of nearest pixels determined based on the cluster center distance closest to the pixel center. [Figure 86] 10 illustrates one implementation of encoding, for each pixel, the minimum distance value calculated between the pixel and the closest one of the clusters. [Figure 87] One embodiment is shown that uses classifications / attributes / classifications between pixel clusters, referred to herein as "cluster shape data." [Figure 88] 10 illustrates one embodiment of calculating distance values ​​using cluster shape data. [Figure 89] 10 illustrates one implementation of per-pixel encoding of the distance value calculated between the pixel and its assigned cluster. [Figure 90]1 shows one embodiment of a dedicated architecture for a neural network-based base caller used to separate the processing of data in different sequencing cycles. [Figure 91] 1 illustrates one implementation of separated convolution. [Figure 92a] 1 illustrates one embodiment of combinatorial folding. [Figure 92b] 10 illustrates another embodiment of combinatorial folding. [Figure 93] 1 illustrates one implementation of convolutional layers of a neural network-based base caller, where each convolutional layer has a bank of convolutional filters. [Figure 94] Two configurations of the scaling channel that complement the image channel are shown. [Figure 95a] 1 shows one embodiment of input data for a single sequencing cycle that produces red and green images. [Figure 95b] 10 illustrates one implementation of a distance channel that provides an additive bias that is incorporated into feature maps generated from an image channel. [Figure 96a] 1 shows one embodiment of base calling a single target cluster. [Figure 96b] 1 shows one embodiment of base calling a single target cluster. [Figure 96c] 1 shows one embodiment of base calling a single target cluster. [Figure 97] 1 shows one embodiment of simultaneously base calling multiple target clusters. [Figure 98] 1 illustrates one embodiment in which multiple target clusters are base called simultaneously in multiple successive sequencing cycles, thereby simultaneously generating base called sequences for each of the multiple target clusters. [Figure 99] FIG. 1 shows a dimensionality diagram for a single cluster base calling embodiment. [Figure 100] Dimensionality diagram for multiple clusters, single sequencing cycle base calling embodiment. [Figure 101]Dimensionality diagram for multiple clusters, multiple sequencing cycle base calling embodiment. [Figure 102a] 1 illustrates an exemplary array input configuration for multi-cycle input data. [Figure 102b] 1 illustrates an exemplary stack input configuration for multi-cycle input data. [Figure 103a] 10 illustrates one embodiment of rearranging the pixels of an image patch to center the center of a target cluster that is base called at the center pixel. [Figure 103b] 10 shows another example reconstructed / shifted image patch where (i) the center of the central pixel coincides with the center of the target cluster, and (ii) the non-central pixels are equidistant from the center of the target cluster. [Figure 104] 1 shows one embodiment of using a standard convolutional neural network and reconstructed inputs to base call a single target cluster in the current sequencing cycle. [Figure 105] 1 shows one embodiment of using a standard convolutional neural network and aligned inputs to base call multiple target clusters in the current sequencing cycle. [Figure 106] 1 shows one implementation of base calling multiple target clusters over multiple sequencing cycles using a standard convolutional neural network and aligned inputs. [Figure 107] 1 illustrates one embodiment of training a neural network-based base caller. [Figure 108a] 1 illustrates one embodiment of a hybrid neural network used as a neural network-based base caller. [Figure 108b] 1 illustrates one implementation of 3D convolution used by the iterative module of a hybrid neural network to generate a current hidden state representation. [Figure 109]FIG. 1 illustrates one embodiment of processing input data per cycle for a single sequencing cycle among a series of t sequencing cycles to be base called through a cascade of convolutional layers of a convolution module. [Figure 110] 1 illustrates one embodiment of blending input data per cycle of a single sequencing cycle with its corresponding convolutional representation produced by a cascade of convolutional layers of a convolution module. [Figure 111] 1 illustrates one embodiment of arranging flattened mixed representations of successive sequencing cycles as a stack. [Figure 112a] The stack in Figure 111 shows one embodiment in which 3D convolution is applied iteratively in the forward and backward directions to generate base calls for each of the clusters in each of a series of t sequencing cycles. [Figure 112b] 1 illustrates one embodiment in which a 3D input volume x(t) containing a group of flattened mixed representations is processed through input, activation, forget, and output gates of a Long Short-Term Memory (LSTM) network that applies 3D convolutions. The LSTM network is part of the recurrent module of a hybrid neural network. [Figure 113] 1 illustrates one embodiment of balancing trinucleotides (trimers) in the training data used to train a neural network-based base caller. [Figure 114] Compare the base calling accuracy of the RTA base caller against a neural network-based base caller. [Figure 115] The generalization across tiles of the RTA base caller is compared to that of a neural network-based base caller on the same tiles. [Figure 116] The cross-tile generalization of the RTA base caller is compared with that of a neural network-based base caller on the same tile and on different tiles. [Figure 117]We also compare the cross-tile generalization of the RTA base caller with that of a neural network-based base caller on different tiles. [Figure 118] We show how different sizes of image patches fed as input to a neural network-based base caller affect base calling accuracy. [Figure 119] We show the lane-to-lane generalization of a neural network-based base caller on training data from A. baumannii and E. coli. [Figure 120] We show the lane-to-lane generalization of a neural network-based base caller on training data from A. baumannii and E. coli. [Figure 121] We show the lane-to-lane generalization of a neural network-based base caller on training data from A. baumannii and E. coli. [Figure 122] We show the lane-to-lane generalization of a neural network-based base caller on training data from A. baumannii and E. coli. [Figure 123] 119, 120, 121, and 122 show the error profiles for the lane-to-lane generalization described above. [Figure 124] The source of the error detected by the error profile in FIG. 123 is attributed to the low cluster intensity in the green channel. [Figure 125] The error profiles of the RTA base caller and the neural network-based base caller are compared for two sequencing runs (read 1 and read 2). [Figure 126a] We demonstrate generalization between the behavior of neural network-based basecallers on four different instruments. [Figure 126b] We demonstrate the generalization between the behaviors of a neural network-based base caller across four different behaviors performed on the same instrument. [Figure 127]1 shows genomic statistics of the training data used to train the neural network-based base caller. [Figure 128] 1 shows the genomic context of the training data used to train the neural network-based base caller. [Figure 129] Figure 1 shows the base calling accuracy of a neural network-based base caller when calling long reads (e.g., 2 × 250). [Figure 130] 1 illustrates one embodiment of how a neural network-based base caller addresses a central cluster pixel(s) and its neighboring pixels across an image patch. [Figure 131] 1 illustrates various hardware components and configurations used to train and operate a neural network-based base caller, according to one embodiment. In other embodiments, different hardware components and configurations are used. [Figure 132] 1 illustrates various sequencing tasks that can be performed using a neural network-based base caller. [Figure 133] A scatter plot visualized by t-Distributed Stochastic Neighbor Embedding (t-SNE) showing base calling results from a neural network-based base caller. [Figure 134] 1 illustrates one embodiment of selecting base call confidence probabilities produced by a neural network-based base caller for quality scoring. [Figure 135] 1 illustrates one implementation of neural network-based quality scoring. [Figure 136a] 1 shows one embodiment of the correspondence between quality scores and base call confidence predictions produced by a neural network-based base caller. [Figure 136b]1 shows one embodiment of the correspondence between quality scores and base call confidence predictions produced by a neural network-based base caller. [Figure 137] 1 illustrates one embodiment of inferring quality scores from base call confidence predictions made by a neural network-based base caller during inference. [Figure 138] We present one embodiment in which a neural network-based quality scorer is trained to process input data derived from sequencing images and directly generate a quality index. [Figure 139] 10 illustrates one embodiment of generating a quality index directly as an output of a neural network-based quality scorer during inference. [Figure 140] 1 illustrates one embodiment that uses a lossless transformation to generate transformed data that can be provided as input to a neural network-based template generator, a neural network-based base caller, and a neural network-based quality scorer. [Figure 141] 1 illustrates one embodiment of integrating a neural network-based template generator with a neural network-based base caller using region weighting factors. [Figure 142] We present another embodiment that integrates a neural network-based template generator with a neural network-based base caller using upsampling and background masking. [Figure 143] An example of a region weighting factor 14300 for contributions from only a single cluster per pixel is shown. [Figure 144] 10 shows an example of region weighting factors for contributions from multiple clusters per pixel. [Figure 145] 1 shows an example of using interpolation for upsampling and background masking. [Figure 146] 10 shows an example of using subpixel count weighting for upsampling and background masking. [Figure 147A]1 illustrates one embodiment of a sequencing system, the sequencing system including a configurable processor. [Figure 147B] 1 illustrates one embodiment of a sequencing system, the sequencing system including a configurable processor. [Figure 147C] FIG. 1 is a simplified block diagram of a system for analysis of sensor data from a sequencing system, such as base call sensor output. [Figure 148A] FIG. 1 is a simplified diagram illustrating aspects of a base call operation, including the functionality of a runtime program executed by a host processor. [Figure 148B] FIG. 147D is a simplified diagram of a configuration of a configurable processor such as that shown in FIG. 147C. [Figure 149] 147B is a computer system that can be used by the sequencing system of FIG. 147A to implement the techniques disclosed herein. [Figure 150] 1 illustrates different implementations of data pre-processing, which may include data normalization and data augmentation. [Figure 151] Figure 150 shows that the data normalization technique (DeepRTA(norm)) and the data augmentation technique (DeepRTA(augment)) reduce the base calling error rate when a neural network-based base caller is trained on bacterial data and tested on human data, where the bacterial data and human data share the same assay (e.g., both contain intron data). [Figure 152] Figure 151 shows that the data normalization technique (DeepRTA(norm)) and the data augmentation technique (DeepRTA(augment)) reduce the base calling error rate when a neural network-based base caller is trained on non-exonic data (e.g., intronic data) and tested on exonic data. DETAILED DESCRIPTION OF THE INVENTION

[0091] The following description is presented to enable any person skilled in the art to make and use the disclosed technology, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Thus, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. (introduction)

[0092] Base calling from digital images is massively parallel and computationally intensive, which presents a number of technical challenges to identify before implementing our novel technique.

[0093] The signal from the image set being evaluated becomes increasingly weaker as the base classification progresses periodically, especially over longer and longer strands of bases. As the base classification extends over the length of the strand, the signal-to-noise ratio decreases and confidence decreases. An updated estimate of confidence is projected as the estimated confidence of the base classification change.

[0094] A digital image is captured from the amplified clusters of sample strands. The sample is amplified by replicating the strands using various physical structures and chemicals. During sequencing by synthesis, tags are chemically bound in cycles and stimulated to glow. A digital sensor collects photons from the tags, which are read out as pixels to generate an image.

[0095] Interpreting digital images to classify bases requires resolving positional uncertainty, which is hindered by limited image resolution. At resolutions higher than those collected during base calling, it is clear that imaged clusters have irregular shapes and uncertain center locations. Cluster positions are not mechanically controlled, and therefore cluster centers are not aligned with pixel centers. The pixel center can be an integer coordinate assigned to the pixel. In other embodiments, it can be the upper left corner of the pixel. In yet other embodiments, it can be the centroid or center of mass of the pixel. Amplification does not produce uniform cluster shapes. Therefore, the distribution of cluster signals in the digital image is a statistical distribution rather than a regular pattern. We determine this positional uncertainty.

[0096] One of the signal classes does not produce a detectable signal and can be classified into a specific location based on the "dark" signal. Therefore, a template is required for classification during the dark cycle. Template generation resolves the initial location uncertainty using multiple imaging cycles to avoid missing dark signals.

[0097] Tradeoffs in image sensor size, magnification, and stepper design lead to pixel sizes that are too large to manipulate cluster centers to coincide with sensor pixel centers. This disclosure uses pixel in two senses. A physical sensor pixel is an area of ​​a photosensor that reports detected photons. A logical pixel, simply called a pixel, is the data corresponding to at least one physical pixel and readout from the sensor pixel. Pixels can be subdivided or "upsampled" into subpixels (e.g., 4x4 subpixels). To account for the possibility that all photons hit one side of a physical pixel and not the other, values ​​can be assigned to the subpixels by interpolation, such as bilinear interpolation or area weighting. Interpolation or bilinear interpolation is also applied when a pixel is reframed by applying an affine transformation to the data from the physical pixel.

[0098] Larger physical pixels are more sensitive to weak signals than smaller pixels. Digital sensors improve over time, but the physical limitations of collector surface area are unavoidable. Considering design tradeoffs, legacy systems are designed to collect and analyze image data from a 3x3 patch of sensor pixels, with the center of that cluster located at the center pixel of the patch.

[0099] High-resolution sensors capture only a portion of the imaged medium at a time. The sensor steps over the imaged medium, covering the entire field of view. Thousands of digital images can be collected during one processing cycle.

[0100] The sensor and illumination design are combined to distinguish at least four illumination response values ​​used to classify bases. Using a conventional RGB camera with a Bayer color filter array, the four sensor pixels are combined into a single RGB value. This would reduce the effective sensor resolution by four times. Alternatively, images can be collected at a single location using different illumination wavelengths and / or different filters rotated in position between the imaged medium and the sensor. The number of images required to distinguish between the four basic classes varies between systems. Some systems use one image with four intensity levels for different classes of bases. Other systems use two images with different illumination wavelengths (e.g., red and green) and / or filters with a single base for classifying bases. Systems can also use four images with different illumination wavelengths and / or filters tuned to specific base classes.

[0101] Highly parallel processing of digital images requires the alignment of relatively short strands, on the order of 30-2000 base pairs, to sequences of much longer lengths, potentially millions or even billions in length. Because redundant samples are desirable on the imaged medium, portions of a sequence may be covered by multiple sample reads. Thousands of sample clusters are imaged from a single imaged medium. Large-scale processing of such many clusters increases sequencing capacity while decreasing costs.

[0102] Sequencing capacity is increasing at a pace that replicates Moore's Law. While the first sequencing costs billions of dollars, 2018 services such as Illumina™ provide results for hundreds of dollars. As sequencing becomes mainstream and unit costs fall, less computing power is available for classification, which increases the challenge of near-real-time classification. With these technical challenges in mind, the inventors turn to the disclosed technology.

[0103] The disclosed techniques improve processing both during template generation to resolve positional uncertainty and during base classification of clusters at resolved positions. Applying the disclosed techniques can use cheaper hardware to reduce machine costs. Near real-time analysis can be cost-effective and reduce the delay between image collection and base classification.

[0104] The disclosed technology can use upsampled images generated by interpolating sensor pixels into subpixels to generate templates that resolve positional uncertainty. The resulting subpixels are submitted to a base caller for classification, which treats the subpixels as if they were the center of a cluster. Clusters are identified from groups of adjacent subpixels that repeatedly receive the same base classification. This aspect of the technology can leverage existing base calling techniques to identify cluster shapes and super-search for cluster centers at subpixel resolution.

[0105] Another aspect of the disclosed technology is creating ground truth, pairing images with reliably identified cluster centers and / or cluster shapes. Deep learning systems and other machine learning approaches require substantial training sets. Human-curated data is expensive to compile. The disclosed technology can be used to generate large sets of sensitively classified training data in non-standard modes of operation without the intervention or expense of a human curator. The training data correlates raw images with cluster centers and / or cluster shapes available from existing classifiers in non-standard modes of operation, such as CNN-based deep learning systems. A single training image can be rotated and reflected to generate additional, equally valid examples. Training examples can focus on regions of a predetermined size within the entire image. The context evaluated during base calling determines the size of the example training region, rather than the size of the image or the entire imaged medium.

[0106] The disclosed technology can generate different types of maps that can be used as training data or templates for base classification, correlating cluster centers and / or cluster shapes with digital images. First, subpixels can be classified as cluster centers, thereby localizing the cluster centers within a physical sensor pixel. Second, cluster centers can be calculated as the centroid of the cluster shape. This location can be reported to a selected numerical precision. Third, cluster centers can be reported at surrounding subpixels in an attenuation map, either at subpixel or pixel resolution. The attenuation map reduces the weight given to photons detected within a region as the region's separation from the cluster center increases, attenuating signals from more distant locations. Fourth, binary or ternary classification can be applied to subpixels or pixels within a cluster of neighboring regions. In binary classification, regions are classified as belonging to the cluster center or as background. In ternary classification, a third class type is assigned to regions that include the cluster interior but are not the cluster center. Subpixel classification of cluster center locations can be substituted for real-valued cluster center coordinates within a larger optical pixel.

[0107] Alternative map styles can be initially generated as a ground truth dataset or can be generated using a neural network after training. For example, clusters can be depicted as discontinuous regions of adjacent subpixels with appropriate classifications. The intensities of the mapped clusters from the neural network can be post-processed by a peak detector filter to calculate cluster centers if the centers have not already been determined. Adjacent regions can be assigned to distinct clusters by applying a so-called watershed analysis. When generated by a neural network inference engine, the map can be used as a template for evaluating the sequence of a digital image and classifying bases over a cycle of base calling.

[0108] When bases are sorted into a sequence of digital images, the neural network processes multiple image channels in the current cycle, along with image channels from past and future cycles. In clusters, some strands may operate before or after the main process of synthesis, and out-of-phase tagging is known as prephasing or phasing. Given the empirically observed low rates of prephasing and postphasing, almost all of the noise in the signal resulting from prephasing and postphasing can be handled by a neural network that processes the digital image in only three cycles: the current, past, and future cycles.

[0109] Careful registration between digital image channels within the current cycle to align images within the cycle strongly contributes to accurate base classification. Combinations of wavelengths and mismatched illumination sources, among other sources of error, produce small, correctable differences in measured cluster center locations. A general affine transformation involving translation, rotation, and scaling can be used to accurately align cluster centers across image tiles. The affine transformation can be used to reconstruct image data and resolve offsets to cluster centers.

[0110] Reconstructing image data typically means interpolating the image data by applying an affine transformation. Reconstruction may center the cluster center of interest at the center pixel of the pixel patch. Or it may align the image with a template to overcome jitter and other inconsistencies during image acquisition. Reconstruction involves adjusting the intensity values ​​of all pixels within the pixel patch. Bilinear and bicubic interpolation and weighted region adjustment are alternative strategies.

[0111] In some implementations, the cluster center coordinates may be fed into the neural network as an additional image channel.

[0112] Distance signals can also contribute to base classification. Some types of distance signals reflect the separation of a region from the cluster center. The strongest optical signal is considered to coincide with the cluster center. Optical signals along the cluster perimeter sometimes contain stray signals from nearby clusters. Classification has been observed to be more accurate when the contribution of a signal component is attenuated according to its separation from the cluster center. The distance signals used include a single-cluster distance channel, a multi-cluster distance channel, and a multi-cluster shape-based distance channel. The single-cluster distance channel is applied to a patch with a cluster center within the central pixel. The distance of all regions within the patch is then the distance from the cluster center within the central pixel. Pixels that do not belong to the same cluster as the central pixel can be flagged as background instead of a given calculated distance. The multi-cluster distance channel pre-calculates the distance of each region to the nearest cluster center. This has the potential to connect a region to the wrong cluster center, but this is unlikely. The multi-cluster shape-based distance channel associates regions (subpixels or pixels) through neighboring regions to pixel centers that generate the same base classification. At some computational cost, this avoids the possibility of measuring distance to the wrong pixel. A multi-cluster shape-based approach to multi-cluster and distance signals has the advantage that they are pre-computed and used on multiple clusters in the image.

[0113] To improve the signal-to-noise ratio, shape information can be used by a neural network to separate the signal from the noise. The above discussion has identified several approaches to region classification and the provision of distance channel information. In any approach, regions can be marked as background rather than as part of a cluster to define cluster edges. A neural network can be trained to take advantage of the obtained information about irregular cluster shapes. Distance information and background classification can be combined or used separately. Separation of signals from neighboring clusters becomes increasingly important as cluster density increases.

[0114] One way to increase the scale of parallelism is to increase the cluster density on the imaged medium. Increasing density has the downside of increasing background noise when reading clusters with adjacent neighbors. Using shape data instead of arbitrary (e.g., 3x3 pixel) patches, for example, can help maintain signal separation as cluster density increases.

[0115] Applying one aspect of the disclosed technology, base classification scores can also be leveraged to predict quality. The disclosed technology involves correlating classification scores, either directly or via a predictive model, with traditional Sanger or Phred quality Q-scores. Scores such as Q20, Q30, or Q40 are calculated using the following formula: Q=-10log 10 It is logarithmically related to the base classification error probability by P. Correlation of class scores with Q scores can be performed using multi-output neural networks or multivariate regression analysis. An advantage of real-time calculation of quality scores is that faulty sequencing runs can be terminated early during base classification. Applicant has found that occasional (rare) decisions to terminate runs can be made one-eighth to one-quarter of the way through the analyzed sequence. The decision to terminate can be made after 50 cycles or after 25-75 cycles. Alternatively, in a continuous process running 300-1000 cycles, early termination can result in substantial resource savings.

[0116] A dedicated convolutional neural network (CNN) structure can be used to classify bases across multiple cycles. One specialization involves separating digital image channels in the early layers of processing. Convolutional filter stacks can be structured to separate processing between cycles and prevent crosstalk between digital image sets from different cycles. The motivation for separating processing between cycles is that images taken in different cycles have residual registration errors and are therefore misaligned and have random translational offsets from each other. This occurs due to the finite precision of the sensor's motion stage movement and because images taken in different frequency channels have different optical paths and wavelengths.

[0117] The motivation for using image sets from consecutive cycles is that the pre- and post-phasing contributions to the signal at a particular cycle are quadratic. It follows that structurally separating the underlying convolutions of the digital image set between image acquisition cycles can be useful for convolutional neural networks.

[0118] Convolutional neural network structures can also be specialized to handle information related to clustering. Templates for cluster centers and / or shapes provide additional information that the convolutional neural network combines with the digital image data. The cluster center classification and distance data can be applied repeatedly over cycles.

[0119] A convolutional neural network can be structured to classify multiple clusters in an image field. When multiple clusters are classified, the distance channel for a pixel or subpixel may more compactly contain distance information to either the nearest cluster center to which the pixel or subpixel belongs or to adjacent cluster centers. Alternatively, a large distance vector can be provided for each pixel or subpixel, or at least for each one that contains a cluster center, which gives complete distance information from the cluster center to all other pixels in the context of a given pixel.

[0120] Some combinations of template generation with base calling may use variations in region weighting to replace the distance channel. The discussion here describes how the output of the template generator can be used directly in place of the distance channel.

[0121] We discuss three considerations that affect the direct application of template images to pixel value modification: whether the image set is processed in the pixel or sub-pixel domain, how region weights are calculated in either domain, and applying the template image as a mask to modify interpolated intensity values ​​in the sub-pixel domain.

[0122] Performing base classification in the pixel domain has the advantage of not requiring the increased computation, such as 16x, that results from upsampling. In the pixel domain, even the upper layers of convolution may have sufficient cluster density to justify performing computations that are not collected, instead of adding logic to cancel computations that are not needed. We start with an example in the pixel domain that uses the template image data directly, without a distance channel.

[0123] In some implementations, classification focuses on specific clusters. In these examples, pixels around a cluster may have different modified intensity values ​​depending on which neighboring cluster is the focus of classification. A template image in the subpixel domain may indicate that overlapping pixels contribute intensity values ​​to two different clusters. We refer to an optical pixel as an "overlapping pixel" when two or more neighboring or adjacent clusters both overlap the pixel and both contribute to the intensity reading from the optical pixel. Watershed analysis, named for the separation of rainflow into different watersheds at ridges, may be applied to further separate adjacent clusters. When data is received for cluster-by-cluster classification, the template image may be used to modify the intensity data for overlapping pixels along the perimeter of the cluster. The overlapping pixels may have different modified intensities depending on which cluster is the focus of classification.

[0124] The modified intensity of a pixel may be reduced based on the subpixel contribution within the overlapping pixel to the home cluster (i.e., the cluster to which the pixel belongs or the cluster in which the pixel primarily exhibits intensity emission), as opposed to the away cluster (i.e., the non-home cluster in which the pixel exhibits intensity emission). Assume five subpixels are part of the home cluster and two subpixels are part of the away cluster. Then, seven subpixels contribute intensity to either the home or away cluster. While focusing on the home cluster, in one embodiment, the overlapping pixel has its intensity reduced by 7 / 16 because seven of the 16 subpixels contribute intensity to either the home or away cluster. In another embodiment, the intensity is reduced by 5 / 16 based on the area of ​​the subpixels contributing to the home cluster divided by the total number of subpixels. In a third embodiment, the intensity is reduced by 5 / 7 based on the area of ​​the subpixels contributing to the home cluster divided by the total area of ​​the contributing subpixels. The latter two calculations change when focus changes to the away cluster, producing a fraction with a "2" in the numerator.

[0125] Of course, if the distance channel is considered together with the sub-pixel map of cluster shapes, further reduction in intensity can be applied.

[0126] Once the pixel intensities for the cluster that is the focus of classification have been corrected using the template image, the corrected pixel values ​​are convolved through layers of a neural network-based classifier to produce a corrected image that is used to classify bases in successive sequencing cycles.

[0127] Alternatively, classification in the pixel domain can proceed in parallel for all pixels or all clusters in the aggregate image. Only a single modification of pixel values ​​can be applied in this scenario to ensure reuse of intermediate calculations. Any of the fractions given above can be used to modify pixel intensities, depending on whether a smaller or larger intensity attenuation is desired.

[0128] Once the pixel intensities for the clustered image have been corrected using the template image, the pixels and surrounding context may be convolved through layers of a neural network-based classifier to generate a corrected image. Performing the convolution on the clustered image allows for reuse of intermediate computations among pixels that share context. The corrected image is used to classify bases in successive sequencing cycles.

[0129] This description can be parallelized for the application of area weights within the subpixel domain. The parallelism is that weights can be calculated for each individual subpixel. The weights can be the same for different subpixel portions of the optical pixel, but they do not have to be. Repeating the above scenario of home and away clusters with five and two subpixels of the overlapping pixel, respectively, the intensity assignment to subpixels belonging to the home cluster can be 7 / 16, 5 / 16, or 5 / 7 of the pixel intensity. Again, if the distance channel is considered along with the cluster-shaped subpixel map, further intensity reduction can be applied.

[0130] Once the pixel intensities for the clustered image have been corrected using the template image, the subpixels and surrounding context may be convolved through layers of a neural network-based classifier to generate a corrected image. Performing the convolution on the clustered image allows for reuse of intermediate computations among subpixels that share context. The corrected image is used to classify bases in successive sequencing cycles.

[0131] Another alternative is to apply a template image as a binary mask to the image data to be interpolated in the subpixel domain. The template image can be positioned to require background pixels between clusters or to allow subpixels from different clusters to be adjacent. The template image can be applied as a mask. The mask determines whether the interpolated pixel retains the value assigned by the interpolation or receives a background value (e.g., zero) if it is classified as background in the template image.

[0132] Again, once the pixel intensities for the clustered image are masked using the template image, the subpixels and surrounding context can be convolved through layers of a neural network-based classifier to generate a modified image. Performing the convolution on the clustered image allows for reuse of intermediate computations among subpixels that share context. The modified image is used to classify bases in successive sequencing cycles.

[0133] Features of the disclosed technology can be combined to classify any number of clusters within a shared context, reusing intermediate computations. At optical pixel resolution, in one embodiment, approximately 10 percent of the pixels hold cluster centers for classification. In legacy systems, assuming the observation of irregularly shaped clusters, 3x3 optical pixels were grouped together for analysis as potential signal contributors to the cluster center. Even with a single 3x3 filter away from the upper convolutional layer, the cluster density is such that substantially more optical signal than half of the optical pixels are likely to roll up into the pixel at the cluster center. Only at supersampling resolution does the cluster center density for the upper convolutional layer drop to less than 1 percent.

[0134] In some embodiments, the shared context is substantial. For example, a 15x15 optical pixel context can contribute to accurate base classification. An equivalent 4x upsampled context is 60x60 subpixels. This range of context helps the neural network recognize the effects of uneven lighting and background during imaging.

[0135] The disclosed technique uses small filters in low convolutional layers to combine cluster boundaries in a template input with boundaries detected in a digital image input. The cluster boundaries help the neural network separate signal from background conditions and normalize image processing to the background.

[0136] The disclosed technique substantially reuses intermediate computations. Assume that 20-25 cluster centers appear within a context region of 15x15 optical pixels. The first layer convolutions are then reused 20-25 times in a block-by-block convolution rollup. The reuse factor is reduced layer by layer until the final layer, which is the first time the reuse factor at optical resolution drops below 1x.

[0137] Block-wise rollup training and inference from multiple convolutional layers applies successive rollups to blocks of pixels or subpixels. Around the block, there is an overlap zone where data used during rollup of a first data block overlaps and can be reused with a second block of rollup. Pixel values ​​and intermediate calculations that can be rolled up and reused are located within the block in a central region surrounded by the overlap zone. Within the overlap zone, convolution results that progressively reduce the size of the context field, for example, from 15x15 to 13x13, by applying a 3x3 filter can be written into the same memory block that holds the convolved values, saving memory without compromising the reuse of the underlying calculations within the block. With larger blocks, sharing intermediate calculations within the overlap zone requires fewer resources. With smaller blocks, it may be possible to compute multiple blocks in parallel to share intermediate calculations within the overlap zone.

[0138] Larger filters and dilations reduce the number of convolutional layers, which can speed calculations without impairing classification after lower convolutional layers react to cluster boundaries in the template and / or digital image data.

[0139] The input channel for the template data can be selected to create a template structure that is consistent with the classification of multiple cluster centers within the digital image field. The two alternatives described above, namely, global context reconstruction and distance mapping, do not meet this consistency criterion. The reconstruction places the center of only one cluster at the center of the optical pixel. Providing a center offset for pixels classified as holding the cluster center is better for classifying multiple clusters.

[0140] Distance mapping, if provided, is difficult to perform across the entire context domain unless every pixel has its own distance map across the entire context. Simpler distance maps provide useful consistency for classifying multiple clusters from a digital image input block.

[0141] The neural network may learn from classifications within a template of pixels or subpixels at cluster boundaries, so the distance channel may be replaced by a template that provides a binary or ternary classification along with a cluster center offset channel. When used, the distance map may give the pixel's (or subpixel's) distance from the cluster center to which it belongs. Alternatively, the distance map may give the distance to the nearest cluster center. The distance map may encode the binary classification in a flag value assigned to background pixels, or it may be a separate channel from the pixel classification. Combined with the cluster center offset, the distance map may encode a ternary classification. In some implementations, particularly those that encode pixel classification with one or two bits, it may be desirable, at least during development, to use separate channels for pixel classification and distance.

[0142] The disclosed techniques may include computation reduction to save some computational resources in upper layers. A cluster center offset channel or ternary classification map may be used to identify pixel convolution centers that do not contribute to the final classification of a pixel center. In many hardware / software implementations, performing a lookup during inference and skipping the convolution rollup may be more efficient in upper layer(s) than performing even nine multiplications and eight additions to apply a 3x3 filter. In custom hardware that pipelines computations for parallel execution, all pixels may be classified in the pipeline. Then, to collect results only for pixels that match the cluster centers, the cluster center map may be used after the final convolution, since final classification is desired only for those pixels. Again, in the optical pixel domain, with currently observed cluster densities, rollup calculations for approximately 10 percent of the pixels are collected. In the 4x upsampled domain, less than 1 percent of the subpixel classifications in the upper layers are collected, so on some hardware, more layers may benefit from skipped convolutions. (Neural network-based template generation)

[0143] The first step in template generation is to identify cluster metadata, which identifies the spatial distribution of clusters, including their centers, shapes, sizes, backgrounds, and / or boundaries. (Identifying cluster metadata)

[0144] FIG. 1 shows one embodiment of a processing pipeline for identifying cluster metadata using subpixel base calls.

[0145] Figure 2 shows one embodiment of a flow cell containing clusters within its tiles. The flow cell is divided into lanes. The lanes are further divided into non-overlapping regions called "tiles." During the sequencing procedure, the populations on the tiles and their surrounding background are imaged.

[0146] Figure 3 shows an exemplary Illumina GA-IIx™ flow cell with eight lanes. Figure 3 also shows a magnification of one tile and its cluster and their surrounding background.

[0147] FIG. 4 depicts an image set of a four-channel chemistry sequencing image; that is, the image set has four sequencing images captured using four different wavelength bands (image / imaging channels) in the pixel domain. Each image in the image set covers a tile of a flow cell and shows the intensity emission of clusters on the tile and their surrounding background captured for a particular image channel during a particular one of multiple sequencing cycles of a sequencing operation performed on the flow cell. In one embodiment, each imaging channel corresponds to one of multiple filter wavelength bands. In another embodiment, each imaging channel corresponds to one of multiple imaging events during a sequencing cycle. In yet another embodiment, each imaging channel corresponds to a combination of illumination with a particular laser and imaging through a particular optical filter. The intensity emission of the clusters comprises a signal detected from the analyte that can be used to classify bases associated with the analyte. For example, the intensity emission can be a signal indicative of photons emitted by a tag chemically attached to the analyte during a cycle in which the tag is stimulated and can be detected by one or more digital sensors.

[0148] Figure 5 illustrates one embodiment for dividing a sequencing image into subpixels (or subpixel regions). In another embodiment shown, quarter (0.25) subpixels are used, resulting in each pixel in the sequencing image being divided into 16 subpixels. Assuming the sequencing image shown has a resolution of 20x20 pixels, i.e., 400 pixels, the division produces 6400 subpixels. Each of the subpixels is treated by the base caller as a region center for subpixel base calling. In some embodiments, the base caller does not use neural network-based processing. In other embodiments, the base caller is a neural network-based base caller.

[0149] For a given sequencing cycle and a particular subpixel, the base caller is configured with logic to generate base calls for the particular subpixel for the given sequencing cycle by performing image processing steps to extract the subpixel's intensity data from the corresponding image set for the sequencing cycle. This is done for each subpixel and for each of multiple sequencing cycles. Experiments were also performed using a quarter subpixel division of an 1800 x 1800 pixel resolution tile image from an Illumina MiSeq sequencer. Subpixel base calls were performed for 50 sequencing cycles and 10 tile lanes.

[0150] Figure 6 shows the preliminary center coordinates of clusters identified by the base caller during subpixel base calling. Figure 6 also shows the "origin subpixel" or "center subpixel" that contains the preliminary center coordinates.

[0151] 7 shows an example of merging subpixel base calls generated over multiple sequencing cycles to generate a so-called "cluster map" that includes cluster metadata. In the illustrated embodiment, the subpixel base calls are merged using a first search approach.

[0152] Figure 8a shows an example of a cluster map generated by merging subpixel base calls. Figure 8b shows an example of a subpixel base call. Figure 8b also shows an embodiment in which a cluster map is generated by analyzing the subpixel-by-subpixel base call sequence generated from the subpixel base calls. (Sequencing image)

[0153] Cluster metadata determination involves analyzing image data generated by a sequencing device 102 (e.g., Illumina's iSeq, HiSeqX, HiSeq3000, HiSeq4000, HiSeq2500, NovaSeq 6000, NextSeq, NextSeqDx, MiSeq, and MiSeqDx). The following description outlines how image data is generated and what it describes, according to one embodiment.

[0154] Base calling is the process by which the raw signal of the sequencing instrument 102, i.e., intensity data extracted from the image, is decoded into DNA sequence and quality scores. In one embodiment, the Illumina platform employs cyclic reversible termination (CRT) chemistry for base calling. This process relies on growing emerging DNA strands complementary to template DNA strands bearing modified nucleotides, tracking the emission signal of each newly added nucleotide. The modified nucleotides have a 3' removable block that anchors the fluorophore signal of the nucleotide type.

[0155] Sequencing is performed in repeated cycles, each of which involves three steps: (a) extending the transnucleotide strand by adding modified nucleotides; (b) exciting the fluorophore using one or more lasers in the optical system 104 and imaging through different filters in the optical system 104 to generate a sequencing image 108; and (c) cleaving the fluorophore and removing the 3' block in preparation for the next sequencing cycle. The incorporation and imaging cycle is repeated for a specified number of sequencing cycles, defining the read length of the entire population. Using this approach, each cycle interrogates a new position along the template strand.

[0156] The Illumina platform's power stems from its ability to simultaneously run and sense millions or even billions of clusters undergoing CRT reactions. The sequencing process begins in a flow cell 202, a small glass slide that holds input DNA fragments during the sequencing process. The flow cell 202 is connected to a high-throughput optical system 104, which includes a microscope imager, excitation laser, and fluorescence filters. The flow cell 202 contains multiple chambers, called lanes 204. The lanes 204 are physically separated from one another and may contain differently tagged sequencing libraries, distinguishable without sample cross-contamination. An imager 106 (e.g., a solid-state imager such as a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor) takes snapshots at multiple locations along the lane 204 in a series of non-overlapping regions called tiles 206.

[0157] For example, there are 100 tiles per lane on the Illumina Genome Analyzer II and 64 tiles per lane in the Illumina HiSeq2000. Tile 206 holds hundreds of thousands to millions of clusters. An image generated from a tile with clusters shown as bright spots is shown at 208. Cluster 302 contains approximately 1,000 identical copies of the template molecule, but the clusters vary in size and shape. Clusters are grown from the template molecule by bridge amplification of the input library prior to sequencing operations. The purpose of amplification and cluster growth is to increase the intensity of the emitted signal, since the imaging device 106 cannot reliably sense a single fluorophore. However, because the physical distance between DNA fragments within cluster 302 is small, the imaging device 106 perceives the cluster of fragments as a single spot 302.

[0158] The output of the sequencing operation is a sequencing image 108 showing the intensity emission of clusters on the tile in the pixel domain for a particular combination of lane, tile, sequencing cycle, and fluorophore (208A, 208C, 208T, 208G).

[0159] In one embodiment, the biosensor comprises an array of optical sensors. The optical sensors are configured to sense information from corresponding pixel regions (e.g., reaction sites / wells / nanocells) on the detection surface of the biosensor. An analyte disposed within a pixel region is said to be associated with the pixel region, i.e., the associated analyte. During a sequencing cycle, the optical sensors corresponding to the pixel regions are configured to detect / capture / sense luminescence / photons from the associated analyte and, accordingly, generate a pixel signal for each imaged channel. In one embodiment, each imaging channel corresponds to one of a plurality of filter wavelength bands. In another embodiment, each imaging channel corresponds to one of a plurality of imaging events during a sequencing cycle. In yet another embodiment, each imaging channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter.

[0160] The pixel signals from the optical sensors are communicated (e.g., via a communications port) to a signal processor coupled to the biosensor. For each sequencing cycle and each imaging channel, the signal processor generates an image in which pixels depict / contain / show / represent / characterize the pixel signals obtained from the corresponding optical sensors. In this manner, a pixel in the image corresponds to (i) the optical sensor of the biosensor that generated the pixel signal represented by the pixel, (ii) the analyte of interest whose radiation was detected and converted into a pixel signal by the corresponding optical sensor, and (iii) the pixel area on the detection surface of the biosensor that carries the analyte of interest.

[0161] For example, consider a sequencing operation that uses two different imaging channels: a red channel and a green channel. Then, in each sequencing cycle, the signal processor generates a red image and a green image. In this way, for a series of k sequencing cycles of the sequencing operation, an array having k pairs of red and green images is generated as output.

[0162] Pixels in the red and green images (i.e., different imaging channels) have a one-to-one correspondence within a sequencing cycle. This means that corresponding pixels in a pair of red and green images show intensity data of the same associated analyte in different imaging channels. Similarly, pixels across pairs of red and green images have a one-to-one correspondence between sequencing cycles. This means that corresponding pixels in different pairs of red and green images show intensity data of the same associated analyte for different acquisition events / time steps (sequencing cycles) of the sequencing operation.

[0163] Corresponding pixels in the red and green images (i.e., different imaging channels) can be considered as pixels of an "image per cycle," representing intensity data in a first red channel and a second green channel. An image per cycle, whose pixels depict pixel signals of a subset of the pixel area, i.e., a region (tile) of the sensing surface of the biosensor, is called a "tile per cycle image." Patches extracted from the tile per cycle image are called "image per cycle patches." In one embodiment, patch extraction is performed by an input preparer.

[0164] The image data includes a series of per-cycle image patches generated for a series of k sequencing cycles of the sequencing operation. Pixels in the per-cycle image patch include intensity data for associated analytes, the intensity data being acquired for one or more imaging channels (e.g., red and green channels) by corresponding optical sensors configured to detect emissions from the associated analytes. In one embodiment, when a single target cluster is base called, the per-cycle image patch is centered on a central pixel that includes intensity data for the target-associated analyte and non-central pixels, and the non-central pixels in the per-cycle image patch include intensity data for associated analytes neighboring the target-associated analyte. In one embodiment, the image data is prepared by an input preparer. (subpixel base call)

[0165] The disclosed technology accesses a series of image sets generated during a sequencing operation. The image sets include sequencing images 108. Each successive image set is captured during each sequencing cycle of the sequencing operation. The series of images (or sequencing images) captures the clusters on the tiles of the flow cell and their surrounding background.

[0166] In one embodiment, the sequencing operation utilizes four channel chemistry and each image set has four images. In another embodiment, the sequencing operation utilizes two channel chemistry and each image set has two images. In yet another embodiment, the sequencing operation utilizes one channel chemistry and each image set has two images. In yet another embodiment, each image set has only one image.

[0167] The pixel domain sequencing image 108 is first converted to the sub-pixel domain by a sub-pixel addresser 110 to generate a sequencing image 112 in the sub-pixel domain. In one embodiment, each pixel in the sequencing image 108 is divided into 16 sub-pixels 502. Thus, in one embodiment, the sub-pixels 502 are quarter sub-pixels. In another embodiment, the sub-pixels 502 are half sub-pixels. As a result, each of the sequencing images 112 in the sub-pixel domain has multiple sub-pixels 502.

[0168] The subpixels are then separately provided as inputs to base caller 114, which obtains base calls from base caller 114 that classify each of the subpixels as one of four bases (A, C, T, and G). This generates base call sequence 116 for each of the subpixels over multiple sequencing cycles of the sequencing operation. In one embodiment, subpixels 502 are identified to base caller 114 based on their integer or non-integer coordinates. By tracking the emission signals from subpixels 502 over a set of images generated during multiple sequencing cycles, base caller 114 recovers the underlying DNA sequence of each subpixel. An example of this is shown in FIG. 8b.

[0169] In other embodiments, the disclosed technology classifies each of the subpixels as one of five bases (A, C, T, G, and N) from the base caller 114. In such embodiments, the N base calls represent undetermined base calls, typically due to low levels of extracted intensity.

[0170] Some examples of base callers 114 include non-neural network-based Illumina offerings, such as Real Time Analysis (RTA), the Firecrest program in the Genome Analyzer Analysis Pipeline, the Integrated Primary Analysis and Reporting (IPAR) machine, and the Off-Line Basecaller (OLB). For example, the base caller 114 generates base call sequences by interpolating subpixel intensities, including at least one of nearest neighbor intensity extraction, Gaussian intensity extraction, intensity extraction based on an average 2x2 subpixel area, intensity extraction based on a brightest test of a 2x2 subpixel area, intensity extraction based on an average 3x3 subpixel area, bilinear intensity extraction, bicubic intensity extraction, and / or intensity extraction based on weighted area coverage. These techniques are described in detail in the Appendix entitled "Intensity Extraction Methods."

[0171] In other embodiments, the base caller 114 may be a neural network-based base caller, such as the neural network-based base caller 1514 disclosed herein.

[0172] The base call sequences 116 for each subpixel are then provided as input to a searcher 118, which searches for the underlying base call sequences of successive subpixels. The base call sequences of successive subpixels "substantially match" when a predetermined portion of the base calls match a criterion for each ordinal position (e.g., 41 matches in >= 45 cycles, 4 mismatches in <= 45 cycles, 4 mismatches in <= 50 cycles, or 2 mismatches in <= 34 cycles).

[0173] The searcher 118 then generates a cluster map 802 that identifies clusters, such as 804a-d, of adjacent subpixels that share substantially matching base call sequences. This application uses "discontiguous," "disjoint," and "non-overlapping" interchangeably. The search involves base calling subpixels that comprise part of a cluster and allowing them to link the called subpixels to adjacent subpixels that share substantially matching base call sequences. In some embodiments, the searcher 118 requires that at least some of the discontiguous regions have a predetermined minimum number of subpixels (e.g., more than 4, 6, or 10 subpixels) to be treated as a cluster.

[0174] In some embodiments, the base caller 114 also identifies preliminary center coordinates of the cluster. The subpixel containing the preliminary center coordinate is referred to as the origin subpixel. Some exemplary preliminary center coordinates (604a-c) identified by the base caller 114 and corresponding origin subpixels (606a-c) are shown in FIG. 6. However, as explained below, identifying the origin subpixel (preliminary center coordinate of the cluster) is not required. In some embodiments, the searcher 118 uses a first search, starting with the origin subpixels 606a-c and continuing through successive non-origin subpixels 702a-c, to identify substantially matching base call sequences of the subpixels. This is optional, as explained below. (cluster map)

[0175] FIG. 8a shows an example of a cluster map 802 generated by merging subpixel base calls. The cluster map identifies multiple discontinuous regions (shown in various colors in FIG. 8a). Each discontinuous region includes non-overlapping groups of contiguous subpixels (from whose sequencing image the cluster map is generated via subpixel base calls) that represent respective clusters on the tile. The regions between the discontinuous regions represent the background on the tile. Subpixels within the background regions are referred to as "background subpixels." Subpixels within discontinuous regions are referred to as "cluster subpixels" or "intra-cluster subpixels." In this description, the origin subpixel is the subpixel at which the preliminary center cluster coordinates, determined by the RTA or another base caller, are located.

[0176] The origin subpixel contains the preliminary center cluster coordinate, meaning that the area covered by the origin subpixel contains a coordinate location that coincides with the preliminary center cluster coordinate location. Because the cluster map 802 is an image of logical subpixels, the origin subpixel is a subset of the subpixels in the cluster map.

[0177] The search to identify clusters having substantially matching base call sequences for the subpixels does not need to begin with the identification of an origin subpixel (the preliminary center coordinate of the cluster), because the search can be performed for all subpixels and can begin at any subpixel (e.g., the 0,0 subpixel or any random subpixel). Thus, because each subpixel is evaluated to determine whether it shares a substantially matching base call sequence with another contiguous subpixel, the search does not depend on the origin subpixel, and the search can begin at any subpixel.

[0178] Regardless of whether the origin subpixel is used, certain clusters that do not include the origin subpixel (the initial center coordinate of the cluster) are identified as predicted by base caller 114. Some examples of clusters that are identified by merging subpixel base calls and do not include the origin subpixel are clusters 812a, 812b, 812c, 812d, and 812e in FIG. 8a. Thus, the disclosed techniques identify additional or redundant clusters whose centers may not have been identified by base caller 114. Therefore, the use of base caller 114 to identify the origin subpixel (the initial center coordinate of the cluster) is optional and not required to search for substantially matching base call sequences of consecutive subpixels.

[0179] In one embodiment, the origin subpixels (initial center coordinates of the clusters) identified by base caller 114 are first used to identify a first set of clusters (by identifying substantially matching base call sequences of consecutive subpixels). Subpixels that are not part of the first set of clusters are then used to identify a second set of clusters (by identifying substantially matching base call sequences of consecutive subpixels). This enables the disclosed techniques to identify additional or extra clusters whose centers are not identified by base caller 114. Finally, subpixels that are not part of the first and second sets of clusters are identified as background subpixels.

[0180] Figure 8b shows an example of sub-pixel base calling, where each sequencing cycle has an image set with four different images (i.e., A, C, T, and G images) captured using four different wavelength bands (image / imaging channels) and four different fluorescent dyes (one for each base).

[0181] In this example, a pixel in an image is divided into 16 subpixels. The subpixels are then base called separately by the base caller 114 in each sequencing cycle. To base call a given subpixel in a particular sequencing cycle, the base caller 114 uses the intensity of the given subpixel in each of the four A, C, T, and G images. For example, the intensity of the image region covered by subpixel 1 in each of the four A, C, T, and G images in cycle 1 is used to base call subpixel 1 in cycle 1. For subpixel 1, these image regions include the upper-left 1 / 16 region of the upper-left pixel in each of the four A, C, T, and G images in cycle 1. Similarly, the intensity of the image region covered by subpixel m in each of the four A, C, T, and G images in cycle n is used to base call subpixel m in cycle n. For subpixel m, these image regions include the lower-right 1 / 16 region of the lower-right pixel in each of the four A, C, T, and G images in cycle 1.

[0182] This process generates base call sequences 116 for each subpixel over multiple sequencing cycles. Searcher 118 then evaluates pairs of consecutive subpixels to determine whether they have substantially matching base call sequences. If yes, the pair of subpixels is stored in cluster map 802 as belonging to the same cluster within a discontinuous region. If no, the pair of subpixels is stored in cluster map 802 as not belonging to the same discontinuous region. Thus, cluster map 802 identifies consecutive sets of subpixels for which the base calls for the subpixels substantially match over multiple cycles. Cluster map 802 thus uses information from the multiple clusters to provide multiple clusters, each of which has a high degree of confidence in providing sequence data for a single DNA strand.

[0183] The cluster metadata generator 122 then processes the cluster map 802 to determine cluster metadata, which includes determining the spatial distribution of the clusters, including their centers (810a), shapes, sizes, backgrounds, and / or boundaries (FIG. 9).

[0184] In some implementations, the cluster metadata generator 122 identifies subpixels in the cluster map 802 as background that do not belong to any of the disjoint regions and therefore do not contribute to any clusters. Such subpixels are referred to as background subpixels 806a-c.

[0185] In some embodiments, the cluster map 802 identifies cluster boundaries 808a-c between two consecutive subpixels where the base call sequences do not substantially match.

[0186] The cluster map is stored in memory (e.g., cluster map data store 120) for use as ground truth for training classifiers such as neural network-based template generator 1512 and neural network-based base caller 1514. Cluster metadata may also be stored in memory (e.g., cluster metadata data store 124).

[0187] FIG. 9 shows another example of a cluster map that identifies cluster metadata including the spatial distribution of clusters, along with cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries. (Center of mass (COM))

[0188] Figure 10 shows how the center of mass (COM) of a discontinuous region in a cluster map is calculated, which can be used as the "corrected" or "improved" center of the corresponding cluster in downstream processing.

[0189] In some implementations, for each cluster, a center of mass calculator 1004 determines the cluster's superposition center coordinate 1006 by calculating the center of mass of the discontinuous regions of the cluster map as the average of the coordinates of each contiguous subpixel that forms the discontinuous region. The cluster's superposition center coordinates are then stored for each cluster in memory for use as ground truth for training the classifier.

[0190] In some implementations, the subpixel classifier identifies, for each cluster, a centroid subpixel 1008 within the discontinuous regions 804a-d of the cluster map 802 at the cluster's superposition center coordinate 1006.

[0191] In another alternative embodiment, the cluster map is upsampled using interpolation, and the upsampled cluster map is stored in memory for use as ground truth for training the classifier. (Attenuation coefficient and attenuation map)

[0192] FIG. 11 illustrates one implementation of a calculation of a weighted attenuation coefficient for a subpixel based on the Euclidean distance from the subpixel to the center of mass (COM) of the discontinuous region to which the subpixel belongs. In another illustrated implementation, the weighted attenuation coefficient is assigned its highest value to the subpixel containing the COM and decreases for subpixels further away from the COM. The weighted attenuation coefficient is used to derive a ground truth attenuation map 1204 from the cluster map generated from the subpixel base calls described above. The ground truth attenuation map 1204 includes an array of units, with each unit in the array assigned at least one output value. In some implementations, the units are subpixels, and each subpixel is assigned an output value based on the weighted attenuation coefficient. The ground truth attenuation map 1204 is then used as ground truth for training the disclosed neural network-based template generator 1512. In some implementations, information from the ground truth attenuation map 1204 is also used to prepare the input of the disclosed neural network-based base caller 1514.

[0193] 12 shows one implementation of an exemplary ground truth attenuation map 1204 derived from an exemplary cluster map generated by subpixel base calling as described above. In some implementations, in the upsampled cluster map for each cluster, a value is assigned to each neighboring subpixel in a discontinuous region based on an attenuation coefficient 1102 that is proportional to the neighboring subpixel's distance 1106 from the center-of-mass subpixel 1104 in the discontinuous region to which the neighboring subpixel belongs.

[0194] 12 shows a ground truth attenuation map 1204. In one implementation, the subpixel values ​​are intensity values ​​normalized between zero and one. In another implementation, all subpixels identified as background in the upsampled cluster map are assigned the same predetermined value. In some implementations, the predetermined value is a zero intensity value.

[0195] In some implementations, a ground truth attenuation map 1204 is generated by the ground truth attenuation map generator 1202 from an upsampled cluster map representing contiguous subpixels in discontinuous regions and subpixels identified as background based on their assigned values. The ground truth attenuation map 1204 is stored in memory for use as ground truth for training a classifier. In one implementation, each subpixel in the ground truth attenuation map 1204 has a normalized value between zero and one. (Three-class map)

[0196] FIG. 13 illustrates one implementation of deriving a ground truth ternary map 1304 from a cluster map. The ground truth ternary map 1304 includes an array of units and assigns at least one output value to each unit in the array. By name, the ternary map implementation of the ground truth ternary map 1304 assigns three output values ​​to each unit in the array, such that for each unit, the first output value corresponds to the classification label or score of the background class, the second output value corresponds to the classification label or score of the cluster center class, and the third output value corresponds to the classification label or score of the cluster / cluster inner class. The ground truth ternary map 1304 is used as ground truth data for training the neural network-based template generator 1512. In some implementations, information from the ground truth ternary map 1304 is also used to prepare the input of the neural network-based base caller 1514.

[0197] 13 shows an exemplary ground truth ternary map 1304. In another implementation, in the upsampled cluster map, contiguous subpixels in discontinuous regions are classified by cluster by the ground truth ternary map generator 1302 as cluster interior subpixels that belong to the same cluster, center-of-mass subpixels as cluster center subpixels, and background subpixels as subpixels that do not belong to any cluster. In some implementations, the classifications are stored in the ground truth ternary map 1304. These classifications and the ground truth ternary map 1304 are stored in memory for use as ground truth for training a classifier.

[0198] In another alternative embodiment, for each cluster, the coordinates of the cluster interior subpixel, cluster center subpixel, and background subpixel are stored in memory for use as ground truth for training the classifier. The coordinates are then downscaled by the factor used to upsample the cluster map. The downscaled coordinates for each cluster are then stored in memory for use as ground truth for training the classifier.

[0199] In yet another embodiment, the ground truth ternary map generator 1302 uses the cluster map to generate ternary ground truth data 1304 from the upsampled cluster map. The ternary ground truth data 1304 labels background subpixels as belonging to a background class, cluster center subpixels as belonging to a cluster center class, and cluster interior subpixels as belonging to a cluster interior class. In some visualization embodiments, color coding can be used to depict and distinguish between different class labels. The ternary ground truth data 1304 is stored in memory for use as ground truth for training a classifier. (Binary (2-class) map)

[0200] FIG. 14 illustrates one embodiment of deriving a ground truth binary map 1404 from a cluster map. The binary map 1404 includes an array of units and assigns at least one output value to each unit in the array. By name, the binary map assigns two output values ​​to each unit in the array, such that for each unit, the first output value corresponds to the classification label or score of the cluster center class and the second output value corresponds to the classification label or score of a non-center class. The binary map is used as ground truth data for training a neural network-based template generator 1512. In some embodiments, information from the binary map is also used to prepare input for the neural network-based base caller 1514.

[0201] 14 shows a ground truth binary map 1404. The ground truth binary map generator 1402 uses the cluster map 120 to generate binary ground truth data 1404 from the upsampled cluster map. The binary ground truth data 1404 labels cluster center subpixels as belonging to a cluster center class and labels all other subpixels as belonging to a non-center class. The binary ground truth data 1404 is stored in memory for use as ground truth for training a classifier.

[0202] In some embodiments, the disclosed technology generates a cluster map 120 for multiple tiles of a flow cell, stores the cluster map in memory, and determines the spatial distribution of clusters within the tile based on the cluster map 120, including their shapes and sizes. The disclosed technology then classifies the subpixels for each cluster in the upsampled cluster map 120 of the clusters within the tile into cluster interior subpixels, cluster center subpixels, and background subpixels that belong to the same cluster. The disclosed technology then stores the classification in memory for use as ground truth for training a classifier, and for each cluster within the cluster, stores the coordinates of the cluster interior subpixels, cluster center subpixels, and background subpixels in memory for use as ground truth for training the classifier. The disclosed technology then downscales the coordinates by the factor used to upsample the cluster map and stores the downscaled coordinates for each cluster in memory for use as ground truth for training the classifier.

[0203] In some embodiments, the flow cell has at least one patterned surface with an array of wells that occupy clusters. In such embodiments, based on the determined shapes and sizes of the clusters, the disclosed techniques identify (1) which wells are substantially occupied by at least one group, (2) which wells are minimally occupied, and (3) which wells are co-occupied by multiple groups. This allows for the determination of metadata for each of multiple clusters that co-occupy the same well, i.e., the center, shape, and size of two or more clusters that share the same well.

[0204] In some embodiments, the solid support on which a sample is amplified into clusters comprises a patterned surface. A "patterned surface" refers to the arrangement of distinct regions within or on an exposed layer of a solid support. For example, one or more regions can be features where one or more amplification primers are present. The features can be separated by interstitial regions where no amplification primers are present. In some embodiments, the pattern can be an xy format of features in rows and columns. In some embodiments, the pattern can be a repetitive sequence of features and / or interstitial regions. In some embodiments, the pattern can be a random sequence of features and / or interstitial regions. Exemplary patterned surfaces that can be used in the methods and compositions described herein are described in U.S. Pat. Nos. 8,778,849, 9,079,148, 8,778,848, and U.S. Patent Application Publication No. 2014 / 0243224, each of which is incorporated herein by reference.

[0205] In some embodiments, the solid support comprises an array of wells or depressions on its surface, which can be fabricated as commonly known in the art using a variety of techniques, including but not limited to photolithography, stamping techniques, molding techniques, and microetching techniques. As understood in the art, the technique used will depend on the composition and shape of the array substrate.

[0206] The features within the patterned surface may be wells in an array of wells (e.g., microwells or nanowells) on glass, silicon, plastic, or other suitable solid support with a patterned covalently attached gel, such as poly(N-(5-azidoacetamylpentyl)acrylamide-co-acrylamide) (PAZAM; see, e.g., U.S. Patent Application Publication Nos. 2013 / 184796, WO 2016 / 066586, and 2015-002813, each of which is incorporated by reference in its entirety). This process creates a gel pad used for sequencing, which can be stable over many cycles of sequencing operations. Covalently attaching a polymer to the wells is useful for maintaining the gel in the structured features throughout the life of the structured substrate during various applications. However, in many embodiments, the gel need not be covalently attached to the wells. For example, in some conditions, silane free acrylamide (SFA) (SFA, see, e.g., U.S. Pat. No. 8,563,477, which is incorporated herein by reference in its entirety) that is not covalently bonded to any portion of the structured substrate can be used as the gel material.

[0207] In certain other embodiments, structured substrates can be created by patterning a solid support material with wells (e.g., microwells or nanocells), coating the patterned support with a gel material (e.g., PAZAM, SFA, or a chemically modified variant thereof), such as an azido-SFA version, and polishing the gel-coated support, e.g., by chemical or mechanical polishing, thereby retaining the gel within the wells but removing or inactivating substantially all of the gel from the interstitial regions on the surface of the structured substrate between the wells. Primer nucleic acids can then be attached to the gel material. A solution of target nucleic acids (e.g., a fragmented human genome) can then be contacted with the polished substrate such that individual target nucleic acids seed individual wells through interaction with primers bound to the gel material, but the target nucleic acids do not occupy the interstitial regions due to the inactivity or inactivity of the gel material. Amplification of the target nucleic acids will be confined to the wells because the absence or inactivity of gel in the interstitial regions prevents outward migration of growing nucleic acid colonies. The process is conveniently manufacturable and scalable, utilizing micro- or nano-fabrication methods.

[0208] As used herein, the term "flow cell" refers to a chamber that includes a solid surface through which one or more fluidic reagents can flow. Examples of flow cells and related fluidic systems and detection platforms that can be easily used in the methods of the present disclosure are described in, for example, Bentley et al., Nature 456:53-59 (2008), WO 04 / 018497, U.S. Patent No. 7,057,026, WO 91 / 06678, U.S. Patent No. 07 / 123744, U.S. Patent No. 7,329,492, U.S. Patent No. 7,211,414, U.S. Patent No. 7,315,019, U.S. Patent No. 7,405,281, and U.S. Patent No. 2008 / 0108082, each of which is incorporated herein by reference.

[0209] Throughout this disclosure, the terms "P5" and "P7" are used when referring to amplification primers. It will be understood that any suitable amplification primers can be used in the methods presented herein, and the use of P5 and P7 is exemplary only. The use of amplification primers such as P5 and P7 on flow cells is known in the art, as exemplified by the disclosures of International Publication Nos. WO 2007 / 010251, WO 2006 / 064199, WO 2005 / 065814, WO 2015 / 106941, WO 1998 / 044151, and WO 2000 / 018957, the entire contents of which are incorporated herein by reference. For example, any suitable forward amplification primer, whether immobilized or in solution, can be useful in the methods presented herein for amplifying complementary sequences and sequences. Similarly, any suitable reverse amplification primer can be useful in the methods provided herein for complementary sequences and amplification of sequences, whether immobilized or in solution. Those skilled in the art will understand how to design and use suitable primer sequences for capture and amplification of nucleic acids provided herein.

[0210] In some embodiments, the flow cell has at least one non-patterned surface, and the clusters are scattered non-uniformly on the non-patterned surface.

[0211] In some embodiments, the density of the clusters is about 100,000 clusters / mm 2 ~approximately 1,000,000 clusters / mm 2 In another embodiment, the density of the clusters is in the range of about 1,000,000 clusters / mm 2 ~approximately 10,000,000 clusters / mm 2 The range is.

[0212] In one embodiment, the preliminary center coordinates of the clusters determined by the base caller are defined within a template image of the tile, hi some embodiments, the pixel resolution, image coordinate system, and measurement scale of the image coordinate system are the same as the template image and the image.

[0213] In another embodiment, the disclosed technology relates to determining metadata about clusters on tiles of a flow cell. First, the disclosed technology accesses (1) a set of images of the tile captured during a sequencing run and (2) preliminary center coordinates of the clusters determined by a base caller.

[0214] Then, for each image set, the disclosed technique obtains, as one of four bases, (1) an origin subpixel containing preliminary center coordinates and (2) a predetermined neighborhood of contiguous subpixels contiguously adjacent to each of the origin subpixels. This generates a base call sequence for each origin subpixel and each predetermined neighborhood of contiguous subpixels. The predetermined neighborhood of contiguous subpixels can be an m×n subpixel patch centered on a subpixel containing the origin subpixel. In one embodiment, the subpixel patch is 3×3 subpixels. In other embodiments, the image patch can be any size, such as 5×5, 15×15, 20×20, etc. In other embodiments, the predetermined neighborhood of contiguous subpixels can be an n-connected subpixel neighborhood centered on a subpixel containing the origin subpixel.

[0215] In one embodiment, the disclosed technique identifies as background those subpixels in the cluster map that do not belong to any of the disjoint regions.

[0216] The disclosed techniques then generate a cluster map that identifies clusters as discontinuous regions of adjacent subpixels that (a) are contiguous to at least a portion of a corresponding one of the origin subpixels and (b) share a substantially matching base call sequence of one of four bases with at least a portion of a corresponding one of the origin subpixels.

[0217] The disclosed technique then stores the cluster map in memory and determines the shape and size of the clusters based on the discontinuous regions in the cluster map. In other embodiments, the centers of the clusters are also determined. (Generating training data for the template generator)

[0218] FIG. 15 is a block diagram illustrating one embodiment for generating the training data used to train the neural network-based template generator 1512 and the neural network-based base caller 1514.

[0219] 16 illustrates characteristics of the disclosed training examples used to train the neural network-based template generator 1512 and the neural network-based base caller 1514. Each training example corresponds to a tile and is labeled with a corresponding ground truth data representation. In some implementations, the ground truth data representation is a ground truth mask or ground truth map that identifies ground truth cluster metadata in the form of a ground truth attenuation map 1204, a ground truth ternary map 1304, or a ground truth binary map 1404. In some implementations, multiple training examples correspond to the same tile.

[0220] In one embodiment, the disclosed technology relates to generating training data 1504 for neural network-based template generation and base calling. The disclosed technology first accesses a number of images 108 of a flow cell 202 captured over multiple cycles of a sequencing operation. The flow cell 202 has a number of tiles. In the number of images 108, each tile has a series of image sets generated over multiple cycles. Each image in the sequence of image sets 108 shows the intensity emission of a particular cluster 302 of tiles and their surrounding background 304 during a particular cycle.

[0221] Training set constructor 1502 then constructs training set 1504 having a plurality of training examples. As shown in FIG. 16 , each training example corresponds to a particular one of the tiles and includes image data from at least some of the image sets in the array of image sets 1602 for that particular one of the tiles. In one implementation, the image data includes images in at least some of the image sets in the array of image sets 1602 for that particular one of the tiles. For example, the images may have a resolution of 1800×1800. In other implementations, the images may have any resolution, such as 100×100, 3000×3000, 10000×10000, etc. In yet other implementations, the image data includes at least one image patch from each of the images. In one implementation, the image patch covers a portion of a particular one of the tiles. In one example, the image patch may have a resolution of 20×20. In other embodiments, the image patches can have any resolution, such as 50x50, 70x70, 90x90, 100x100, 3000x3000, 10000x10000, and the like.

[0222] In some implementations, the image data includes an upsampled representation of the image patch. The upsampled representation may have a resolution of, for example, 80 x 80. In other implementations, the upsampled representation may have any resolution, such as 50 x 50, 70 x 70, 90 x 90, 100 x 100, 3000 x 3000, 10000 x 10000, etc.

[0223] In some implementations, the multiple training examples correspond to the same particular one of the tiles and each include a different image patch from a respective image of at least some of the image sets in the array of image sets 1602 for the same particular one of the tiles. In such implementations, at least some of the different image patches overlap with each other.

[0224] The ground truth generator 1506 then generates at least one ground truth data representation for each of the training examples, which identifies the spatial distribution of the clusters and at least one of their surrounding background, as represented by the image data, including at least one of the cluster shapes, cluster sizes, and / or cluster boundaries, and / or cluster centers.

[0225] In one implementation, the ground truth data representation identifies clusters as discontinuous regions of adjacent sub-pixels, with the centers of the clusters being the centers of the mass sub-pixels within corresponding ones of the discontinuous regions, and their surrounding background.

[0226] In one implementation, the ground truth data representation has an upsampled resolution of 80 x 80. In other implementations, the ground truth data representation can have any resolution, such as 50 x 50, 70 x 70, 90 x 90, 100 x 100, 3000 x 3000, 10000 x 10000, etc.

[0227] In one embodiment, the ground truth data representation identifies each sub-pixel as being either cluster-centered or non-centered, hi another embodiment, the ground truth data representation identifies each sub-pixel as being either cluster-interior, cluster-centered, or surrounding background.

[0228] In some implementations, the disclosed technology stores a training set 1504 and associated ground truth data 1508 in memory as training data 1504 for training a neural network-based template generator 1512 and a neural network-based base caller 1514. The training is handled by a trainer 1510.

[0229] In some embodiments, the disclosed techniques generate training data for a variety of flow cells, sequencing instruments, sequencing protocols, sequencing chemistries, sequencing reagents, and cluster densities. (Neural network-based template generator)

[0230] In inferential or fabrication embodiments, the disclosed techniques use peak detection and segmentation to determine cluster metadata. The disclosed techniques process input image data 1702 derived from a series of image sets 1602 via a neural network 1706 to generate alternative representations 1708 of the input image data 1702. For example, an image set may be for a particular sequencing cycle and include four images, one for each image channel A, C, T, and G. Thus, for a sequencing operation with 50 sequencing cycles, there would be 50 such image sets, for a total of 200 images. When arranged in time, the image sets, with four image patches per image set, form the series of image sets 1602. In some embodiments, image patches of a particular size are extracted from each image in the 50 image sets, forming four image patch sets per image patch set, which in one embodiment is the input image data 1702. In other implementations, the input image data 1702 includes image patch sets having four image patches per image patch set for image patch sets with fewer than 50 sequencing cycles, i.e., fewer than 1, 2, 3, 15, or 20 sequencing cycles.

[0231] 17 illustrates one embodiment of processing input image data 1702 through a neural network-based template generator 1512 to generate an output value for each unit in an array. In one embodiment, the array is an attenuation map 1716. In another embodiment, the array is a ternary map 1718. In yet another embodiment, the array is a binary map 1720. Thus, the array may represent one or more characteristics of each of multiple locations represented in the input image data 1702.

[0232] Unlike training the template generator using the structure of the previous figure, including the ground truth attenuation map 1204, the ground truth ternary map 1304, and the ground truth binary map 1404, the attenuation map 1716, the ternary map 1718, and / or the binary map 1720 are generated by forward propagation of the trained neural network-based template generator 1512. The forward propagation can be during training or during inference. During training, backpropagation-based gradient updates cause the attenuation map 1716, the ternary map 1718, and the binary map 1720 (i.e., cumulatively the output 1714) to progressively match or approach the ground truth attenuation map 1204, the ground truth ternary map 1304, and the ground truth binary map 1404, respectively.

[0233] The size of the image array analyzed during inference, according to one embodiment, depends on the size of the input image data 1702 (e.g., the same or an upscaled or downscaled version). Each unit can represent a pixel, a subpixel, or a superpixel. The output value for each unit of the array can characterize / represent / indicate an attenuation map 1716, a ternary map 1718, or a binary map 1720. In some embodiments, the input image data 1702 is also an array of units at pixel, subpixel, or superpixel resolution. In another such embodiment, the neural network-based template generator 1512 uses semantic segmentation techniques to generate an output value for each unit in the input array. Further details regarding the input image data 1702 can be found in Figures 21b, 22, 23, and 24 and their discussions.

[0234] In some embodiments, the neural network-based template generator 1512 is a fully convolutional network, such as that described in J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," CVPR, (2015), incorporated herein by reference. In other embodiments, the neural network-based template generator 1512 is a U-Net network with skip connections between the decoder and the encoder, such as that described in Ronneberger O, Fischer P, Brox T., "U-net: Convolutional networks for biomedical image segmentation," Med. Image Comput. Comput. Assist. Interv. (2015), available at http: / / link.springer.com / chapter / 10.1007 / 978-3-319-24574-4_28, incorporated herein by reference. The U-Net structure resembles an autoencoder with two main substructures: The system includes: 1) an encoder that takes an input image and reduces its spatial resolution through multiple convolutional layers to generate an encoding; and 2) a decoder that takes the representation and encodes and increases the spatial resolution to generate a reconstructed image as output. U-Net introduces two innovations to this structure: first, the objective function is set to reconstruct the segmentation mask using a loss function, and second, the convolutional layers of the encoder are connected to corresponding layers of the same resolution in the decoder using skip connections. In yet a further embodiment, the neural network-based template generator 1512 is a deep fully convolutional segmentation neural network having an encoder subnetwork and a corresponding decoder network. In another such embodiment, the encoder subnetwork includes a hierarchy of encoders, and the decoder subnetwork includes a hierarchy of decoders that map low-resolution encoder feature maps to full-input-resolution feature maps.Further details regarding segmentation networks can be found in the appendix entitled "Segmentation Networks."

[0235] In one embodiment, neural network-based template generator 1512 is a convolutional neural network. In another embodiment, neural network-based template generator 1512 is a recurrent neural network. In yet another embodiment, neural network-based template generator 1512 is a residual neural network having residual Box and residual connections. In a further embodiment, neural network-based template generator 1512 is a combination of a convolutional neural network and a recurrent neural network.

[0236] It will be appreciated that the neural network-based template generator 1512 (i.e., the neural network 1706 and / or the output layer 1710) can use various padding and striding configurations. It can use different output functions (e.g., classification or regression) and may or may not include one or more fully connected layers. It can use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or asexual convolution, transposed convolution, depth-separable convolution, 1x1 convolution, group convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It can use one or more loss functions, such as logistic regression / logarithmic loss, multiclass cross-entropy / softmax loss, binary cross-entropy loss, mean squared error loss, L1 loss, L2 loss, smoothed L1 loss, and Huber loss. It can use any parallel, efficient, and compression schemes, such as TFRecords, compression encoding (e.g., PNG), sharpening, parallel calls to map transforms, batching, prefetching, model parallelism, data parallelism, and synchronous / asynchronous SGD. It includes nonlinear transformation functions, such as upsampling layers, downsampling layers, recurrent connections, gates and gated memory units (such as LSTM or Gated Recurrent Units (GRUs)), residual blocks, residual connections, highway connections, skip connections, Pejol connections, activation functions (nonlinear transformation functions, e.g., Rectifying Linear Unit (ReLU), Leaky ReLU, Exponential Linear Unit (ELU), Sigmoid and Hyperbolic Tangent (tanh)), batch normalization layers, regularization layers, dropout, pooling layers (e.g., max or mean pooling), global mean pooling layers, and attention mechanisms.

[0237] In some embodiments, each image in the array of image set 1602 covers a tile and shows the intensity emission of a cluster on the tile and their surrounding background, captured for a particular imaging channel during a particular one of multiple sequencing cycles of a sequencing operation performed on the flow cell. In one embodiment, input image data 1702 includes at least one image patch from each of the images in the array of image set 1602. In another such embodiment, the image patch covers a portion of a tile. In one example, the image patch has a resolution of 20x20. In other cases, the resolution of the image patch may range from 20x20 to 10000x10000. In another embodiment, input image data 1702 includes upsampled subpixel resolution representations of image patches from each of the images in the array of image set 1602. In one example, the upsampled subpixel representation has a resolution of 80x80. In other cases, the resolution of the upsampled subpixel representation may range from 80x80 to 10000x10000.

[0238] The input image data 1702 has an array of units 1704 that describe clusters and their surrounding background. For example, an image set may be for a particular sequencing cycle and include four images, one for each image channel A, C, T, and G. Thus, for a sequencing operation with 50 sequencing cycles, there would be 50 such image sets, for a total of 200 images. When arranged in time, the image sets, with four image sets per image set, form the sequence of image sets 1602. In some implementations, image patches of a particular size are extracted from each image in the 50 image sets, forming four image patch sets per image patch set, which in one implementation is the input image data 1702. In other implementations, the input image data 1702 includes image patch sets with fewer than 50 sequencing cycles, i.e., fewer than 1, 2, 3, 15, or 20 sequencing cycles per image patch. An alternative representation is a feature map. The feature maps may be convolutional features or convolutional representations when the neural network is a convolutional neural network. The feature maps may be hidden state features or hidden state representations when the neural network is a recurrent neural network.

[0239] The disclosed technique then processes the alternative representation 1708 through an output layer 1710 to generate an output 1714 having an output value 1712 for each unit in the array 1704. The output layer may be a classification layer, such as a softmax or sigmoid, that generates a per-unit output value. In one implementation, the output layer is a ReLU layer or any other activation function layer that generates a per-unit output value.

[0240] In one embodiment, the units in the input image data 1702 are pixels, and therefore an output value 1712 for each pixel is generated at the output 1714. In another embodiment, the units in the input image data 1702 are sub-pixels, and therefore an output value 1712 for each sub-pixel is generated at the output 1714. In yet another embodiment, the units in the input image data 1702 are super-pixels, and therefore an output value 1712 for each super-pixel is generated at the output 1714. Deriving cluster metadata from attenuation maps, ternary maps and / or binary maps

[0241] 18 illustrates one embodiment of a post-processing technique applied to the attenuation map 1716, ternary map 1718, or binary map 1720 generated by the neural network-based template generator 1512 to derive cluster metadata including cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries. In some embodiments, the post-processing technique is applied by a post-processor 1814, which further includes a threshold holder 1802, a peak locator 1806, and a segmenter 1810.

[0242] The input to the thresholder 1802 is an attenuation map 1716, a ternary map 1718, or a binary map 1720 generated by a template generator 1512, such as the disclosed neural network-based template generator. In one implementation, the thresholder 1802 applies a threshold to values ​​in the attenuation map, ternary map, or binary map to identify background units 1804 (i.e., subpixels that characterize non-cluster background) and non-background units. In other words, once the output 1714 is generated, the thresholder 1802 applies a threshold to the output values ​​of the units 1712 to classify or reclassify a first subset of the units 1712 as “background units” 1804 that depict the background surrounding the cluster and “non-background units” that represent units that may belong to the cluster. The threshold applied by the thresholder 1802 may be preset.

[0243] The input to the peak locator 1806 is also the attenuation map 1716, the ternary map 1718, or the binary map 1720 generated by the neural network-based template generator 1512. In one embodiment, the peak locator 1806 applies peak detection of values ​​in the attenuation map 1716 to the ternary map 1718 or the binary map 1720 to identify center units 1808 (i.e., center subpixels that characterize cluster centers). In other words, the peak locator 1806 processes the output values ​​of the units 1712 in the output 1714 and classifies a second subset of the units 1712 as "center units" 1808 that contain the centers of the clusters. In some embodiments, the centers of the clusters detected by the peak locator 1806 are also the centers of mass of the clusters. The center units 1808 are then provided to the segmenter 1810. Further details regarding the peak locator 1806 can be found in the appendix entitled "Peak Detection."

[0244] Thresholding and peak detection can occur in parallel or after one another, i.e., they are not dependent on each other.

[0245] Also input to the segmenter 1810 is the attenuation map 1716, ternary map 1718, or binary map 1720 generated by the neural network-based template generator 1512. Additional supplemental inputs to the segmenter 1810 include thresholded units (background, non-background) 1804 identified by the thresholder 1802 and center units 1808 identified by the peak locator 1806. The segmenter 1810 uses the background, non-background 1804, and center units 1808 to identify discontinuous regions 1812 (i.e., non-overlapping groups of adjacent clusters / inter-cluster sub-pixels that characterize clusters). In other words, the segmenter 1810 processes the output values ​​of the units 1712 in output 1714 and uses the background and non-background units 1804 and the center unit 1808 to determine the shape of the cluster 1812 as a non-overlapping region of contiguous units separated by the background unit 1804 and centered on the center unit 1808. The output of the segmenter 1810 is cluster metadata 1812. The cluster metadata 1812 identifies the cluster center, cluster shape, cluster size, cluster background, and / or cluster boundary.

[0246] In one embodiment, segmenter 1810 starts with central unit 1808 and, for each central unit, determines a group of consecutive units that represent the same cluster whose center of mass is contained in the central unit. In one embodiment, segmenter 1810 uses a so-called "watershed" segmentation technique to subdivide consecutive clusters into multiple adjacent clusters at intensity valleys. Further details regarding watershed segmentation techniques and other segmentation techniques can be found in the Appendix entitled "Watershed Segmentation."

[0247] In one implementation, the output values ​​of unit 1712 in output 1714 are continuous values, such as those encoded in ground truth attenuation map 1204. In another implementation, the output values ​​are softmax scores, such as those encoded in ground truth ternary map 1304 and ground truth binary map 1404. In one implementation of ground truth attenuation map 1204, consecutive units in corresponding ones of the non-overlapping regions have output values ​​weighted according to the consecutive units' distance from the central unit in the non-overlapping region to which the adjacent units belong. In such an implementation, the central unit has the highest output value in each one of the non-overlapping regions. As described above, during training, backpropagation-based gradient updates cause attenuation map 1716, ternary map 1718, and binary map 1720 (i.e., cumulatively output 1714) to progressively match or approach ground truth attenuation map 1204, ground truth ternary map 1304, and ground truth binary map 1404, respectively. (Pixel domain - intensity extraction from regular cluster shapes)

[0248] The discussion now turns to how the cluster shapes determined by the disclosed techniques can be used to extract the intensities of the clusters. Because clusters typically have irregular shapes and contours, the disclosed techniques can be used to identify which sub-pixels contribute to the irregularly shaped, discontinuous regions that represent the cluster shape.

[0249] 19 illustrates one implementation of extracting cluster intensities in the pixel domain. A "template image" or "template" can refer to a data structure that includes or identifies cluster metadata 1812 derived from an attenuation map 1716, a ternary map 1718, and / or a binary map 1718. The cluster metadata 1812 identifies cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries.

[0250] In some implementations, the template image is in the upsampled sub-pixel domain, distinguishing cluster boundaries at a fine level. However, the sequencing image 108, which contains cluster and background intensity data, is typically in the pixel domain. Therefore, the disclosed technology proposes two approaches to extracting the intensities of irregularly shaped clusters from the optical pixel-resolution sequencing image using cluster shape information encoded in the upsampled sub-pixel resolution template image. In the first approach, shown in FIG. 19, non-overlapping groups of contiguous sub-pixels identified in the template image are located in the pixel-resolution sequencing image, and their intensities are extracted by interpolation. Further details regarding this intensity extraction technique can be found in FIG. 33 and its discussion.

[0251] In one implementation, when the non-overlapping regions have irregular contours and the units are sub-pixels, the cluster intensity 1912 of a given cluster is determined by intensity extractor 1902 as follows:

[0252] First, subpixel locator 1904 identifies the subpixels that contribute to the cluster intensity of a given cluster based on corresponding non-overlapping regions of adjacent subpixels that identify the shape of the given cluster.

[0253] The subpixel locator 1904 then locates the identified subpixels within one or more optical pixel resolution images 1918 generated for one or more imaging channels in the current sequencing cycle. In one embodiment, integer or non-integer coordinates (e.g., floating points) are located within the optical resolution image, the pixel resolution image, after downscaling based on a downscaling factor that matches the upsampling factor used to create the subpixel domain.

[0254] An interpolator and subpixel intensity combiner 1906 for the identified subpixels in the processed images then combines the interpolated intensities and normalizes the combined interpolated intensities to generate a cluster intensity per image for a given cluster in each of the images. The normalization is performed by normalizer 1908 and is based on a normalization factor. In one embodiment, the normalization factor is the number of identified subpixels. This is done to normalize / account for different cluster sizes and non-uniform illumination that clusters receive depending on their location on the flow cell.

[0255] Finally, a cross-channel sub-pixel intensity accumulator 1910 combines the per-image cluster intensities for each of the images to determine a cluster intensity 1912 for a given cluster in the current sequencing cycle.

[0256] The given cluster is then base called based on the cluster strength 1912 in the current sequencing cycle by any one of the base calls discussed in this application to generate base call 1916.

[0257] However, in some implementations, when the cluster size is large enough, the outputs of the neural network-based base caller 1514, i.e., the attenuation map 1716, the ternary map 1718, and the binary map 1720, are in the optical pixel domain. Thus, in such implementations, the template image is also in the optical pixel domain. (Subpixel domain - intensity extraction from regular cluster shapes)

[0258] FIG. 20 illustrates a second approach to extracting cluster intensities in the subpixel domain. In this second approach, the sequencing image is optically upsampled from pixel resolution to subpixel resolution. This results in a correspondence between the cluster shapes representing the subpixels in the template image and the cluster intensities representing the subpixels in the upsampled sequencing image. Cluster intensities are then extracted based on the correspondence. Further details regarding this intensity extraction technique can be found in FIG. 33 and its discussion.

[0259] In one implementation, when the non-overlapping regions have irregular contours and the units are sub-pixels, the cluster intensity 2012 of a given cluster is determined by the intensity extractor 2002 as follows:

[0260] First, the subpixel locator 2004 identifies the subpixels that contribute to the cluster intensity of a given cluster based on the corresponding non-overlapping regions of adjacent subpixels that identify the shape of the given cluster.

[0261] The subpixel locator 2004 then locates the identified subpixels in one or more subpixel resolution images 2018 that are upsampled from the corresponding optical pixel resolution images 1918 generated for one or more imaging channels in the current sequencing cycle. The upsampling can be performed by nearest neighbor intensity extraction, Gaussian intensity extraction, intensity extraction based on averaging 2x2 subpixel areas, intensity extraction based on brightest test of 2x2 subpixel areas, average 3x3 subpixel areas, bilinear intensity extraction, dual intensity extraction, and / or intensity extraction based on weighted region coverage. These techniques are described in detail in the Appendix entitled "Intensity Extraction Methods." The template image can, in some implementations, serve as a mask for intensity extraction.

[0262] A subpixel intensity combiner 2006 in each of the upsampled images then combines the intensities of the identified subpixels and normalizes the combined intensities to generate a cluster intensity per image for a given cluster in each of the upsampled images. The normalization is performed by a normalizer 2008 and is based on a normalization factor. In one embodiment, the normalization factor is the number of identified subpixels. This is done to normalize / account for different cluster sizes and non-uniform illumination that clusters receive depending on their location on the flow cell.

[0263] Finally, a cross-channel sub-pixel intensity accumulator 2010 combines the cluster intensities per image for each upsampled image to determine a cluster intensity 2012 for a given cluster in the current sequencing cycle.

[0264] The given cluster is then base called based on the cluster strength 2012 in the current sequencing cycle by any one of the base calls discussed in this application to generate base call 2016. (A type of neural network-based template generator)

[0265] This discussion details three different implementations of the neural network-based template generator 1512, shown in Figure 21a, including (1) an attenuation map-based template generator 2600 (also called a regression model), (2) a binary map-based template generator 4600 (also called a binary classification model), and (3) a ternary map-based template generator 5400 (also called a ternary classification model).

[0266] In one embodiment, regression model 2600 is a fully convolutional network. In another embodiment, regression model 2600 is a U-Net network with skip connections between the decoder and encoder. In one embodiment, binary classification model 4600 is a fully convolutional network. In another embodiment, binary classification model 4600 is a U-Net network with skip connections between the decoder and encoder. In one embodiment, ternary classification model 5400 is a fully convolutional network. In another embodiment, ternary classification model 5400 is a U-Net network with skip connections between the decoder and encoder. (Input image data)

[0267] 21b illustrates one embodiment of input image data 1702 provided as input to the neural network-based template generator 1512. The input image data 1702 includes a series of image sets 2100 having sequencing images 108 generated during a particular number of initial sequencing cycles of a sequencing operation (e.g., the first 2 to 7 sequencing cycles).

[0268] In some embodiments, the intensities of the sequencing images 108 are corrected for background and / or aligned with each other using affinity transformation. In one embodiment, the sequencing operation utilizes four channel chemistry and each image set has four images. In another embodiment, the sequencing operation utilizes two channel chemistry and each image set has two images. In yet another embodiment, the sequencing operation utilizes one channel chemistry and each image set has two images. In yet another embodiment, each image set has only one image. These and other different embodiments are described in Appendices 6 and 9.

[0269] Each image 2116 in the series of image set 2100 covers a tile 2104 of flow cell 2102 and shows the intensity emission of cluster 2106 on tile 2104 and their surrounding background captured for a particular image channel in a particular one of multiple sequencing cycles of a sequencing operation. In one example, for cycle t1, the image set includes four images 2112A, 2112C, 2112T, 2112G, including one image for each base A, C, T, and G labeled with a corresponding fluorescent dye and imaged in a corresponding wavelength band (image / imaging channel).

[0270] For illustrative purposes, in image 2112G, Figure 21b shows cluster intensity emission as 2108 and background intensity emission as 2110. In another example, for cycle tn, the image set also includes four images 2114A, 2114C, 2114T, 2114G, including one image for each base A, C, T, and G labeled with a corresponding fluorescent dye and imaged in a corresponding wavelength band (image / imaging channel). Also for illustrative purposes, in image 2114A, Figure 21b shows cluster intensity emission as 2118, and in image 2114T, background intensity emission is shown as 2120. (non-image data)

[0271] The input image data 1702 is encoded using intensity channels (also called imaging channels). For each c-image acquired from the sequencer for a particular sequencing cycle, a separate imaging channel is used to encode its intensity signal data. For example, consider a sequencing operation using a two-channel chemistry that generates a red image and a green image in each sequencing cycle. In such a case, the input data 2632 includes (i) a first red imaging channel having w×h pixels that represents the intensity radiation of one or more clusters and their surrounding background captured in the red image, and (ii) a second green imaging channel having w×h pixels that represents the intensity radiation of one or more clusters and their surrounding background captured in the green image.

[0272] In another embodiment, image data is not used as input to the neural network-based template generator 1512 or the neural network-based base caller 1514. Instead, the input to the neural network-based template generator 1512 and the neural network-based base caller 1514 is based on pH changes induced by the release of hydrogen ions during molecular elongation. The pH changes are detected and converted to voltage changes proportional to the number of bases incorporated (e.g., in the case of Ion Torrent).

[0273] In yet another embodiment, input to the neural network-based template generator 1512 and the neural network-based base caller 1514 is constructed from nanopore sensing, which uses a biosensor to measure the disruption of electrical current as an analyte passes through or near the opening of the nanopore, determining the identity of the base. For example, Oxford Nanopore Technologies (ONT) sequencing is based on the following concept: a single strand of DNA (or RNA) is passed through a membrane via a nanopore, and a potential difference is applied across the membrane. Nucleotides present within the pore affect the electrical resistance of the pore, so that current measurements over time can indicate the sequence of DNA bases passing through the pore. This current signal (due to its "squish" appearance when plotted) is the raw data collected by the ONT sequencer. These measurements are stored as 16-bit integer data acquisition (DAC) values ​​taken at a frequency of 4 kHz (for example). With a DNA strand velocity of ~450 base pairs per second, this gives, on average, about 9 raw observations per base. This signal is then processed to identify breaks in the aperture signal that correspond to individual reads. Extension of these raw signals is base called, a process that converts DAC values ​​into sequences of DNA bases. In some embodiments, input data 2632 includes normalized or scaled DAC values. (patch extraction)

[0274] Figure 22 shows one embodiment of extracting patches from the series of image sets 2100 of Figure 21b to generate a series of "downsized" image sets that form the input image data 1702. In another embodiment shown, the sequencing images 108 in the series of image sets 2100 are of size L x L (e.g., 2000 x 2000). In other embodiments, L is any number ranging from 1 to 10,000.

[0275] In one embodiment, patch extractor 2202 extracts patches from sequencing images 108 in sequence of image sets 2100 to generate a sequence of downsized image sets 2206, 2208, 2210, and 2212. Each image in the sequence of downsized image sets is a patch of size M x M (e.g., 20 x 20) extracted from a corresponding sequencing image in sequence of image sets 2100. The size of the patch can be preset. In other alternative embodiments, M is any number between 1 and 1000.

[0276] 22 shows four exemplary series of downsized image sets. A first exemplary series of downsized image sets 2206 is extracted from coordinates 0,0 to 20,20 in the sequencing images 108 in the series of image sets 2100. A second exemplary series of downsized image sets 2208 is extracted from coordinates 20,20 to 40,40 in the sequencing images 108 in the series of image sets 2100. A third exemplary series of downsized image sets 2210 is extracted from coordinates 40,40 to 60,60 in the sequencing images 108 in the series of image sets 2100. A fourth exemplary series of downsized image sets 2212 is extracted from coordinates 60,60 to 80,80 in the sequencing images 108 in the series of image sets 2100.

[0277] In some implementations, the series of downsized image sets forms the input image data 1702 that is provided as input to the neural network-based template generator 1512. Multiple series of downsized image sets can be provided simultaneously as an input batch, and a separate output can be generated for each series in the input batch. (upsampling)

[0278] FIG. 23 illustrates one implementation of upsampling the series of image sets 2100 of FIG. 21 b to generate a series of “upsampled” image sets 2300 that form the input image data 1702 .

[0279] In one embodiment, upsampler 2302 upsamples sequencing images 108 in the series of image sets 2100 by an upsampling factor (eg, 4x) and a series of upsampled image sets 2300 .

[0280] In another embodiment shown, the sequencing images 108 in the series of image sets 2100 are of size L×L (e.g., 2000×2000) and are upsampled by an upsampling factor of 4 to generate upsampled images of size U×U (e.g., 8000×8000) in the series of upsampled image sets 2300.

[0281] In one embodiment, the sequencing images 108 in the sequence of image sets 2100 are fed directly to the neural network-based template generator 1512, and upsampling is performed by an initial layer of the neural network-based template generator 1512. That is, the upsampler 2302 is part of the neural network-based template generator 1512 and operates as the first layer that upsamples the sequencing images 108 in the sequence of image sets 2100 to generate the sequence of upsampled image sets 2300.

[0282] In some implementations, the series of upsampled image sets 2300 form the input image data 1702 that is provided as input to the neural network-based template generator 1512 .

[0283] FIG. 24 illustrates one implementation of extracting patches from the series of upsampled image sets 2300 of FIG. 23 to generate a series of “upsampled and downsized” image sets 2406, 2408, 2410, and 2412 that form the input image data 1702.

[0284] In one embodiment, patch extractor 2202 extracts patches from upsampled images in a series of upsampled image sets 2300 to generate a series of upsampled image sets 2406, 2408, 2410 and a downsized image set 2412. Each upsampled image in the series of upsampled image sets and the downsized image set is a patch of size M×M (e.g., 80×80) extracted from a corresponding upsampled image in the series of upsampled image sets 2300. The size of the patch can be preset. In other alternative embodiments, M is any number in the range of 1 to 1000.

[0285] 24 shows four example series of upsampled and downsized image sets. A first example series of upsampled and downsized image sets 2406 is extracted from coordinates 0,0 to 80,80 within the upsampled images in the series of upsampled image sets 2300. A second example series of upsampled and downsized image sets 2408 is extracted from coordinates 80,80 to 160,160 within the upsampled images in the series of upsampled image sets 2300. A third example series of upsampled and downsized image sets 2410 is extracted from coordinates 160,160 to 240,240 within the upsampled images in the series of upsampled image sets 2300. A fourth example series of upsampled and downsized image sets 2412 is extracted from coordinates 240,240 to 320,320 within the upsampled images in the series of upsampled image sets 2300.

[0286] In some implementations, a series of upsampled and downsized image sets form the input image data 1702 that is provided as input to the neural network-based template generator 1512. Multiple series of upsampled and downsized image sets can be provided simultaneously as an input batch, and a separate output can be generated for each series in the input batch. (output)

[0287] The three models are trained to produce different outputs. This is achieved by using different types of ground truth data representations as training labels. The regression model 2600 is trained to produce outputs that characterize / represent the so-called "attenuation map" 1716. The binary classification model 4600 is trained to produce outputs that characterize / represent the so-called "binary map" 1720. The ternary classification model 5400 is trained to produce outputs that characterize / represent the so-called "ternary map" 1718.

[0288] The output 1714 of each type of model includes a unit array 1712. The unit 1712 can be a pixel, subpixel, or superpixel. The output of each type of model includes an output value for each unit such that the output values ​​of the unit array together characterize / represent / represent an attenuation map 1716 in the case of a regression model 2600, a binary map 1720 in the case of a binary classification model 4600, or a ternary map 1718 in the case of a ternary classification model 5400. Details are as follows: (Ground truth data generation)

[0289] 25 shows one implementation of an overall exemplary process for generating ground truth data for training the neural network-based template generator 1512. For a regression model 2600, the ground truth data may be the attenuation map 1204. For a binary classification model 4600, the ground truth data may be the binary map 1404. For a ternary classification model 5400, the ground truth data may be the ternary map 1304. The ground truth data is generated from cluster metadata. The cluster metadata is generated by the cluster metadata generator 122. The ground truth data is generated by the ground truth data generator 1506.

[0290] In another embodiment shown, ground truth data is generated for tile A on lane A of flow cell A. The ground truth data is generated from sequencing images 108 of tile A captured during sequencing operation A. The sequencing images 108 of tile A are in the pixel domain. In one example with a four-channel chemistry that generates four sequencing images per sequencing cycle, 200 sequencing images 108 are accessed for 50 sequencing cycles. Each of the 200 sequencing images 108 shows the intensity emissions of clusters on tile A and their surrounding background captured in a particular image channel at a particular sequencing cycle.

[0291] Sub-pixel addresser 110 converts arrayed image 108 into the sub-pixel domain (eg, by dividing each pixel into multiple sub-pixels) and generates arrayed image 112 in the sub-pixel domain.

[0292] A base caller 114 (e.g., RTA) then processes the sequencing image 112 in the subpixel domain and generates base calls for each subpixel and each of the 50 sequencing cycles, referred to herein as "subpixel base calls."

[0293] The subpixel base calls 116 are then merged to generate a base call sequence for each subpixel over 50 sequencing cycles. Each subpixel base call sequence has 50 base calls, i.e., one base call for each of the 50 sequencing cycles.

[0294] The searcher 118 evaluates the base call sequences of consecutive subpixels on a pairwise basis. The search involves evaluating each subpixel to determine which of the consecutive subpixels share substantially matching base call sequences. The base call sequences of consecutive subpixels "substantially match" when a predetermined portion of the base calls match a per-ordinal position criterion (e.g., 41 matches in >= 45 cycles, 4 mismatches in <= 45 cycles, 4 mismatches in <= 50 cycles, or 2 mismatches in <= 34 cycles).

[0295] In some embodiments, the base caller 114 also identifies preliminary center coordinates of the cluster. The subpixel containing the preliminary center coordinate is referred to as the center or origin subpixel. Some exemplary preliminary center coordinates (604a-c) identified by the base caller 114 and corresponding origin subpixels (606a-c) are shown in FIG. 6. However, as explained below, identifying the origin subpixel (preliminary center coordinate of the cluster) is not required. In some embodiments, the searcher 118 uses a first search, starting with the origin subpixels 606a-c and continuing through successive non-origin subpixels 702a-c, to identify substantially matching base call sequences of the subpixels. This is optional, as explained below.

[0296] The search for a subpixel's essentially matching base call sequence does not require identification of an origin subpixel (initial center coordinate of the cluster) because the search can be performed for all subpixels and the search need not start at the origin subpixel, but instead at any subpixel (e.g., the 0,0 subpixel or any random subpixel). Thus, the search can start at any subpixel, without having to utilize the origin subpixel, because each subpixel is evaluated to determine whether it shares a substantially matching base call sequence with another contiguous subpixel.

[0297] Regardless of whether the origin subpixel is used, certain clusters that do not include the predicted origin subpixel (the initial center coordinate of the cluster) are identified by base caller 114. Some examples of clusters that are identified by merging subpixel base calls and do not include the origin subpixel are clusters 812a, 812b, 812c, 812d, and 812e in Figure 8a. Therefore, the use of base caller 114 to identify the origin subpixel (the initial center coordinate of the cluster) is optional and not required for searching for substantially matching base call sequences of subpixels.

[0298] Searcher 118: (1) identifies contiguous subpixels having substantially matching base call sequences as so-called "discontinuous regions," (2) further evaluates the base call sequences of these subpixels that do not belong to any of the non-joint regions already identified in (1) to obtain additional discontinuous regions, and (3) then identifies background subpixels as subpixels that do not belong to any of the discontinuous regions already identified in (1) and (2). Action (2) enables the disclosed techniques to identify additional or additional clusters whose centers are not identified by base caller 114.

[0299] The results of searcher 118 are encoded in a so-called "cluster map" for tile A and stored in cluster map data store 120. In the cluster map, each of the clusters on tile A is identified by respective discontinuous regions of adjacent sub-pixels, and background sub-pixels separate the isolated regions to identify the surrounding background on tile A.

[0300] Center of mass (COM) calculator 1004 determines the center of each of the clusters on Tile A by calculating the COM of each of the discontinuous regions as the average of the coordinates of each contiguous subpixel that forms the discontinuous region. The center of mass of the cluster is stored as COM data 2502.

[0301] Subpixel classifier 2504 uses cluster map and COM data 2502 to generate subpixel classifications 2506. Subpixel classification 2506 classifies (1) background subpixels, (2) COM subpixels (one COM subpixel for each discrete region, including the COM of each discrete region), and (3) cluster / intra-cluster subpixels that form each discrete region. That is, each subpixel in the cluster map is assigned one of three categories.

[0302] Based on the subpixel classification 2506 in some embodiments, (i) a ground truth attenuation map 1204 is generated by the ground truth attenuation map generator 1202, (ii) a ground truth binary map 1304 is generated by the ground truth binary map generator 1302, and (iii) a ground truth ternary map 1404 is generated by the ground truth ternary map generator 1402. 1. (Regression model)

[0303] Figure 26 shows one example of a regression model 2600. In another embodiment shown, the regression model 2600 is a fully convolutional network 2602 that processes input image data 1702 through an encoder sub-network and a corresponding decoder sub-network. The encoder sub-network includes a hierarchy of encoders. The decoder sub-network includes a hierarchy of decoders that map low-resolution encoder feature maps to full-input resolution attenuation maps 1716. In another embodiment, the regression model 2600 is a U-Net network 2604 with skip connections between the decoder and encoder. Further details regarding segmentation networks can be found in the appendix titled "Segmentation Networks." (Attenuation map)

[0304] 27 illustrates one implementation of generating a ground truth attenuation map 1204 from a cluster map 2702. The ground truth attenuation map 1204 is used as ground truth data for training the regression model 2600. In the ground truth attenuation map 1204, the ground truth attenuation map generator 1202 assigns a weighted attenuation value to each neighboring subpixel based on a weighted attenuation coefficient. The weighted attenuation value is proportional to the Euclidean distance of the neighboring subpixel from the center of mass (COM) subpixel within the discontinuous region to which the neighboring subpixel belongs, such that the weighted attenuation value is highest (e.g., 1 or 100) for COM subpixels and decreases for subpixels further away from the COM subpixel. In some implementations, the weighted attenuation value is multiplied by a preset coefficient, such as 100.

[0305] Furthermore, the ground truth attenuation map generator 1202 assigns the same predetermined value (eg, the minimum background value) to all background sub-pixels.

[0306] The ground truth attenuation map 1204 represents contiguous subpixels in discontinuous regions and background subpixels based on assigned values. The ground truth attenuation map 1204 also stores the assigned values ​​in a unit array, with each unit in the array representing a corresponding subpixel in the input. (training)

[0307] FIG. 28 shows one implementation of training 2800 of a regression model 2600 using a back-propagation based gradient update technique to modify the parameters of the regression model 2600 until the attenuation map 1716 generated by the regression model 2600 as the training output during training 2800 progressively approaches or matches the ground truth attenuation map 1204 of the ground.

[0308] Training 2800 involves iteratively optimizing to minimize an error 2806 between the attenuation map 1716 and the ground truth attenuation map 1204 and updating the parameters of the regression model 2600 based on the error 2806. In one embodiment, the loss function is the mean squared error, which is minimized for each subpixel between the weighted attenuation values ​​of corresponding subpixels in the attenuation map 1716 and the ground truth attenuation map 1204.

[0309] Training 2800 includes hundreds, thousands, and / or millions of forward propagations 2808 and back propagations 2810, including parallelogram techniques such as batching. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by an annotator 2806. Training 2800 can be operated on by trainer 1510 using a stochastic gradient update algorithm such as Adam. (speculation)

[0310] 29 is an implementation of template generation by regression model 2600 during inference 2900, where attenuation map 1716 is generated by regression model 2600 as an inference output during inference 2900. An example of attenuation map 1716 is disclosed in the Appendix entitled "Regression_Model_Ouput." The Appendix includes unit-weighted attenuation output values ​​2910 that together represent attenuation map 1716.

[0311] The inference 2900 includes hundreds, thousands, and / or millions of forward propagations 2904, including parallelogram techniques such as batching. The inference 2900 is performed on inference data 2908, which includes a series of upsampled and downsized image sets, as the input image data 1702. The inference 2900 can be operated on by a tester 2906. (basin separation)

[0312] 30 includes (i) thresholding the attenuation map 1716 to identify background sub-pixels that characterize cluster backgrounds, and (ii) peak detection to identify center sub-pixels that characterize cluster centers. Thresholding is performed by a threshold keeper 1802 that uses a local threshold binary to generate a binarized output. Peak detection is performed by a peak locator 1806 to identify cluster centers. Further details regarding peak locators can be found in the Appendix titled "Peak Detection."

[0313] 31 shows one implementation of the watershed segmentation technique, which takes as input the background subpixels and the center subpixels, each identified by thresholder 1802, and peak locator 1806 finds the intensity valleys between adjacent clusters and outputs non-overlapping groups of adjacent cluster / intra-cluster subpixels that characterize the clusters. Further details regarding the watershed segmentation technique can be found in the Appendix entitled "Watershed Segmentation."

[0314] In one implementation, the watershed segmentation 3102 takes as input (1) the attenuation map 1716, (2) the negated output values ​​1802, and (3) the cluster centers identified by the peak locator 1806 as input (1) minus the output values ​​2910. Based on the input, the watershed segmentation 3102 then generates an output 3104. In the output 3104, each cluster center is identified as a unique set / group of subpixels that belong to the cluster center (as long as the subpixels are "1" in the binary output, i.e., not background subpixels). Further, clusters are filtered based on containing at least four subpixels. The watershed segmentation 3102 can be part of the segmenter 1810, which in turn is part of the post-processor 1814. (Network structure)

[0315] 32 is a table showing an example U-Net structure for regression model 2600, including details of the layers, the dimensionality of the layer outputs, the magnitude of the model parameters, and the interconnections between layers for regression model 2600. Similar details are disclosed in the file entitled "Regression_Model_Example_Architecture" submitted as an appendix to this application. (Cluster intensity extraction)

[0316] Figure 33 shows a different approach to extracting cluster intensities using cluster shape information identified in a template image. As described above, the template image identifies cluster shape information at upsampled sub-pixel resolution. However, the cluster intensity information is in the sequencing image 108, which is typically at optical resolution.

[0317] According to the first approach, the coordinates of the sub-pixels are located in the sequencing image 108 and their respective intensities are extracted using bilinear interpolation and normalized based on the count of the sub-pixels contributing to the cluster.

[0318] The second approach uses a weighted area coverage technique to modulate the intensity of a pixel according to the number of subpixels that contribute to the pixel. Again, the modulated pixel intensity is normalized by a subpixel count parameter.

[0319] The third technique uses quadratic interpolation to upsample the sequencing image to the subpixel domain, sums the intensities of the upsampled pixels that belong to a cluster, and normalizes the summed intensity based on the count of the upsampled pixels that belong to the cluster. (Experimental results and discussion)

[0320] Figure 34 shows different approaches to base calling using the output of the regression model 2600. In the first approach, the cluster centers identified from the output of the neural network-based template generator 1512 in the template image are fed into a base caller (e.g., Illumina's Time Analysis software, referred to herein as "RTA base calling") for base calling.

[0321] In the second approach, instead of cluster centers, cluster intensities extracted from sequencing images based on cluster shape information in the template image are fed to the RTA base caller for base calling.

[0322] Figure 35 shows the difference in base calling performance when RTA base calling uses ground truth center of mass (COM) positions as cluster centers, as opposed to using non-COM positions as cluster centers. The results show that using COMs improves base calling. (Example of model output)

[0323] Figure 36 shows, on the left, an exemplary attenuation map 1716 produced by the regression model 2600. Figure 36 also shows, on the right, an exemplary ground truth attenuation map 1204 that the regression model 2600 approximates during training.

[0324] Both the attenuation map 1716 and the ground truth attenuation map 1204 depict clusters as discontinuous regions of adjacent sub-pixels, with the centers of the clusters shown as the central sub-pixels at the center of mass of the corresponding ones of the discontinuous regions, and the clusters as their surrounding background.

[0325] Additionally, consecutive sub-pixels in corresponding ones of the discontinuous regions have values ​​weighted according to the distance of the consecutive sub-pixel from the central sub-pixel in the discontinuous region to which the adjacent sub-pixels belong. In one implementation, the central sub-pixel has the highest value in the corresponding one of the discontinuous regions. In one implementation, all background sub-pixels have the same minimum background value in the attenuation map.

[0326] 37 shows one embodiment of a peak locator 1806 that identifies cluster centers in an attenuation map by detecting peaks 3702. Further details regarding the peak locator can be found in the Appendix entitled "Peak Detection."

[0327] Figure 38 compares the peaks detected by the peak locator 1806 in the attenuation map 1716 generated by the regression model 2600 with the peaks in the corresponding ground truth attenuation map 1204. The red markers are the peaks predicted by the regression model 2600 as cluster centers, and the green markers are the ground truth centers of cluster agglomerations. (Further experimental results and considerations)

[0328] 39 shows the performance of regression model 2600 using precision and recalibration statistics. The precision and recalibration statistics demonstrate that regression model 2600 is successful in recovering all identified cluster centers.

[0329] Figure 40 compares the performance of regression model 2600 with the RTA base caller for a library concentration of 20 pM (normal operation). Running the RTA base caller, regression model 2600 identifies 34,323 (4.46%) clusters in a higher cluster density environment (i.e., 988,884 clusters).

[0330] Figure 40 also shows the results of other sequencing metrics such as the number of clusters that pass the chemistry filter ("%PF" (pass filter)), the number of aligned reads ("% aligned"), the number of overlapping reads ("% duplicates"), all reads that aligned to the reference sequence ("% mismatch"), the number of reads that do not match the reference sequence for a quality score of 30 and the bases referred to above ("%Q30 bases").

[0331] Figure 41 compares the performance of regression model 2600 with the RTA base caller for a 30 pM library concentration (high density operation). Running with the RTA base caller, regression model 2600 identifies 34,323 (6.27%) more clusters in a much higher cluster density environment (i.e., 1,351,588 clusters).

[0332] Figure 41 also shows the results of other sequencing metrics such as the number of clusters that pass the chemistry filter ("%PF" (pass filter)), the number of aligned reads ("% aligned"), the number of overlapping reads ("% duplicates"), all reads that aligned to the reference sequence ("% mismatch"), the number of reads that do not match the reference sequence for a quality score of 30 and the bases referred to above ("%Q30 bases").

[0333] Figure 42 compares the number of non-overlapping (unique or overlapping duplicate) proper read pairs, i.e., the number of paired reads where both reads are aligned within a reasonable distance, detected by regression model 2600 with that detected by the RTA base caller. The comparison is performed in both normal operation at 20 pM and high-density operation at 30 pM.

[0334] More importantly, Figure 42 shows that the disclosed neural network-based template generator can detect more clusters with fewer sequencing cycles input to template generation. With just four sequencing cycles, regression model 2600 identifies 11% more non-overlapping correct read pairs than the RTA base caller during normal operation at 20 pM and 33% more non-overlapping correct read pairs than the RTA base caller during high-density operation at 30 pM. With seven sequencing cycles, regression model 2600 identifies 4.5% more non-overlapping correct read pairs than the RTA base caller during normal operation at 20 pM and 6.3% more non-overlapping correct read pairs than the RTA base caller during high-density operation at 30 pM.

[0335] Figure 43 shows on the right the first attenuation map generated by regression model 2600. The first attenuation map identifies the clusters and their surrounding background imaged during normal operation at 20 pM, along with their spatial distribution showing the cluster shapes, cluster sizes, and cluster centers.

[0336] On the left, Figure 43 shows a second attenuation map generated by regression model 2600. The second attenuation map identifies the clusters and their surrounding background imaged during 30 pM high density operation, along with their spatial distribution showing cluster shape, cluster size, and cluster centers.

[0337] Figure 44 compares the performance of regression model 2600 with RTA base caller for a library concentration of 40 pM (high density operation). Regression model 2600 produced 89,441,688 more aligned bases than RTA base caller in a much higher cluster density environment (i.e., 1,509,395 clusters).

[0338] Figure 44 also shows the results of other sequencing metrics such as the number of clusters that pass the chemistry filter ("%PF" (pass filter)), the number of aligned reads ("% aligned"), the number of overlapping reads ("% duplicates"), all reads aligned to the reference sequence ("% mismatch"), the number of reads that mismatch the reference sequence for a quality score of 30 and the bases referred to above ("%Q30 bases"). (Further examples of model outputs)

[0339] Figure 45 shows, on the left, the first attenuation map generated by regression model 2600. The first attenuation map identifies the clusters and their surrounding background imaged during normal operation at 40 pM, along with their spatial distribution showing cluster shape, cluster size, and cluster centers.

[0340] In the top right, Figure 45 shows the results of thresholding and peak location applied to the first attenuation map to distinguish each cluster from each other and from the background and identify their respective cluster centers. In some embodiments, the intensity of each cluster is identified and a chassis filter (or pass filter) is specified that is applied to reduce the mismatch rate. 2. (Binary classification model)

[0341] Figure 46 shows one example of a binary classification model 4600. In another embodiment shown, the binary classification model 4600 is a deep fully convolutional segmentation neural network that processes input image data 1702 through an encoder sub-network and a corresponding decoder sub-network. The encoder sub-network includes a hierarchy of encoders. The decoder sub-network includes a hierarchy of decoders that map low-resolution encoder feature maps to full input resolution binary maps 1720. In another embodiment, the binary classification model 4600 is a U-Net network with skip connections between the decoders and encoders. Further details regarding segmentation networks can be found in the Appendix titled "Segmentation Networks." (binary map)

[0342] The final output layer of the binary classification model 4600 is a unit-wise classification layer that generates a classification label for each unit in the output array. In some implementations, the unit-wise partitioning layer is a subpixel-wise classification layer that generates a softmax classification score distribution for each subpixel in the binary map 1720 across two classes, i.e., the cluster center class and non-cluster class, and the classification label for a given subpixel are determined from the corresponding softmax classification score distribution.

[0343] In another alternative embodiment, the unit-wise classification layer is a sub-pixel-wise classification layer that generates a sigmoid classification score for each sub-pixel in the binary map 1720 such that the activation of the unit is interpreted as the probability that the unit belongs to a first class, and conversely, one minus one gives the probability of belonging to a second class.

[0344] The binary map 1720 represents each subpixel based on its predicted classification score. The binary map 1720 also stores the predicted classification scores in an array of units, with each unit in the array representing a corresponding subpixel in the input. (training)

[0345] FIG. 47 illustrates one implementation of training 4700 a binary classification model 4600 using a backpropagation-based gradient update technique to modify the parameters of the binary classification model 4600 until the binary map 1720 of the binary classification model 4600 progressively approaches or matches the ground truth binary map 1404.

[0346] In the illustrated implementation, the final output layer of the binary classification model 4600 is a softmax-based per-subpixel classification layer. In another implementation of softmax, the ground truth binary map generator 1402 assigns each ground truth subpixel either (i) a cluster center value pair (e.g., [1, 0]) or (ii) a non-center value pair (e.g., [0, 1]).

[0347] In a cluster center value pair [1,0], the first value [1] represents the cluster center class label and the second value [0] represents the non-center class label. In a non-center value pair [0,1], the first value [0] represents the cluster center class label and the second value [1] represents the non-center class label.

[0348] The ground truth binary map 1404 represents each subpixel based on an assigned value pair / value. The ground truth binary map 1404 also stores the assigned value pairs / values ​​in a unit array, with each unit in the array representing a corresponding subpixel in the input.

[0349] Training involves iteratively optimizing a loss function that minimizes the error 4706 (e.g., softmax error) between the binary map 1720 and the ground truth binary map 1404, and updating the parameters of the binary classification model 4600 based on the error 4706.

[0350] In one implementation, the loss function is a custom weighted binary cross-entropy loss, and the error 4706 is minimized for each subpixel between the predicted classification score (e.g., softmax score) and the labeled class score (e.g., softmax score), as shown in FIG. 47, and between the labeled class score (e.g., softmax score) of the corresponding subpixel in the binary map 1720 and the ground truth binary map 1404.

[0351] The custom-weighted loss function gives more weight to the COM subpixel each time it is misclassified, multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Further details regarding the custom-weighted loss function can be found in the appendix titled "Custom-Weighted Loss Function."

[0352] Training 4700 includes hundreds, thousands, and / or millions of forward propagations 4708 and back propagations 4710, including parallelogram techniques such as batching. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by an annotator 2806. Training 2800 can be operated on by a trainer 1510 using a stochastic gradient update algorithm such as Adam.

[0353] FIG. 48 is another embodiment of training 4800 a binary classification model 4600 in which the final output layer of the binary classification model 4600 is a sigmoid-based subpixel-wise classification layer.

[0354] In another sigmoid implementation, the ground truth binary map generator 1302 assigns each ground truth subpixel either (i) a cluster center value (e.g., [1]) or (ii) a non-center value (e.g., [0]). The COM subpixel is assigned the cluster center value pair / value, and all other subpixels are assigned the non-center value pair / value.

[0355] For cluster central values, values ​​between 0 and 1 (e.g., values ​​above 0.5) represent central class labels. For non-central values, values ​​below the threshold between 0 and 1 (e.g., values ​​below 0.5) represent non-central class labels.

[0356] The ground truth binary map 1404 represents each subpixel based on an assigned value pair / value. The ground truth binary map 1404 also stores the assigned value pairs / values ​​in a unit array, with each unit in the array representing a corresponding subpixel in the input.

[0357] Training involves iteratively optimizing a loss function that minimizes the error 4806 (e.g., a sigmoid error) between the binary map 1720 and the ground truth binary map 1404, and updating the parameters of the binary classification model 4600 based on the error 4806.

[0358] In one implementation, the loss function is a custom weighted binary cross-entropy loss, and the error 4806 is minimized for each subpixel between the predicted scores (e.g., sigmoid scores) of corresponding subpixels in the binary map 1720 and the ground truth binary map 1404, as shown in FIG. 48, and the labeled scores (e.g., sigmoid scores) of corresponding subpixels in the binary map 1720 and the ground truth binary map 1404, as shown in FIG.

[0359] The custom-weighted loss function gives more weight to the COM subpixel each time it is misclassified, multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Further details regarding the custom-weighted loss function can be found in the appendix titled "Custom-Weighted Loss Function."

[0360] Training 4800 includes hundreds, thousands, and / or millions of forward propagations 4808 and back propagations 4810, including parallelogram techniques such as batching. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 2800 can be operated on by trainer 1510 using a stochastic gradient update algorithm such as Adam.

[0361] FIG. 49 shows another implementation of input image data 1702 provided to a binary classification model 4600 and the corresponding class labels 4904 used to train the binary classification model 4600.

[0362] In another illustrated embodiment, input image data 1702 is serially upsampled to comprise downsized image set 4902. Class labels 4904 include two classes: (1) "no cluster center" and (2) "cluster center," which are distinguished using different output values: (1) light green units / subpixels 4906 represent subpixels predicted by binary classification model 4600 to not contain cluster centers, and (2) dark green subpixels 4908 represent units / subpixels predicted by binary classification model 4600 to contain cluster centers. (speculation)

[0363] Figure 50 is an implementation of template generation by a binary classification model 4600 during inference 5000, in which a binary map 1720 is generated by the binary classification model 4600 as an inference output during inference 5000. An example of a binary map 1720 includes binary classification scores 5010 per unit that together represent the binary map 1720. In a softmax application, the binary map 1720 has a first array 5002a of classification scores per unit for non-center classes and a second array 5002b of classification scores per unit for cluster center classes.

[0364] The inference 5000 includes hundreds, thousands, and / or millions of forward propagations 5004, including parallelogram techniques such as batching. The inference 5000 is performed on inference data 2908, which includes a series of upsampled and downsized image sets, as input image data 1702. The inference 5000 can be operated on by a tester 2906.

[0365] In some implementations, the binary map 1720 is subjected to the post-processing techniques described above, such as thresholding, peak detection, and / or waterjet segmentation, to generate cluster metadata. (Peak detection)

[0366] Figure 51 shows one embodiment of subjecting a binary map 1720 to peak detection to identify cluster centers. As described above, the binary map 1720 is an array of units that classify each subpixel based on a predicted classification score, with each unit in the array representing a corresponding subpixel in the input. The classification score can be a softmax score or a sigmoid score.

[0367] In softmax applications, the binary map 1720 includes two arrays: (1) a first array 5002a of classification scores per unit of the non-center classes, and (2) a second array 5002b of classification scores per unit of the cluster center classes, where each unit in both arrays represents a corresponding subpixel in the input.

[0368] To determine which subpixels in the input contain cluster centers and which do not, the peak locator 1806 applies peak detection on the units in the binary map 1720. Peak detection identifies units with classification scores (e.g., softmax / sigmoid scores) above a preset threshold. The identified units are inferred as cluster centers, and their corresponding subpixels in the input are determined to contain cluster centers and are stored as cluster center subpixels in the subpixel classification data store 5102. Further details regarding the peak locator 1806 can be found in the Appendix titled "Peak Detection."

[0369] The remaining units in the input and their corresponding subpixels do not contain cluster centers and are stored as non-central subpixels in the subpixel classification data store 5102.

[0370] In some implementations, before applying peak detection, units with classification scores below a certain background threshold (e.g., 0.3) are set to zero. In some implementations, such units and their corresponding subpixels in the input are inferred to represent the background surrounding the cluster and are stored as background subpixels in subpixel classification data store 5102. In other implementations, such units can be considered noise and ignored. (Example of model output)

[0371] Figure 52a shows an example binary map generated by binary classification model 4600 on the left. Figure 52a also shows an example ground truth binary map on the right that binary classification model 4600 approximates during training. The binary map has multiple subpixels and classifies each subpixel as either a cluster center or a non-center. Similarly, the ground truth binary map has multiple subpixels and classifies each subpixel as either a cluster center or a non-center. (Experimental results and discussion)

[0372] Figure 52b shows the performance of the binary classification model 4600 using recalibration and refinement statistics. By applying these statistics, the binary classification model 4600 performs the RTA base caller. (Network structure)

[0373] 53 is a table illustrating an example structure of a binary classification model 4600, along with details of the layers, the dimensionality of the layer outputs, the magnitude of the model parameters, and the interconnections between layers of the binary classification model 4600. Similar details are disclosed in the appendix entitled "Binary_Classification_Model_Example_Architecture." 3. Three-way (three-class) classification model

[0374] Figure 54 shows one implementation of a ternary classification model 5400. In another implementation shown, the ternary classification model 5400 is a deep fully convolutional segmentation neural network that processes input image data 1702 through an encoder sub-network and a corresponding decoder sub-network. The encoder sub-network includes a hierarchy of encoders. The decoder sub-network includes a hierarchy of decoders that map low-resolution encoder feature maps to full-input-resolution ternary maps 1718. In another implementation, the ternary classification model 5400 is a U-Net network with skip connections between the decoders and encoders. Further details regarding segmentation networks can be found in the Appendix titled "Segmentation Networks." (Triple Map)

[0375] The final output layer of the ternary classification model 5400 is a unit-wise classification layer that generates a classification label for each unit in the output array. In some implementations, the unit-wise segmentation layer is a subpixel-wise classification layer that generates a softmax classification score distribution for each subpixel in the ternary map 1718 across three classes: background class, cluster center class, and cluster / intra-cluster class, and the classification label for a given subpixel is determined from the corresponding softmax classification score distribution.

[0376] The ternary map 1718 represents each subpixel based on its predicted classification score. The ternary map 1718 also stores the predicted classification scores in an array of units, with each unit in the array representing a corresponding subpixel in the input. (training)

[0377] FIG. 55 illustrates one implementation of training 5500 a ternary classification model 5400 using a backpropagation-based gradient update technique that modifies the parameters of the ternary classification model 5400 until the ternary map 1718 of the ternary classification model 5400 progressively approaches or matches the training ground truth ternary map 1304.

[0378] In the illustrated implementation, the final output layer of the ternary classification model 5400 is a softmax-based subpixel-wise classification layer. In another implementation, the ground truth ternary map generator 1402 assigns either (i) a background value triplet (e.g., [1, 0, 0]), (ii) a cluster center value triplet (e.g., [0, 1, 0]), or (iii) a cluster / cluster interior value triplet (e.g., [0, 0, 1]).

[0379] Background subpixels are assigned background value triplets, center of mass (COM) subpixels are assigned cluster center value triplets, and cluster / intra-cluster subpixels are assigned cluster / intra-cluster value triplets.

[0380] In the background value triplet [1,0,0], the first value [1] represents the background class label, the second value [0] represents the cluster center label, and the third value [0] represents the cluster / intra-cluster class label.

[0381] In the cluster center triplet [0, 1, 0], the first value [0] represents the background class label, the second value [1] represents the cluster center label, and the third value [0] represents the cluster / intra-cluster class label.

[0382] In the cluster / cluster inner value triplet [0, 0, 1], the first value [0] represents the background class label, the second value [0] represents the cluster center label, and the third value [1] represents the cluster / cluster inner class label.

[0383] The ground truth ternary map 1304 represents each subpixel based on an assigned value triplet. The ground truth ternary map 1304 also stores the assigned triplets in a unit array, with each unit in the array representing a corresponding subpixel in the input.

[0384] Training involves iteratively optimizing a loss function that minimizes the error 5506 (e.g., softmax error) between the ternary map 1718 and the ground truth ternary map 1304, and updating the parameters of the ternary classification model 5400 based on the error 5506.

[0385] In one implementation, the loss function is a custom weighted categorization cross-entropy loss, and the error 5506 is minimized for each subpixel between the predicted classification score (e.g., softmax score) and the labeled class score (e.g., softmax score), as shown in FIG. 54, and between the labeled class score (e.g., softmax score) of the corresponding subpixel in the ternary map 1718 and the ground truth ternary map 1304.

[0386] The custom-weighted loss function gives more weight to the COM subpixel each time it is misclassified, multiplied by the corresponding reward (or penalty) weight specified in the reward (or penalty) matrix. Further details regarding the custom-weighted loss function can be found in the appendix titled "Custom-Weighted Loss Function."

[0387] Training 5500 includes hundreds, thousands, and / or millions of forward propagations 5508 and back propagations 5510, including parallelogram techniques such as batching. Training data 1504 includes a series of upsampled and downsized image sets as input image data 1702. Training data 1504 is annotated with ground truth labels by annotator 2806. Training 5500 can be operated on by trainer 1510 using a stochastic gradient update algorithm such as Adam.

[0388] FIG. 56 shows one embodiment of input image data 1702 provided to a three-way classification model 5400 and the corresponding class labels used to train the three-way classification model 5400.

[0389] In another illustrated embodiment, input image data 1702 is serially upsampled to comprise downsized image set 5602. Class labels 5604 include three classes: (1) a "background class," (2) a "cluster center class," and (3) a "cluster interior class," which are distinguished using different output values. For example, some of these different output values ​​can be visually represented as follows: (1) gray units / subpixels 5606 represent subpixels predicted by ternary classification model 5400 to be background, (2) dark green units / subpixels 5608 represent subpixels predicted by ternary classification model 5400 to contain cluster centers, and (3) light green subpixels 5610 represent subpixels predicted by ternary classification model 5400 to contain the interiors of clusters. (Network structure)

[0390] 57 is a table illustrating an example structure of a ternary classification model 5400, along with details of the layers, the dimensionality of the layer outputs, the magnitude of the model parameters, and the interconnections between layers of the ternary classification model 5400. Similar details are disclosed in the appendix entitled "Ternary_Classification_Model_Example_Architecture." (speculation)

[0391] Figure 58 illustrates an implementation of template generation by a ternary classification model 5400 during inference 5800, in which a ternary map 1718 is generated by the ternary classification model 5400 as an inference output during inference 5800. An example of the ternary map 1718 is disclosed in the appendix titled "Ternary_Classification_Model_Output." The appendix includes binary classification scores 5810 per unit that together represent the ternary map 1718. In a softmax application, the appendix has a first array 5802a of classification scores per unit for background classes, a second array 5802b of classification scores per unit for cluster center classes, and a third array 5802c of classification scores per unit for cluster / intra-cluster classes.

[0392] The inference 5800 includes hundreds, thousands, and / or millions of forward propagations 5804, including parallelogram techniques such as batching. The inference 5800 is performed on inference data 2908, which includes a series of upsampled and downsized image sets, as input image data 1702. The inference 5800 can be operated on by a tester 2906.

[0393] In some implementations, the ternary map 1718 is generated by the ternary classification model 5400 using the post-processing techniques described above, such as thresholding, peak detection, and / or waterjet segmentation.

[0394] FIG. 59 graphically illustrates the ternary map 1718 generated by the ternary classification model 5400 with the ternary softmax classification score distributions for three corresponding classes, namely, the background class 5906, the cluster center class 5902, and the cluster / cluster inner class 5904, respectively.

[0395] Figure 60 shows the unit array generated by the ternary classification model 5400 along with the output values ​​for each unit. As shown, each unit has three output values ​​for three corresponding classes: background class 5906, cluster center class 5902, and cluster / cluster inner class 5904. For each classification (column-wise), each unit is assigned the class with the highest output value, as indicated by the class in parentheses for each unit. In some implementations, the output values ​​6002, 6004, and 6006 are analyzed for each of the respective classes 5906, 5902, and 5904 (row-wise). (Peak detection and watershed division)

[0396] Figure 61 shows one embodiment in which the ternary map 1718 is subjected to post-processing to identify cluster centers, cluster backgrounds, and cluster interiors. As described above, the ternary map 1718 is an array of units that classify each subpixel based on a predicted classification score, with each unit in the array representing a corresponding subpixel in the input. The classification score may be a softmax score.

[0397] For softmax applications, the ternary map 1718 includes three arrays: (1) a first array 5802a of per-unit classification scores for the background class, (2) a second array 5802b of per-unit classification scores for the cluster center classes, and (3) a third array 5802c of per-unit classification scores for the cluster inner classes. In all three arrays, each unit represents a corresponding subpixel in the input.

[0398] To determine which subpixels in the input contain cluster centers that contain the interior of a cluster and include background, the peak locator 1806 applies peak detection to the softmax values ​​in the ternary map 1718 of the cluster center class 5802b. Peak detection identifies units with classification scores (e.g., softmax scores) above a preset threshold. The identified units are inferred as cluster centers, and their corresponding subpixels in the input are determined to contain cluster centers and are stored as cluster center subpixels in the subpixel classification and segmentation data store 6102. Further details regarding the peak locator 1806 can be found in the Appendix titled "Peak Detection."

[0399] In some implementations, before applying peak detection, units with classification scores below a certain noise threshold (e.g., 0.3) are set to zero. Such units can be considered noise and can be ignored.

[0400] Also, those subpixels that have a classification score for background class 5802a above a certain background threshold (e.g., 0.5 or greater) and their corresponding subpixels in the input are inferred to represent the background surrounding the cluster and are stored as background subpixels in the subpixel classification and segmentation data store 6102.

[0401] A watershed segmentation algorithm operated by watershed segmentation 3102 is then used to determine the shape of the clusters. In some implementations, background units / subpixels are used as a mask by the watershed segmentation algorithm. The classification scores of units / subpixels inferred as cluster centers and cluster interiors are summed to generate so-called "cluster labels." The cluster centers are used as watershed markers for separation by intensity valleys by the watershed segmentation algorithm.

[0402] In one embodiment, the negatively polarized cluster labels are provided as an input image to a watershed segmenter 3102, which performs segmentation and generates cluster shapes as discontinuous regions of adjacent cluster interior subpixels separated by background subpixels. Additionally, each discontinuous region includes a corresponding cluster center subpixel. In some embodiments, the corresponding cluster center subpixel is the center of the region to which it belongs. In other embodiments, the center of mass (COM) of the discontinuous region is calculated based on the underlying location coordinates and stored as the new center of the cluster.

[0403] The output of watershed segmentation 3102 is stored in subpixel classification and segmentation data store 6102. Further details regarding the watershed segmentation algorithm and other segmentation algorithms can be found in the appendix entitled "Watershed Segmentation."

[0404] Example outputs of the peak locator 1806 and watershed division 3102 are shown in FIGS. 62a, 62b, 63, and 64. (Example of model output)

[0405] Figure 62a shows an example prediction of the ternary classification model 5400. Figure 62a shows four maps, each with a unit arrangement. The first map 6202 (far left) shows the output value of each unit of the cluster center class 5802b. The second map 6204 shows the output value of each unit of the cluster / cluster inner class 5802c. The third map 6206 (far right) shows the output value of each unit of the background class 5802a. The fourth map 6208 (bottom) is a binary mask of the ground truth ternary map 6008, assigning each unit the class label with the highest output value.

[0406] Figure 62b shows another example prediction of the ternary classification model 5400. Figure 62b shows four maps, each with a unit arrangement. The first map 6212 (bottom) shows the output value of each unit of the cluster / cluster inner class. The second map 6214 shows the output value of each unit of the cluster center class. The third map 6216 (rightmost) shows the output value of each unit of the background class. The fourth map (top) 6210 is a ground truth ternary map that assigns each unit the class label with the highest output value.

[0407] Figure 62c shows yet another example prediction of the ternary classification model 5400. Figure 64 shows four maps, each with a unit arrangement. The first map 6220 (bottom) shows the output value of each unit of the cluster / cluster inner class. The second map 6222 shows the output value of each unit of the cluster center class. The third map 6224 (rightmost) shows the output value of each unit of the background class. The fourth map 6218 (top) is a ground truth ternary map that assigns each unit the class label with the highest output value.

[0408] Figure 63 shows one embodiment for deriving cluster centers and cluster shapes from the output of the ternary classification model 5400 of Figure 62a by subjecting the output to post-processing. Post-processing (e.g., peak locations, wash segmentation) produces cluster shape data and other metadata identified in a cluster map 6310. (Experimental results and discussion)

[0409] Figure 64 compares the performance of the binary classification model 4600, the regression model 2600, and the RTA base caller. Performance is evaluated using various sequencing metrics. One metric is the total number of clusters detected ("# clusters"), which can be measured by the number of unique cluster centers detected. Another metric is the number of detected clusters that pass the chisity filter ("%PF" (pass filter)). During cycles 1-25 of the sequencing operation, the chisity filter removes the least-confident clusters from the image extraction results. A cluster "passes the filter" if no more than one base call has a chisity value of less than 0.6 in the first 25 cycles. The chisity filter is defined as the ratio of the brightest base intensity divided by the sum of the brightest test and the second brightest base intensity. This metric goes beyond the quantity of detected clusters and also their quality, i.e., how many of the detected clusters can be used for accurate base calling and downstream secondary and ternary analyses, such as variant calling and variant pathogenicity annotation.

[0410] Other metrics measuring how good a detected cluster is for downstream analysis include the number of aligned reads generated from the detected cluster ("% Aligned"), the number of duplicate reads generated from the detected cluster ("% Duplicate"), the number of reads generated from the detected cluster that mismatch the reference sequence for all reads aligned to the reference sequence ("Mismatched"), the number of reads generated from the detected cluster that are ignored for alignment because their portion does not sufficiently match the reference sequence on either side ("% Soft Clipped"), the number of bases called for the detected cluster that have a quality score of 30 and are above ("% Q30 Bases"), the number of paired reads generated from the detected cluster that are aligned within a reasonable distance ("Total Correct Read Pairs"), and the number of unique or overlapping correct read pairs generated from the detected cluster ("Non-Overlapping Correct Read Pairs").

[0411] As shown in FIG. 64, both the binary classification model 4600 and the regression model 2600 outperform the RTA base caller in template generation for the majority of metrics.

[0412] Figure 65 compares the performance of the three-way classification model 5400 with that of the RTA-based caller under three conditions, five sequencing metrics, and two operating densities.

[0413] In the first scenario, referred to as "RTA," cluster centers are detected by the RTA base caller, intensity extraction from the clusters is performed by the RTA base caller, and the clusters are also base called using the RTA base caller. In the second scenario, referred to as "RTA IE," cluster centers are detected by the ternary classification model 5400, but intensity extraction from the clusters is performed by the RTA base caller, and the clusters are also base called using the RTA base caller. In the third scenario, referred to as "Self IE," cluster centers are detected by the ternary classification model 5400, and intensity extraction from the clusters is performed using the cluster shape-based intensity extraction technique disclosed herein (note that cluster shape information is generated by the ternary classification model 5400). However, the clusters are base called using the RTA base caller.

[0414] Performance is compared between the three-way classification model 5400 and the RTA base caller along the following five metrics: (1) total number of detected clusters (“# clusters”), (2) number of detected clusters passing the chemistry filter (“#PF”), (3) number of unique or overlapping suitable read pairs generated from the detected clusters (“#non-overlapping suitable read pairs”), (4) sequence reads generated from the detected clusters and the reference sequence after alignment (“Mismatch Rate”), and (5) percentage of mismatches between detected clusters with a quality score of 30 (“%Q30”).

[0415] We compare performance between the ternary classification model 5400 and the RTA base caller under three conditions, comparing five metrics for two types of sequencing runs: (1) normal run with 20 pM library concentration, and (2) high-density run with 30 pM library concentration.

[0416] As shown in FIG. 65, a three-way classification model 5400 implements the RTA base caller for all metrics.

[0417] FIG. 66 shows that under the same three conditions, five metrics, and two motion densities, regression model 2600 outperforms the RTA base caller for all metrics.

[0418] FIG. 67 focuses on the final layer 6702 of the neural network-based template generator 1512.

[0419] Figure 68 visualizes what the final layer 6702 of the neural network-based template generator 1512 has learned as a result of back-propagation-based gradient update training. The illustrated embodiment visualizes 24 out of 32 convolution filters of the final layer 6702 overlaid on the ground truth cluster shapes. As shown in Figure 68, the final layer 6702 has learned cluster metadata including the spatial distribution of clusters, such as cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and cluster boundaries.

[0420] Figure 69 overlays the cluster center predictions of the binary classification model 4600 (in blue) on those of the RTA base caller (in pink). Predictions are made on sequencing image data from an Illumina NextSeq sequencer.

[0421] Figure 70 overlays the cluster center predictions made by the RTA base caller (in pink) on a visualization of the trained convolutional filters in the final layer of binary classification model 4600. These convolutional filters are learned as a result of sequencing image data from an Illumina NextSeq sequencer.

[0422] Figure 71 shows one embodiment of training data used to train the neural network-based template generator 1512. In this alternative embodiment, the training data is obtained from a high-density flow cell that generates data using Storm probe images. In another embodiment, the training data is obtained from a high-density flow cell that generates data with fewer bridge amplification cycles.

[0423] FIG. 72 is an example of using beads for image registration based on cluster center prediction in a neural network-based template generator 1512.

[0424] Figure 73 shows one embodiment of cluster statistics for clusters identified by the neural network-based template generator 1512. The cluster statistics include the number of contributing subpixels and the cluster size based on GC content.

[0425] Figure 74 shows how the ability of the neural network-based template generator 1512 to distinguish between adjacent clusters improves as the number of initial sequencing cycles in which the input image data 1702 is used increases from 5 to 7. For five sequencing cycles, a single cluster is identified by a single, discontinuous region of contiguous subpixels. For seven sequencing cycles, the single cluster is split into two adjacent clusters, each with its own discontinuous region of adjacent subpixels.

[0426] Figure 75 shows the difference in base calling performance when the RTA base caller uses ground truth mass (COM) positions as cluster centers as opposed to when non-COM positions are used as cluster centers.

[0427] FIG. 76 shows the performance of the neural network-based template generator 1512 on additional detected clusters.

[0428] FIG. 77 shows different data sets used to train the neural network-based template generator 1512.

[0429] Figure 78 shows the processing steps used by the RTA base caller for base calling, according to one embodiment. Figure 78 also shows the processing steps used by the disclosed neural network-based base caller for base calling, according to two embodiments. As shown in Figure 78, the neural network-based base caller 1514 can streamline the base calling process by eliminating many of the processing steps used by the RTA base caller. The streamlining improves base calling accuracy and scale. In a first embodiment of the neural network-based base caller 1514, it performs base calling using location / position information of cluster centers identified from the output of the neural network-based template generator 1512. In a second embodiment, the neural network-based base caller 1514 does not use location / position information of cluster centers for base calling. The second embodiment is used when a patterned flow cell design is used for cluster generation. The patterned flow cell contains nanowells that are precisely positioned relative to known reference locations and provide a pre-positioned cluster distribution on the patterned flow cell. In another embodiment, the neural network-based base caller 1514 base calls clusters generated on a random flow cell. (Neural network-based base calling)

[0430] The discussion herein describes neural network-based base calling, in which a neural network is trained to map sequencing images to base calls. The discussion is organized as follows: First, the input to the neural network is described. Next, the structure and form of the neural network is described. Finally, the output of the neural network is described. input

[0431] FIG. 79 shows one embodiment of base calling using a neural network 7906. Primary input: Image channel

[0432] The primary input to the neural network 7906 is image data 7902. The image data 7902 is derived from the sequencing image 108 generated by the sequencer 102 during a sequencing operation. In one embodiment, the image data 7902 includes n×n image patches extracted from the sequencing image 102, where n is any number ranging from 1 to 10,000. The sequencing operation generates m image(s) per sequencing cycle for m corresponding image channels, and an image patch is extracted from each of the m image(s) to prepare the image data for a particular sequencing cycle. In different embodiments, such as 4-, 2-, and 1-channel chemistries, m is 4 or 2. In other embodiments, m is greater than 1, 3, or 4. The image data 7902 is in the optical pixel domain in some embodiments and in the upsampled subpixel domain in other embodiments.

[0433] Image data 7902 includes data from multiple sequencing cycles (e.g., a current sequencing cycle, one or more preceding sequencing cycles, and one or more successive sequencing cycles). In one embodiment, image data 7902 includes data from three sequencing cycles, such that the current (time t) sequencing cycle data being base called is accompanied by (i) data from the left flanking / context / previous / preceding / previous (time t-1) sequencing cycle, and (ii) data from the right flanking / context / next / successive / successive (time t+1) sequencing cycle. In other embodiments, image data 7902 includes data from a single sequencing cycle.

[0434] Image data 7902 shows the intensity emissions of one or more clusters and their surrounding background. In one embodiment, when a single target cluster is base called, image patches are extracted from sequencing image 108 in such a way that each image patch contains the center of the target cluster within its center pixel, a concept referred to herein as "target cluster center patch extraction."

[0435] Image data 7902 is encoded in input data 7904 using intensity channels (also called image channels). A separate image channel is used to encode the intensity data for each of the m images acquired from sequencer 102 for a particular sequencing cycle. For example, consider a sequencing operation using a two-channel chemistry that produces a red image and a green image in each sequencing cycle, then input data 7904 includes (i) a first red image channel having n x n pixels representing the intensity emission of one or more clusters and their surrounding background captured in the red image, and (ii) a second green image channel having n x n pixels representing the intensity emission of one or more clusters and their surrounding background captured in the green image. Supplementary input: distance channel

[0436] The image data 7902 is accompanied by supplemental distance data (also called the distance channel). The distance channel provides an additive bias that is incorporated into the feature maps generated from the image channel. This additive bias contributes to base calling accuracy because it is based on the pixel-center-cluster center(s) distance encoded for each pixel in the distance channel.

[0437] In a "single target cluster" base calling implementation, for each image channel (image patch) in the input data 7904, a supplemental distance channel identifies and base calls the distance of the center of that pixel from the center of the target cluster containing that center pixel. The distance channel thereby indicates the distance of each of the pixels of the image patch from the center pixel of the image patch.

[0438] In a "multi-cluster" base calling implementation, for each image channel (image patch) in the input data 7904, the supplemental distance channel identifies the center-to-center distance of each pixel from the closest one of the clusters, which is selected based on the center-to-center distance between the pixel and each of the clusters.

[0439] In a "multi-cluster shape-based" base calling implementation, for each image channel (image patch) in the input data 7904, the supplemental distance channel identifies the center-to-center distance of each cluster pixel from its assigned cluster, which is selected based on classifying each cluster pixel into only one cluster. Supplementary input: Scaling channel

[0440] The image data 7902 is accompanied by supplemental scaling data (also called a scaling channel) that accounts for different cluster sizes and non-uniform lighting conditions. The scaling channel also provides an additive bias that is incorporated into the feature maps generated from the image channel. This additive bias contributes to base calling accuracy because it is based on the average intensity of the central cluster pixel(s) encoded per pixel in the scaling channel. Supplementary input: Cluster center coordinates

[0441] In some implementations, location / position information 7916 (e.g., x, y coordinates) of the cluster center(s) identified from the output of the neural network-based template generator 1512 is provided as a supplemental input to the neural network 7906. Supplementary input: Cluster attribute information

[0442] In some implementations, the neural network 7906 receives cluster attribute information as supplemental input that classifies which pixels or subpixels are background pixels or subpixels, cluster center pixels or subpixels, and cluster / cluster interior pixels or subpixels that indicate / contribute to / belong to the same cluster. In other implementations, attenuation maps, binary maps, and / or ternary maps, or variations thereof, are provided as supplemental inputs to the neural network 7906. Pre-processing: Intensity correction

[0443] In some embodiments, the input data 7904 does not include a range channel, but instead the neural network 7906 receives as input modified image data that is modified based on the output of the neural network-based template generator 1512, i.e., the output of the attenuation map, the binary map, and / or the ternary map. In such embodiments, the intensity of the image data 7902 is modified to account for the absent range channel.

[0444] In other embodiments, the image data 7902 is subjected to one or more lossless transformation operations (e.g., convolution, deconvolution, Fourier transform) and the resulting modified image data is provided as input to the neural network 7906. Network Structure and Topology

[0445] The neural network 7906 is also referred to herein as a "neural network-based base caller" 1514. In one embodiment, the neural network-based base caller 1514 is a Multilayer Perceptron (MLP). In another embodiment, the neural network-based base caller 1514 is a feed-forward neural network. In yet another embodiment, the neural network-based base caller 1514 is a fully connected neural network. In a further embodiment, the neural network-based base caller 1514 is a fully convolutional neural network. In yet a further embodiment, the neural network-based base caller 1514 is a semantic segmentation neural network.

[0446] In one embodiment, the neural network-based base caller 1514 is a convolutional neural network (CNN) with multiple convolutional layers. In another embodiment, it is a recurrent neural network (RNN) such as a long short-term memory network (LSTM), a bidirectional LSTM (Bi-LSTM), or a gated recurrent unit (GRU). In yet another embodiment, it includes both a CNN and an RNN.

[0447] In yet another embodiment, the neural network-based base caller 1514 can use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or expanded convolution, transposed convolution, depth-separable convolution, pointwise convolution, 1x1 convolution, group convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It can use one or more loss functions such as logistic regression / logarithmic loss, multi-class cross-entropy / softmax loss, binary cross-entropy loss, mean squared error loss, L1 loss, L2 loss, smoothed L1 loss, and Huber loss. It can use any parallel, efficient, and compression schemes such as TFRecord, compression encoding (e.g., PNG), sharpening, parallel calls to map transforms, batching, prefetching, model parallelism, data parallelism, and synchronous / asynchronous SGD. This includes nonlinear transformation functions such as upsampling layers, downsampling layers, recurrent connections, gates and gated memory units (such as LSTM or GRU), residual blocks, residual connections, highway connections, skip connections, Pejol connections, activation functions (e.g., nonlinear transformation functions such as rectified linear unit (ReLU), leaky ReLU, exponential linear unit (ELU), sigmoid, and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout, pooling layers (e.g., max or mean pooling), global mean pooling layers, and attention mechanisms.

[0448] The neural network-based base caller 1514 processes input data 7904 and generates alternate representations 7908 of the input data 7904. The alternate representations 7908 are convolutional representations in some embodiments and hidden representations in other embodiments. The alternate representations 7908 are then processed by an output layer 7910 to generate output 7912. The output 7912 is used to generate base call(s), as described below. (output)

[0449] In one embodiment, the neural network-based base caller 1514 outputs a base call for a single target cluster in a particular sequencing cycle. In another embodiment, it outputs a base call for each target cluster in a plurality of target clusters in a particular sequencing cycle. In yet another embodiment, it outputs a base call for each target cluster in a plurality of target clusters in each sequencing cycle in the plurality of sequencing cycles, thereby generating a base call sequence for each target cluster. Distance Channel Calculation

[0450] The discussion herein describes how appropriate location / position information (e.g., x-y coordinates) of the cluster center(s) is obtained for use in calculating the distance values ​​of the distance channel. Coordinate downscale

[0451] FIG. 80 is one implementation of converting the location / position information of cluster centers identified from the output of the neural network based template generator 1512 from the sub-pixel domain to the pixel domain.

[0452] The cluster center location / position information is used at least (i) to construct input data by extracting image patches from the sequencing image 108 that contain the centers of target clusters to be base called at the center pixel, (ii) to construct distance channels that identify the distances of the centers of pixels of the image patches from the centers of the target clusters in which the center pixel is included, and / or (iii) as supplemental input 7916 to the neural network-based base caller 1514 for neural network-based base calling.

[0453] In some implementations, cluster center location / position information is identified from the output of the neural network-based template generator 1512 in upsampled sub-pixel resolution. However, in some implementations, the neural network-based base caller 1514 operates on image data that is in optical pixel resolution. Thus, in one implementation, the cluster center location / position information is converted to the pixel domain by downscaling the coordinates of the cluster centers by the same upsampling factor used to upsample the image data provided as input to the neural network-based template generator 1512.

[0454] For example, consider that the image patch data provided as input to the neural network-based template generator 1512 is derived by upsampling the sequencing image 108 from several initial sequencing cycles by an upsampling factor f. Then, in one embodiment, the coordinates of the cluster centers 8002 generated by the neural network-based template generator 1512 and stored in the template / template image 8004 by the post processor 1814 are divided by f (the divisor). These downscaled cluster center coordinates are referred to herein as "reference cluster centers" 8008 and are stored in the template / template image 8004. In one embodiment, the downscaling is performed by a downscaler 8006. Coordinate conversion

[0455] Figure 81 shows one implementation that uses cycle-specific and image channel-specific transformations to derive so-called "transformed cluster centers" 8104 from reference cluster centers 8008. The motivation for doing so will first be explained.

[0456] Sequencing images taken at different sequencing cycles are misaligned and have random translational offsets from each other. This occurs due to the finite precision of the sensor's motion stage movement and because images taken at different image / frequency channels have different optical paths and wavelengths. As a result, there is an offset between the reference cluster center and the location / position of the cluster center in the sequencing image. This offset varies between images captured at different sequencing cycles and within images captured at the same sequencing cycle but in different image channels.

[0457] To account for this offset, cycle-specific and image channel-specific transformations are applied to the reference cluster centers to generate respective transformed cluster centers for the image patches of each sequencing cycle. The cycle-specific and image channel-specific transformations are derived by an image registration process that uses image correlation to determine a full six-parameter affine transformation (e.g., translation, rotation, scaling, shear, right reflection, left reflection) or Procrustes transformation (e.g., translation, rotation, scaling, optionally extended to aspect ratio), additional details of which can be found in Appendices 1, 2, 3, and 4.

[0458] For example, consider that the reference cluster centers for the four cluster centers are (x1, y1); (x2, y2); (x3, y3); (x4, y4), and the sequencing operation uses two-channel chemistry where a red image and a green image are generated in each sequencing cycle. Then, for example, for sequencing cycle 3, the cycle-specific and image channel-specific transformations are

number

number

[0459] Similarly, for example, for sequencing cycle 9, the cycle-specific and image channel-specific transforms are

number

number

[0460] Then, sequencing cycle 3

number

number

number

number

[0461] Similarly, sequencing cycle 9

number

number

number

number

[0462] In one embodiment, the conversion is performed by converter 8102 .

[0463] The transformed cluster centers 8104 are stored in the template / template image 8004 and (i) used for patch extraction from the corresponding sequencing image 108 (e.g., by patch extractor 8106) and (ii) used to compute the distance equation

number

[0464] Figure 82 shows an image patch 8202 that is part of the input data provided to the neural network-based base caller 1514. The input data includes an array of image patch sets per cycle that are generated for a series of sequencing cycles of a sequencing operation. Each image patch set per cycle in the array has an image patch for a respective one of one or more image channels.

[0465] For example, consider a sequencing operation using two-channel chemistry that produces red and green images at each sequencing cycle, and the input data includes data spanning a series of three sequencing cycles of the sequencing operation: the current (time t) sequencing cycle being base-called, the previous (time t-1) sequencing cycle, and the next (time t+1) sequencing cycle.

[0466] The input data then includes the following arrangement of image patch sets for each cycle: a current cycle image patch set having a current red image patch and a current green image patch, respectively extracted from the red and green sequencing images captured in the current sequencing cycle; a previous cycle image patch set having a previous red image patch and a previous green image patch, respectively extracted from the red and green sequencing images captured in the previous sequencing cycle; and a next cycle image patch set having a next red image patch and a next green image patch, respectively extracted from the red and green sequencing images captured in the next sequencing cycle.

[0467] The size of each image patch can be n x n, where n can be any number ranging from 1 to 10,000. Each image patch can be in the optical pixel domain or an upsampled subpixel domain. In the embodiment shown in Figure 82, the extracted image page 8202 has pixel intensity data for pixels covering / representing multiple clusters 1-m and their surrounding background. Also, in the illustrated embodiment, the image patch 8202 is extracted in such a way that the center of the target cluster contained within its center pixel is base called.

[0468] In Figure 82, pixel centers are indicated by black rectangles and have integer location / position coordinates, and cluster centers are indicated by purple circles and have floating point location / position coordinates. (Distance calculation for a single target cluster)

[0469] Figure 83 shows one implementation of determining distance values ​​8302 for the distance channel when a single target cluster is being base called by the neural network-based base caller 1514. The center of the target cluster is contained in the center pixel of the image patch that is provided as input to the neural network-based base caller 1514. Distance values ​​are calculated for each pixel, so that for each pixel, the distance between its center and the center of the target cluster is determined. Thus, a distance value is calculated for each pixel in each of the image patches that are part of the input data.

[0470] Figure 83 shows three distance values ​​d1, dc, and dn for a particular image patch. In one implementation, the distance value 8302 is calculated using the following distance formula:

number

[0471] In other embodiments, when the image patch is upsampled to a sub-pixel resolution, the distance value 8302 is calculated in the sub-pixel domain.

[0472] Thus, in a single target cluster base calling embodiment, the distance channel is calculated only for the target cluster being base called.

[0473] Figure 84 shows one implementation of encoding 8402 the distance value 8302 calculated between the pixel and the target cluster per pixel. In one implementation, in the input data, the distance value 8302 as part of the distance channel supplements each corresponding image channel (image patch) as "pixel distance data." Returning to the example of red and green images being generated per sequencing cycle, the input data includes a red distance channel and a green distance channel that supplement the red and green image channels, respectively, as pixel distance data.

[0474] In another implementation, when the image patch is upsampled to a sub-pixel resolution, the distance channel is encoded on a sub-pixel basis. (Distance calculation for multiple target clusters)

[0475] Figure 85a shows one embodiment of determining distance values ​​8502 for the distance channel when multiple target clusters 1-m are being simultaneously basecalled by the neural network-based base caller 1514. Distance values ​​are calculated on a pixel-by-pixel basis, so that for each pixel, the distance between its center and the respective center of each of the multiple clusters 1-m is determined, and the smallest distance value (in red) is assigned to the pixel.

[0476] Thus, the distance channel identifies the center-to-center distance of each pixel from the closest one of the clusters, which is selected based on the center-to-center distance between the pixel and each of the clusters. In the illustrated embodiment, Figure 85a shows pixel-to-cluster center-to-cluster distances for two pixels and four cluster centers. Pixel 1 is closest to cluster 1, and pixel n is closest to cluster 3.

[0477] In one embodiment, the distance value 8502 is calculated using the following distance formula:

number

[0478] In other embodiments, when the image patch is upsampled to a sub-pixel resolution, the distance value 8502 is calculated in the sub-pixel domain.

[0479] Thus, in multi-cluster base calling implementations, distance channels are calculated for the closest cluster among multiple clusters.

[0480] FIG. 85b shows several nearest pixels for each of target clusters 1-m, determined based on pixel center-to-nearest cluster center distance 8504 (d1, d2, d23, d29, d24, d32, dn, d13, d14, etc.).

[0481] Figure 86 shows one implementation for each pixel that encodes 8602 the minimum distance value calculated between the pixel and the closest one of the clusters. In another implementation, when the image patch is in an upsampled sub-pixel resolution, the distance channel is encoded for each sub-pixel. (Distance calculation for multiple target clusters based on cluster shape)

[0482] Figure 87 shows one embodiment in which classifications / attributes / classifications 8702 between pixel clusters, referred to herein as "cluster shape data" or "cluster shape information," are used to determine cluster distance values ​​8802 in the distance channel when multiple target clusters 1-m are being simultaneously base called by a neural network-based base caller 1514. First, the following is a brief discussion of how cluster shape data is generated.

[0483] As described above, the output of the neural network-based template generator 1512 is used to classify pixels as background pixels, center pixels, and cluster / intra-cluster pixels that indicate / contribute to / belong to the same cluster. This pixel inter-cluster classification information is used to assign each pixel to only one cluster, regardless of the distance between the pixel center and the cluster center, and is stored as cluster shape data.

[0484] In the embodiment shown in Figure 87, background pixels are colored gray, pixels belonging to cluster 1 are colored yellow (cluster 1 pixels), pixels belonging to cluster 2 are colored green (cluster 2 pixels), pixels belonging to cluster 3 are colored red (cluster 3 pixels), and pixels belonging to cluster m are colored blue (cluster m pixels).

[0485] Figure 88 shows one embodiment for using cluster shape data to calculate distance values ​​8802. First, we explain why distance information calculated without considering cluster shape is prone to error. Then, we explain how cluster shape data overcomes this limitation.

[0486] In "multi-cluster" base calling embodiments that do not use cluster shape data (Figures 85a-b and 86), a pixel's center-to-center distance value is calculated relative to the closest cluster among multiple clusters. Now consider a scenario when a pixel that belongs to cluster A is further away from the center of cluster A, but closer to the center of cluster B. In such a case, without cluster shape data, the pixel would be assigned a distance value calculated relative to cluster B (to which it does not belong), instead of being assigned a distance value toward cluster A (to which it truly belongs).

[0487] The "multi-cluster shape-based" base calling implementation avoids this by using true pixel-cluster to-pixel cluster mappings as defined in the raw image data and generated by the neural network-based template generator 1512.

[0488] The contrast between the two embodiments can be seen with respect to pixels 34 and 35. In Figure 85b, the distance values ​​of pixels 34 and 35 are calculated with respect to the nearest center of cluster 3, without considering the cluster shape data. However, in Figure 88, based on the cluster shape data, the distance value 8802 of pixels 34 and 35 is calculated with respect to cluster 2 (to which they actually belong).

[0489] In Figure 88, the cluster pixels indicate cluster intensity and the background pixels indicate background intensity. The cluster distance value identifies the center-to-center distance of each cluster pixel from an assigned one of the clusters selected based on classifying each cluster pixel into only one of the clusters. In some implementations, the background pixels are assigned a predetermined background distance value, such as 0 or 0.1, or some other minimum value.

[0490] In one embodiment, as described above, the cluster distance value 8802 is calculated using the following distance formula:

number

[0491] In another embodiment, when the image patch is upsampled to a sub-pixel resolution, the cluster distance values ​​8802 are calculated in the sub-pixel domain, and the cluster and background attributes 8702 occur on a sub-pixel basis.

[0492] Thus, in multi-cluster shape-based base calling implementations, distance channels are calculated for assigned clusters from among multiple clusters, where the assigned clusters are selected based on classifying each cluster pixel into only one of the clusters according to a true pixel-to-cluster mapping defined in the raw image data.

[0493] Figure 89 shows one implementation of per-pixel encoding of the distance value 8702 calculated between the pixel and the assigned cluster. In another implementation, when the image patch is in upsampled sub-pixel resolution, the distance channel is encoded per sub-pixel.

[0494] Deep learning is a powerful machine learning technique that uses multilayer neural networks. One particularly successful network architecture within the computer vision and image processing domains is the convolutional neural network (CNN), where each layer performs a feedforward convolutional transformation from an input tensor (image-like, multidimensional dense array) to an output tensor of a different shape. CNNs are particularly well-suited for image-like inputs due to the spatial coherence of images and the emergence of general-purpose graphics processing units (GPUs), which make training fast on arrays up to 3- or 4-D. Leveraging these image-like properties leads to superior empirical performance compared to other learning methods, such as support vector machines (SVMs) or multilayer perceptrons (MLPs).

[0495] We introduce a specialized architecture that augments standard CNNs to handle both image data and complementary distance and scaling data, as detailed below. (Special structure)

[0496] Figure 90 shows one embodiment of a dedicated architecture for the neural network-based base caller 1514 that is used to separate the processing of data in different sequencing cycles. The motivation for using a dedicated architecture is first explained.

[0497] As described above, the neural network-based base caller 1514 processes data from the current sequencing cycle, one or more preceding sequencing cycles, and one or more subsequent sequencing cycles. Data from additional sequencing cycles provides sequence-specific context. The neural network-based base caller 1514 learns the sequence-specific context during training and calls bases based on it. Additionally, data from pre- and post-sequencing cycles provide secondary contributions of pre-phasing and phasing signals to the current sequencing cycle. (spatial convolutional layer)

[0498] However, as mentioned above, images captured in different sequencing cycles and in different image channels are misaligned and have residual registration errors with each other. To account for this misalignment, the dedicated architecture includes spatial convolution layers that do not mix information between sequencing cycles, but only mix information within a sequencing cycle.

[0499] The spatial convolutional layer uses so-called "decoupled convolutions" that operate on decoupling by processing the data for each of multiple sequencing cycles independently through a "dedicated, non-shared" array of convolutions. Decoupled convolutions convolve on the data and resulting feature maps only within a given sequencing cycle, i.e., the cycle, without convolving on the data and resulting feature maps of any other sequencing cycles.

[0500] For example, consider the input data as including (i) current data for the current (time t) sequencing cycle to be base-called, (ii) previous data for the previous (time t-1) sequencing cycle, and (iii) next data for the next (time t+1) sequencing cycle. The dedicated architecture then initiates three separate data processing pipelines (or convolution pipelines): a current data processing pipeline, a previous data processing pipeline, and a next data processing pipeline. The current data processing pipeline receives the current data for the current (time t) sequencing cycle as input and processes it independently through multiple spatial convolution layers to generate a so-called "current spatial convolutional representation" as the output of the final spatial convolutional layer. The previous data processing pipeline receives the previous data for the previous (time t-1) sequencing cycle as input and processes it independently through multiple spatial convolutional layers to generate a so-called "previous spatial convolutional representation" as the output of the final spatial convolutional layer. The next data processing pipeline receives the next data for the next (time t+1) sequencing cycle as input and processes it independently through multiple spatial convolutional layers to produce the so-called “next spatially convolved representation” as the output of the final spatial convolutional layer.

[0501] In some embodiments, the current, previous, and next processing pipelines execute in parallel.

[0502] In some implementations, the spatial convolutional layer is part of a spatial convolutional network (or sub-network) within a dedicated structure. (Temporal convolutional layer)

[0503] The neural network-based base caller 1514 further includes temporal convolutional layers that blend information between sequencing cycles, i.e., between cycles. The temporal convolutional layers receive their inputs from the spatial convolutional networks and operate on the spatial convolutional representations produced by the final spatial convolutional layer for each data processing pipeline.

[0504] The inter-cycle operational freedom of the temporal convolutional layers arises from the fact that misalignment features present in the image data fed as input to the spatial convolutional network are purged from the spatial convolutional representation by the cascade of decoupled convolutions performed by the array of spatial convolutional layers.

[0505] The temporal convolutional layer uses so-called "combinatorial convolution," which convolves group-wise over input channels with successive inputs on a sliding window basis. In one embodiment, the successive inputs are successive outputs generated by a previous spatial convolutional layer or a previous temporal convolutional layer.

[0506] In some embodiments, the temporal convolutional layer is part of a temporal convolutional network (or sub-network) within a dedicated structure. The temporal convolutional network receives its input from a spatial convolutional network. In one embodiment, the first temporal convolutional layer of the temporal convolutional network combines the spatial convolutional representations between sequencing cycles by group. In another embodiment, subsequent temporal convolutional layers of the temporal convolutional network combine successive outputs of previous temporal convolutional layers.

[0507] The output of the final temporal convolutional layer is fed into an output layer that generates outputs that are used to base call one or more clusters in one or more sequencing cycles.

[0508] What follows is a more detailed discussion of decoupled and combined convolution. (separated convolution)

[0509] During forward propagation, the dedicated structure processes information from multiple inputs in two stages. In the first stage, disjunctive convolutions are used to prevent mixing of information between inputs. In the second stage, combinatorial convolutions are used to mix information between inputs. The results from the second stage are used to make a single guess for the multiple inputs.

[0510] This differs from batch-mode techniques, in which the convolutional layer processes multiple inputs in a batch simultaneously and makes a corresponding guess for each input in the batch. In contrast, dedicated structures map multiple inputs to a single guess, which may include two or more predictions, such as classification scores for each of the four bases (A, C, T, and G).

[0511] In one embodiment, the inputs have a temporal ordering such that each input is generated at a different time step and has multiple input channels. For example, the multiple inputs may include three inputs: a current input generated by a current sequencing cycle at time step (t), a previous input generated by a previous sequencing cycle at time step (t-1), and a next input generated by a next sequencing cycle at time step (t+1). In another embodiment, each input is derived from the current, previous, and next inputs by one or more previous convolutional layers, respectively, and includes k feature maps.

[0512] In one implementation, each input may include five input channels: a red image channel (red), a red distance channel (yellow), a green image channel (green), a green distance channel (purple), and a scaling channel (blue). In another implementation, each input may include k feature maps generated by a previous convolutional layer, with each feature map treated as an input channel.

[0513] Figure 91 shows one implementation of decoupled convolution. Decoupled convolution processes multiple inputs at once by applying a convolution filter to each input in parallel. In decoupled convolution, a convolution filter combines input channels within the same input and does not combine input channels within different inputs. In one implementation, the same convolution filter is applied to each input in parallel. In another implementation, a different convolution filter is applied to each input in parallel. In some implementations, each spatial convolution layer includes a bank of k convolution filters, each of which is applied to each input in parallel. (Combinatorial Folding)

[0514] Combinatorial convolution blends information between different inputs by grouping corresponding input channels of the different inputs and applying a convolution filter to each group. The grouping of corresponding input channels and application of the convolution filter occurs on a sliding window basis. In this context, a window spans two or more consecutive input channels, e.g., representing the output for two consecutive sequencing cycles. Because the windows are sliding windows, most input channels are used in two or more windows.

[0515] In some implementations, the distinct inputs originate from an output array generated by a preceding spatial or temporal convolutional layer, where the distinct inputs are arranged as successive outputs and are therefore viewed as successive inputs by the next temporal convolutional layer, where a combinatorial convolution then applies a convolutional filter to groups of corresponding input channels in the successive inputs.

[0516] In one embodiment, the successive inputs have a temporal ordering such that: the current input is generated by the current sequencing cycle at time step (t), the previous input is generated by the previous sequencing cycle at time step (t-1), and the next input is generated by the next sequencing cycle at time step (t+1). In another embodiment, each successive input is derived from the current, previous, and next inputs by one or more previous convolutional layers, respectively, and includes k feature maps.

[0517] In one implementation, each input may include five input channels: a red image channel (red), a red distance channel (yellow), a green image channel (green), a green distance channel (purple), and a scaling channel (blue). In another implementation, each input may include k feature maps generated by a previous convolutional layer, with each feature map treated as an input channel.

[0518] The depth B of the convolution filter depends on the number of consecutive inputs whose corresponding input channels are convolved with the convolution filter on a sliding window basis for each group. In other words, the depth B is equal to the number of consecutive inputs in each sliding window and group size.

[0519] In Figure 92a, corresponding input channels from two consecutive inputs are combined within each sliding window, so B = 2. In Figure 92b, corresponding input channels from three consecutive inputs are combined within each sliding window, so B = 3.

[0520] In one embodiment, the sliding windows share the same convolutional filter. In another embodiment, a different convolutional filter is used for each sliding window. In some embodiments, each temporal convolutional layer includes a bank of k convolutional filters, each of which is applied to successive inputs on a sliding window basis. (filter bank)

[0521] Figure 93 shows one implementation of the convolutional layers of the neural network-based base caller 1514, where each convolutional layer has a bank of convolutional filters. Five convolutional layers are shown in Figure 93, each having a bank of 64 convolutional filters. In some implementations, each spatial convolutional layer has a bank of k convolutional filters, where k can be any number, such as 1, 2, 8, 64, 128, 256, etc. In some implementations, each temporal convolutional layer has a bank of k convolutional filters, where k can be any number, such as 1, 2, 8, 64, 128, 256, etc.

[0522] The discussion here describes the supplemental scaling channel and how it is calculated. (scaling channel)

[0523] Figure 94 shows two configurations of a scaling channel that complements the image channel. The scaling channel is coded per pixel in the input data provided to the neural network-based basis caller 1514. Different cluster sizes and uneven lighting conditions result in a wide range of cluster intensities being extracted. The additive bias provided by the scaling channel equalizes cluster intensities across clusters. In another implementation, when the image patch is upsampled to a sub-pixel resolution, the scaling channel is coded per sub-pixel.

[0524] When a single target cluster is being base called, the scaling channel assigns the same scaling value to all pixels. When multiple target clusters are being base called simultaneously, the scaling channel assigns different scaling values ​​to groups of pixels based on the cluster shape data.

[0525] The scaling channel 9410 has the same scaling value (s1) for all pixels. The scaling value (s1) is based on the average intensity of the central pixel that comprises the center of the target cluster. In one embodiment, the average intensity is calculated by averaging the intensity values ​​of the central pixel observations during two or more prior sequencing cycles that generated A and T base calls for the target cluster.

[0526] The scaling channel 9408 has a different scaling value (s1, s2, s3, sm) for each pixel group that belongs to the corresponding cluster based on the cluster shape data. Each pixel group includes a central cluster pixel that contains the center of the corresponding cluster. The scaling value for a particular pixel group is based on the average intensity of that central cluster pixel. In one embodiment, the average intensity is calculated by averaging the intensity values ​​of the central cluster pixel observations during two or more prior sequencing cycles that generated A and T base calls for the corresponding cluster.

[0527] In some implementations, background pixels are assigned a background scaling value (sb), which may be 0 or 0.1, or some other minimum value.

[0528] In one embodiment, the scaling channels 9406 and their scaling values ​​are determined by an intensity scaler 9404. The intensity scaler 9404 uses cluster intensity data 9402 from the previous sequencing cycle to calculate the average intensity.

[0529] In other implementations, the supplemental scaling channel may be provided as input in different ways, such as before or at the last layer of the neural network-based base caller 1514, before or at one or more intermediate layers of the neural network-based base caller 1514, and as a single value instead of encoding it per pixel to match the image size.

[0530] The discussion here describes the input data provided to the neural network-based base caller 1514. Input data: image channel, distance channel, and scaling channel

[0531] 95a shows one embodiment of input data 9500 for a single sequencing cycle that produces red and green images. The input data 9500 includes: Red intensity data 9502 (red) for pixels in an image patch extracted from the red image. The red intensity data 9502 is encoded in the red image channel. Red distance data 9504 (yellow) that complements the red intensity data 9502 on a pixel-by-pixel basis. The red distance data 9504 is encoded in the red distance channel. Green intensity data 9506 (green) for pixels in an image patch extracted from the green image. The green intensity data 9506 is encoded in the green image channel. Green distance data 9508 (purple) that complements the green intensity data 9506 on a pixel-by-pixel basis. The green distance data 9508 is coded in the green distance channel. Scaling data 9510 (blue) that compensates for the red intensity data 9502 and the green intensity data 9506 on a pixel-by-pixel basis. The scaling data 9510 is coded in a scaling channel.

[0532] In other embodiments, the input data may include a fewer or greater number of image channels and supplemental distance channels. In one example, for a sequencing operation using four-channel chemistry, the input data includes four image channels for each sequencing cycle and four supplemental distance channels.

[0533] The discussion here explains how the distance and scaling channels contribute to base calling accuracy. (additive bias)

[0534] Figure 95b shows one implementation of a distance channel that provides an additive bias that is incorporated into feature maps generated from the image channel. This additive bias is based on the pixel-center-cluster center(s) distance encoded for each pixel in the distance channel, and therefore contributes to base calling accuracy.

[0535] On average, approximately 3x3 pixels contain one cluster. The density at the center of the cluster is expected to be higher than the periphery because the cluster essentially grows outward from the central location. The periphery cluster pixels may contain competing signals from nearby clusters. Therefore, the central cluster pixel is considered the region of greatest intensity and acts as a beacon to reliably identify the cluster.

[0536] The pixels of an image patch represent the intensity emissions of multiple clusters (e.g., 10-200 clusters) and their surrounding background. The additional clusters contribute to base call prediction by incorporating information from a wider radius and identifying the base bases whose intensity emissions are represented in the image patch. In other words, the intensity emissions from groups of clusters cumulatively produce an intensity pattern that can be assigned to distinct bases (A, C, T, or G).

[0537] We observe that explicitly communicating each pixel's convolution filter distance from the cluster center(s) in the supplemental distance channel results in higher base calling accuracy. The distance channel tells the convolution filter which pixels contain the cluster center and which pixels are farther away from the cluster center. The convolution filter uses this information to assign sequencing signals to their appropriate source clusters by addressing (a) the central cluster pixels, their neighboring pixels, and the feature maps derived from them more than (b) the surrounding cluster pixels, background pixels, and the feature maps derived from them. In one example of addressing, the distance channel provides a positive additive bias that is incorporated into the feature maps resulting from (a) but a negative additive bias that is incorporated into the feature maps resulting from (b).

[0538] The range channel has the same dimensionality as the image channel, which allows the convolution filter to estimate the image and range channels separately within the local receptive field and then coherently combine the estimates.

[0539] When a single target cluster is being base called, the distance channel identifies only one central cluster pixel at the center of the image patch. When multiple target clusters are being base called simultaneously, the distance channel identifies multiple central cluster pixels distributed across the image patch.

[0540] The "single cluster" distance channel is applied to an image patch containing the center of a single target cluster that is base called at its center pixel. The single cluster distance channel contains the center-to-center distance of each pixel in the image patch to the single target cluster. In this embodiment, the image patch also contains additional clusters adjacent to the single target cluster, but the additional clusters are not base called.

[0541] The "multi-cluster" distance channel is applied to an image patch containing the centers of multiple target clusters, each of which is base-called at its respective center cluster pixel. The multi-cluster distance channel contains the center-to-center distance of each pixel in the image patch to the nearest cluster among the multiple target clusters. This has the potential, but unlikely, to measure the center-to-center distance to the wrong cluster.

[0542] The "multi-cluster shape-based" distance channel is applied to an image patch containing the centers of multiple target clusters base-called at their respective center cluster pixels, and for which pixel-cluster attribute information is known. The multi-cluster distance channel contains the center-to-center distance of each cluster pixel in the image patch to the cluster to which it belongs or to which it belongs among multiple target clusters. Background pixels can be flagged as background instead of given a calculated distance.

[0543] Figure 95b also shows one implementation of a scaling channel that provides an additive bias that is incorporated into the feature maps generated from the image channel. This additive bias contributes to base calling accuracy because it is based on the average intensity of the central cluster pixel(s) encoded per pixel in the scaling channel. The considerations regarding additive bias in the context of the distance channel apply analogously to the scaling channel. (Example of additive bias)

[0544] FIG. 95b further illustrates an example of how additive biases are derived from the distance and scaling channels and incorporated into feature maps generated from the image channel.

[0545] In Figure 95b, convolution filter i 9514 evaluates the local receptive field 9512 (magenta) across two image channels 9502 and 9506, two distance channels 9504 and 9508, and a scaling channel 9510. Because the distance and scaling channels are coded separately, an additive bias arises when the intermediate outputs 9516a-e of each of the channel-specific convolution kernels (or feature detectors) 9516a-e (plus bias 9516f) are accumulated 9518 per channel as the final output / feature map element 9520 for the local receptive field 9512. In this example, the additive biases provided by the two distance channels 9504 and 9508 are intermediate outputs 9516b and 9516d, respectively. The additive bias provided by the scaling channel 9510 is intermediate output 9516e.

[0546] The additive bias guides the feature map compilation process by placing more weight on those features within the image channel that are deemed more important and reliable for base calling, i.e., the pixel intensities of the central cluster pixels and their neighboring pixels. During training, backpropagation of gradients computed from comparisons with ground truth base calls updates the weights of the convolution kernel to generate stronger activations for the central cluster pixels and their neighboring pixels.

[0547] For example, consider that a pixel within a group of adjacent pixels covered by local receptive field 9512 comprises a cluster center, and then distance channels 9504 and 9508 reflect the pixel's proximity to the cluster center. As a result, when intensity intermediate outputs 9516a and 9516c are merged with distance channel additive biases 9516b and 9516d in per-channel accumulation 9518, the result is a positively biased convolved representation 9520 of the pixel.

[0548] In contrast, if the pixels covered by the local receptive field 9512 are not near the cluster center, the distance channels 9504 and 9508 reflect their separation from the cluster center. As a result, when the intensity intermediate outputs 9516a and 9516c are merged with the distance channel additive biases 9516b and 9516d in the per-channel accumulation 9518, the result is a negatively biased convolved representation 9520 of the pixel.

[0549] Similarly, a scaling channel additive bias 9516e derived from the scaling channel 9510 may bias the convolved representation 9520 of the pixel positively or negatively.

[0550] For clarity, Figure 95b shows the application of a single convolution filter i 9514 to input data 9500 in a single sequencing cycle. Those skilled in the art will understand that the discussion can be extended to multiple convolution filters (e.g., a filter bank of k filters, where k can be 8, 16, 32, 64, 128, 256, etc.), multiple convolution layers (e.g., multiple spatial and temporal convolution layers), and multiple sequencing cycles (e.g., t, t+1, t-1).

[0551] In another embodiment, because the distance and scaling channels and the image channel have the same dimensionality, the distance and scaling channels are applied directly to the image channel instead of being coded separately to generate a modulated pixel multiplication. In a further embodiment, the weights of the convolution kernel are determined based on the distance and image channel to detect the most important features in the image channel during element-wise multiplication. In yet another embodiment, instead of being fed to the first layer, the distance and scaling channels are provided as auxiliary inputs to downstream layers and / or networks (e.g., fully connected networks or classification layers). In yet a further embodiment, the distance and scaling channels are fed to the first layer and re-fed to downstream layers and / or networks (e.g., via residual connections).

[0552] The above discussion is for 2D input data with k input channels. Extensions to 3D inputs will be understood by those skilled in the art. Briefly, the volumetric input is a 4D tensor with dimensions k×l×w×h, where l is an additional dimension, the length. Each individual kernel is a 4D tensor swept by a 4D tensor, resulting in a 3D tensor (the channel dimensions collapse because they are not swept all the way through).

[0553] In another embodiment, when the input data 9500 is in upsampled sub-pixel resolution, the distance and scaling channels are coded separately for each sub-pixel, and the additive bias occurs at the sub-pixel level. (Base calling using dedicated structure and input data)

[0554] The discussion here describes how the custom structure and input data are used for neural network-based base calling. (single cluster base call)

[0555] Figures 96a, 96b, and 96c show one embodiment for base calling a single target cluster. A dedicated architecture processes input data for three sequencing cycles: the current (time t) sequencing cycle being base called, the previous (time t-1) sequencing cycle, and the next (time t+1) sequencing cycle, and generates base calls for a single target cluster in the current (time t) sequencing cycle.

[0556] Figures 96a and 96b show spatial convolutional layers. Figure 96c shows temporal convolutional layers along with several other non-convolutional layers. In Figures 96a and 96b, vertical dotted lines demarcate the spatial convolutional layers from the feature maps, and horizontal dashed lines demarcate the three convolutional pipelines corresponding to the three sequencing cycles.

[0557] For each sequencing cycle, the input data includes a tensor of dimensionality n×n×m (e.g., input tensor 9500 in FIG. 95a), where n represents the width and height of the square tensor and m represents the number of input channels, making the dimensionality of the input data for three cycles n×n×m×t.

[0558] Here, each tensor per cycle contains the center of a single target cluster at the center pixel of that image channel. It also shows the intensity emission of the single target cluster, of several neighboring clusters, and of their surrounding background, captured in each of the image channels in a particular sequencing cycle. In Figure 96a, two example image channels are shown: a red image channel and a green image channel.

[0559] Each tensor per cycle also includes a distance channel that complements the corresponding image channel (e.g., a red distance channel and a green distance channel). The distance channel identifies the center-to-center distance of each pixel in the corresponding image channel to a single target cluster. Each tensor per cycle further includes a scaling channel that scales the intensity values ​​for each pixel in each of the image channels.

[0560] The dedicated structure has five spatial convolutional layers and two temporal convolutional layers. Each spatial convolutional layer applies decoupled convolution using a bank of k convolutional filters of dimensionality j×j×∂, where j represents the width and height of the square filter and ∂ represents its depth. Each temporal convolutional layer applies combinatorial convolution using a bank of k convolutional filters of dimensionality j×j×α, where j represents the width and height of the square filter and α represents its depth.

[0561] The specialized architecture includes a pre-classification layer (e.g., a flattening layer and a densification layer) and an output layer (e.g., a softmax classification layer). The pre-classification layer prepares the input for the output layer, which generates base calls for a single target cluster in the current (time t) sequencing cycle. (Consistent reduction of spatial dimensionality)

[0562] Figures 96a, 96b, and 96c also show the resulting feature maps (convolutional representations or intermediate convolutional representations or convolutional features or activation maps) generated by the convolutional filters. Starting from the tensors per cycle, the spatial dimensionality of the resulting feature maps is reduced by a constant step size from one convolutional layer to the next, a concept referred to herein as "consistent reduction of spatial dimensionality." In Figures 96a, 96b, and 96c, an exemplary constant step size of 2 is used for consistent reduction of spatial dimensionality.

[0563] Consistent spatial dimensionality reduction is expressed by the following formula: Current Feature Map Spatial Dimensionality = Previous Feature Map Spatial Dimensionality - Convolutional Filter Spatial Dimensionality + 1. Consistent spatial dimensionality reduction allows the convolutional filter to progressively narrow the focus of attention on the central cluster pixel and its neighboring pixels, producing a feature map with features that capture the local dependencies between the central cluster pixel and its neighboring pixels. This, in turn, helps to accurately base call clusters whose centers are contained in the central cluster pixel.

[0564] The separated convolutions of the five spatial convolution layers prevent mixing of information between the three sequencing cycles and maintain three separate convolution pipelines.

[0565] The combined convolution of the two temporal convolutional layers blends information across three sequencing cycles. The first temporal convolutional layer convolves on the next and current spatial convolutional representations generated by the final spatial convolutional layer for the next and current sequencing cycles, respectively. This results in a first temporal output. The first temporal convolutional layer also convolves on the current and previous spatial convolutional representations generated by the final spatial convolutional layer for the current and previous sequencing cycles, respectively. This results in a second temporal output. The second temporal convolutional layer convolves on the first and second temporal outputs to generate a final temporal output.

[0566] In some embodiments, the final temporal output is fed to a flattening layer to generate a flattened output. The flattened output is then fed to a densification layer to generate a densification output. The densification output is processed by an output layer to generate base calls for a single target cluster in the current (time t) sequencing cycle.

[0567] In some embodiments, the output layer generates likelihoods (classification scores) that the bases incorporated into a single target cluster in the current sequencing cycle are A, C, T, and G, and classifies the bases as A, C, T, or G based on the likelihood (e.g., the base with the greatest likelihood is selected, such as base A in Figure 96a). In such embodiments, the likelihood is an exponentially normalized score produced by the softmax classification layer, which equals 1.

[0568] In some embodiments, the output layer derives an output pair for a single target cluster. The output pair identifies a class label for whether the base incorporated into the single target cluster in the current sequencing cycle is A, C, T, or G, and base calls are made for the single target cluster based on the class label. In one embodiment, a class label of 1,0 identifies an A base, a class label of 0,1 identifies a C base, a class label of 1,1 identifies a T base, and a class label of 0,0 identifies a G base. In another embodiment, a class label of 1,1 identifies an A base, a class label of 0,1 identifies a C base, a class label of 0.5,0.5 identifies a T base, and a class label of 0,0 identifies a G base. In yet another embodiment, the class label 1,0 identifies an A base, the class label 0,1 identifies a C base, the class label 0.5,0.5 identifies a T base, and the class label 0,0 identifies a G base. In yet a further embodiment, the class label 1,2 identifies an A base, the class label 0,1 identifies a C base, the class label 1,1 identifies a T base, and the class label 0,0 identifies a G base.

[0569] In some embodiments, the output layer derives class labels for the single target cluster that identify whether the base incorporated into the single target cluster in the current sequencing cycle is A, C, T, or G, and base calls the single target cluster based on the class labels. In one embodiment, a class label of 0.33 identifies an A base, a class label of 0.66 identifies a C base, a class label of 1 identifies a T base, and a class label of 0 identifies a G base. In another embodiment, a class label of 0.50 identifies an A base, a class label of 0.75 identifies a C base, a class label of 1 identifies a T base, and a class label of 0.25 identifies a G base.

[0570] In some embodiments, the output layer derives a single output value, compares the single output value against class value ranges corresponding to bases A, C, T, and G, assigns the single output value to a particular class value range based on the comparison, and basecalls a single target cluster based on the assignment. In one embodiment, the single output value is derived using a sigmoid function, and the single output value ranges from 0 to 1. In another embodiment, a class value range of 0 to 0.25 represents A bases, a class value range of 0.25 to 0.50 represents C bases, a class value range of 0.50 to 0.75 represents T bases, and a class value range of 0.75 to 1 represents G bases.

[0571] Those skilled in the art will understand that in other embodiments, the dedicated architecture may process input data for fewer or more sequencing cycles and may include fewer or more spatial and temporal convolutional layers. The dimensionality of the input data, the tensors per cycle in the input data, the convolutional filters, the resulting feature maps, and the output may also vary. The number of convolutional filters in a convolutional layer may also vary. It may use different padding and striding configurations. It may use different classification functions (e.g., sigmoid or regression) and may or may not include fully connected layers. It may use 1D convolution, 2D convolution, 3D convolution, 4D convolution, 5D convolution, dilated or asexual convolution, transposed convolution, depth-separable convolution, 1x1 convolution, group convolution, flattened convolution, spatial and cross-channel convolution, shuffled grouped convolution, spatially separable convolution, and deconvolution. It can use one or more loss functions such as logistic regression / logarithmic loss, multiclass cross-entropy / softmax loss, binary cross-entropy loss, mean squared error loss, L1 loss, L2 loss, smoothed L1 loss, and Huber loss. It can use any parallel, efficient, and compression schemes such as TFRecords, compression encoding (e.g., PNG), sharpening, parallel calls to map transforms, batching, prefetching, model parallelism, data parallelism, and synchronous / asynchronous SGD. This includes nonlinear transformation functions such as upsampling layers, downsampling layers, recurrent connections, gates and gated memory units (such as LSTM or GRU), residual blocks, residual connections, highway connections, skip connections, Pejol connections, activation functions (e.g., nonlinear transformation functions such as rectified linear unit (ReLU), leaky ReLU, exponential linear unit (ELU), sigmoid, and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout, pooling layers (e.g., max or mean pooling), global mean pooling layers, and attention mechanisms.

[0572] While single cluster base calling has been described, the discussion here describes multiple cluster base calling. (multiple cluster base calls)

[0573] Depending on the size of the input data and the cluster density on the flow cell, anywhere from 100,000 to 300,000 clusters are simultaneously base called by the neural network-based base caller 1514 per input. Extending this to a data-parallel and / or model-parallel strategy implemented on a parallel processor and using batches or mini-batches of size 10 results in 1-3 million clusters being simultaneously base called per batch or mini-batch.

[0574] Depending on the sequencing configuration (e.g., cluster density, number of tiles on a flow cell), a tile contains 20,000 to 300,000 clusters. In another embodiment, Illumina's NovaSeq sequencer has up to 4 million clusters per tile. Thus, a sequencing image of a tile (tile image) may show 20,000 to 300,000 clusters and their intensity emanation from the surrounding background. Thus, in one embodiment, using input data containing the entire tile image results in 300,000 clusters being simultaneously base-called per input. In another embodiment, using image patches of size 15x15 pixels in the input data results in fewer than 100 clusters being simultaneously base-called per input. Those skilled in the art will understand that these numbers may vary depending on the sequencing configuration, parallelization strategy, structural details (e.g., based on optimal structural hyperparameters), and available computation.

[0575] Figure 97 shows one embodiment for simultaneously base calling multiple target clusters. The input data has three tensors for the three sequencing cycles described above. Each tensor for each cycle (e.g., input tensor 9500 in Figure 95a) represents the intensity radiation of multiple target clusters to be base called and their surrounding background captured in each of the image channels in a particular sequencing cycle. In other embodiments, several additional neighboring clusters that are not base called are also included in the context.

[0576] In a multi-cluster base calling implementation, each tensor for each cycle includes a distance channel that complements the corresponding image channel (e.g., a red distance channel and a green distance channel) and identifies the center-to-center distance of each pixel in the corresponding image channel to the nearest cluster among multiple target clusters.

[0577] In a multi-cluster shape-based base calling implementation, each tensor per cycle includes a distance channel that complements the corresponding image channel (e.g., a red distance channel and a green distance channel), identifying the center-to-center distance of each cluster pixel in the corresponding image channel to the cluster to which it belongs or to which it belongs among multiple target clusters.

[0578] Each tensor for each cycle further includes a scaling channel that scales the intensity values ​​pixel by pixel in each of the image channels.

[0579] In Figure 97, the spatial dimensionality of each tensor per cycle is greater than that shown in Figure 96a. That is, in the single target cluster base calling embodiment in Figure 96a, the spatial dimensionality of each tensor per cycle is 15 x 15, whereas in the multiple cluster base calling embodiment in Figure 97, the spatial dimensionality of each tensor per cycle is 114 x 114. According to some embodiments, having a greater amount of pixelated data representing the intensity emissions of additional clusters improves the accuracy of base calls predicted for multiple clusters simultaneously. (Avoiding redundant convolutions)

[0580] Furthermore, the image channels in each tensor per cycle are obtained from image patches extracted from the sequencing image. In some implementations, there are overlapping pixels between extracted image patches that are spatially contiguous (e.g., contiguous left, right, top, and bottom). Thus, in one implementation, the overlapping pixels do not undergo redundant convolutions, and the results from previous convolutions are reused in subsequent instances when the overlapping pixels are part of the subsequent input.

[0581] For example, consider a first image patch of size n×n pixels extracted from a sequencing image, and a second image patch of size m×m pixels also extracted from the same sequencing image, such that the first and second image patches are spatially contiguous and share an overlapping region of o×o pixels. Further, consider that the o×o pixels are convolved as part of the first image patch to generate a first convolved representation that is stored in memory. Then, when the second image patch is convolved, the o×o pixels are not convolved again; instead, the first convolved representation is retrieved from memory and reused. In some implementations, n=m. In other implementations, they are not equal.

[0582] The input data is then processed through specially designed spatial and temporal convolutional layers to produce a final temporal output of dimensionality w×w×k. Again, under a consistent reduction of spatial dimensionality events, the spatial dimensionality is reduced by a constant step size of 2 at each convolutional layer. That is, starting with n×n spatial dimensionality of the input data, a final temporal output of w×w spatial dimensionality is derived.

[0583] Then, based on the final temporal output of spatial dimensionality w×w, an output layer generates a base call for each unit in the w×w set of units. In one embodiment, the output layer is a softmax layer that generates four classification scores for the four bases (A, C, T, and G) per unit. That is, each unit in the w×w set of units is assigned a base call based on the maximum classification score in the corresponding softmax quadruple, as shown in Figure 97. In some embodiments, the w×w set of units is derived as a result of processing the final temporal output through a flattening layer and a dense layer to generate a flattened output and a dense output, respectively. In such embodiments, the flattened output has w×w×k elements, and the dense output has w×w elements forming a w×w set of units.

[0584] Base calls for multiple target clusters are obtained by identifying which of the base-called units in a w x w set of units match or correspond to center cluster pixels, i.e., pixels in the input data that contain the centers of each of the multiple target clusters. A given target cluster is assigned a base call of a unit that matches or corresponds to a pixel that contains the center of the given target cluster. In other words, base calls of units that do not match or correspond to center cluster pixels are filtered. This function is handled by a base call filtering layer, which in some embodiments is part of a dedicated architecture or, in other embodiments, is implemented as a post-processing module.

[0585] In another embodiment, base calls for multiple target clusters are obtained by identifying which groups of base-called units in a w x w set of units cover the same cluster, i.e., by identifying groups of pixels in the input data that represent the same cluster. Then, for each cluster and its corresponding group of pixels, the average classification score (softmax probability) of each of the four base classes (A, C, T, and G) is calculated over the pixels in the group of pixels, and the base class with the highest average classification score is selected to base call the cluster.

[0586] During training, in some embodiments, ground truth comparison and error calculation occurs only for those units that match or correspond to the central cluster pixel, so that their predicted base calls are evaluated against the correct base calls, which are identified as ground truth labels.

[0587] While multiple cluster base calling has been described, the discussion here describes multiple cluster and multiple cycle base calling. (multiple clusters and multiple cycle base calls)

[0588] Figure 98 shows one embodiment in which multiple target clusters are base called simultaneously in multiple successive sequencing cycles, thereby simultaneously generating base called sequences for each of the multiple target clusters.

[0589] In the single and multiple base calling embodiments described above, base calls in one sequencing cycle (the current (time t) sequencing cycle) are predicted using data for three sequencing cycles (the current (time t), previous / left flanking (time t-1), and next / right flanking (time t+1) sequencing cycles), with the right and left flanking sequencing cycles providing sequence-specific context for the base triplet motif and quadratic contributions of prephasing and phasing signals. This relationship is expressed by the following equation: "Number of sequencing cycles for which data is included in input data (t) = Number of sequencing cycles being base called (y) + Number of right and left flanking sequencing cycles (x)."

[0590] In Figure 98, the input data includes t cycle-by-cycle tensors for t sequencing cycles, making the dimensionality nxnxmxt, where n = 114, m = 5, and t = 15. In other embodiments, these dimensionalities are different. 番目 The first and second sequencing cycles serve as the right and left flanking contexts x, and y sequencing cycles between them are base called. Thus, y = 13, x = 2, and t = y + x. Each tensor for each cycle includes an image channel, a corresponding distance channel, and a scaling channel, such as input tensor 9500 in Figure 95a.

[0591] The input data, having t tensors per cycle, is then processed through dedicated spatial and temporal convolutional layers to generate y final temporal outputs, each corresponding to a respective one of the y sequencing cycles being base-called. Each of the y final temporal outputs has a dimensionality of w × w × k. Again, under consistent spatial dimensionality reduction, the spatial dimensionality is reduced by a constant step size of 2 at each convolutional layer. That is, starting with the n × n spatial dimensionality of the input data, a w × w spatial dimensionality is derived for each of the y final temporal outputs.

[0592] Each of the y final temporal outputs is then processed in parallel by an output layer. For each of the y final temporal outputs, the output layer generates a base call for each unit in the w x w set of units. In one embodiment, the output layer is a softmax layer that generates four classification scores for the four bases (A, C, T, and G) per unit. That is, each unit in the w x w set of units is assigned a base call based on the maximum classification score in the corresponding softmax quadruple, as shown in Figure 97. In some embodiments, a w x w set of units is derived for each of the y final temporal outputs as a result of subsequent processing through a flattening layer and a dense layer, respectively, to generate a corresponding flattened output and a dense output. In such embodiments, each flattened output has w x w x k elements, and each dense output has w x w elements forming a w x w set of units.

[0593] For each of y sequencing cycles, base calls for the multiple target clusters are obtained by identifying which of the base-called units in the corresponding w × w set of units match or correspond to center cluster pixels, i.e., pixels in the input data that contain the centers of each of the multiple target clusters. A given target cluster is assigned a base call of a unit that matches or corresponds to a pixel that contains the center of the given target cluster. In other words, base calls of units tha...

Claims

1. 1. A computer-implemented method of end-to-end sequencing comprising integrating neural network-based template generation with neural network-based base calling, comprising: accessing first image data and second image data comprising pixels at an optical pixel resolution, the first image data comprising an image of the cluster and a background surrounding the cluster captured by a sequencing system during an initial one of a sequencing cycle of a sequencing operation; accessing the second image data, the second image data including images of the clusters and surrounding background of the clusters captured by the sequencing system during the initial and additional sequencing cycles of the sequencing operation; processing the first image data through a neural network based template generator to generate a cluster map that identifies cluster metadata, the cluster metadata identifying spatial distribution information of the clusters based on cluster centers, cluster shapes, cluster sizes, cluster backgrounds, and / or cluster boundaries; generating, the neural network based template generator being trained on the task of mapping the images of the clusters to the cluster metadata; encoding the spatial distribution information of the clusters in a template image at an upsampled sub-pixel resolution, where the sub-pixels of the template image and the pixels of the image of the clusters represent the same imaging area; modifying intensity values ​​of the pixels of the second image data based on the template image to generate an intensity-modified version of the second image data having an intensity distribution that takes into account the spatial distribution information of the clusters; and processing the intensity-corrected version of the second image data through a neural network-based base caller to generate base calls for one or more of the clusters in one or more sequencing cycles of the sequencing operation, wherein the neural network-based base caller is trained with the task of mapping the images of the clusters to the base calls.

2. supplementing the template image with the second image data instead of modifying the intensity values ​​of the pixels of the second image data; 2. The computer-implemented method of claim 1, further comprising: processing the second image data supplemented with the template image through the neural network-based base caller to generate base calls for one or more of the clusters in one or more sequencing cycles of the sequencing operation.

3. The computer-implemented method of claim 1 , wherein each subpixel in the template image is identified as either a background subpixel, a cluster center subpixel, or a cluster interior subpixel.

4. Modifying intensity values ​​of the pixels of the second image data includes: calculating a region weighting factor for one or more pixels in the second image data based on how many sub-pixels in the template image that correspond to pixels in the image of the second image data include portions of one or more of the clusters; and modifying the intensity of the pixel based on the region weighting factor.

5. Modifying intensity values ​​of the pixels of the second image data includes:

5. The computer-implemented method of claim 1, comprising: upsampling the images of clusters and background surrounding clusters to the upsampled sub-pixel resolution to generate an upsampled image; assigning a background intensity to sub-pixels in the upsampled image that correspond to background sub-pixels in the template image; and assigning a cluster intensity to sub-pixels in the upsampled image that correspond to cluster center sub-pixels and cluster interior sub-pixels in the template image.

6. The computer-implemented method of claim 5 , wherein the background intensity has a value of zero.

7. A computer-implemented method according to any preceding claim, wherein cluster intensities are determined by interpolating intensities of the pixels at the optical pixel resolution.

8. Modifying intensity values ​​of the pixels of the second image data includes:

8. The computer-implemented method of claim 1, comprising: upsampling the image of a cluster and a background surrounding the cluster to the upsampled sub-pixel resolution to generate an upsampled image; and distributing the entire intensity of a pixel in an optical pixel domain only among constituent sub-pixels of the pixel in the upsampled image that correspond to cluster center sub-pixels and cluster interior sub-pixels in the template image.

9. determining a template image for the clusters using a first neural network, the template image identifying at least one of the characteristics selected from the group consisting of a spatial distribution of the clusters, a cluster shape, a center of the cluster, and a cluster boundary; and using a second neural network to base call the clusters based on the template image.

10. modified intensity values ​​for identifying at least one of the characteristics of the template image selected from the group consisting of: spatial distribution of the clusters, cluster shapes, centers of the clusters, and cluster boundaries; and processing the corrected intensity values ​​through the second neural network to base call the clusters.

11. The computer-implemented method of claim 9 or 10, wherein the template image comprises a template image.

12. evaluating the template image in an upsampled sub-pixel domain for at least one particular cluster to identify pixels that include a portion of the at least one particular cluster and pixels adjacent to the pixels that also include a portion of the at least one particular cluster; calculating a region weighting factor for each pixel based on how many sub-pixels in each of the identified pixels include part of the at least one particular cluster; 12. The computer-implemented method of claim 11, further comprising modifying pixel intensity values ​​of the identified pixel and the neighboring pixels for processing based on the region weighting factor for each pixel.

13. evaluating the template image 13. The computer-implemented method of claim 12, further comprising: processing one or more initial image sets, each generated in one or more initial sequencing cycles of a plurality of sequencing cycles, through the first neural network to generate the template image to identify the center, shape, and boundary of the cluster at the upsampled sub-pixel resolution, each image set including one or more images, each of the images showing intensity radiation of the cluster and a background surrounding the cluster in a respective one of one or more imaging channels captured at optical pixel resolution.

14. evaluating the template image evaluating the cluster shape and boundary of the at least one particular cluster to identify at least one pixel that includes a portion of the at least one particular cluster and pixels adjacent to the pixel that also include a portion of the at least one particular cluster, the method further comprising: storing the region weighting factors in the template image; generating a modified version of each of said images having pixels with modified pixel intensity values; processing the modified version of the image through the second neural network to generate alternative representations of the modified version; 14. The computer-implemented method of claim 12 or 13, further comprising: base calling the at least one particular cluster using the alternative representation.

15. The base calling comprises: a current image set generated in a current one of a plurality of sequencing cycles, one or more previous image sets generated in one or more of the plurality of sequencing cycles that precede the current one of the plurality of sequencing cycles, respectively; and accessing one or more images at optical pixel resolution in each of one or more subsequent image sets generated in one or more of the plurality of sequencing cycles respectively subsequent to the current one of the plurality of sequencing cycles; for pixels in each of said images, modifying pixel intensity values ​​based on said region weighting factors in said template image for the respective pixels; generating a modified version of each of said images having pixels with modified pixel intensity values; For the at least one particular cluster, each image patch is an array of pixels; and extracting an image patch from each modified version such that the central pixel of the array contains the center of the particular cluster identified in the template image; convolving an image patch extracted from the modified version of the image through a convolutional neural network of the second neural network to generate a convolved representation of the image patch; processing the convolutional representation through an output layer to generate, relative to the center pixel, the likelihoods of bases being incorporated into the at least one particular cluster in the current one of the plurality of sequencing cycles being A, C, T, and G; 15. The computer-implemented method of claim 14, further comprising: classifying the base as A, C, T, or G based on the likelihood.

16. 16. The computer-implemented method of claim 14 or 15, further comprising aligning each of the images captured at the optical pixel resolution with the template image using a cycle-specific and imaging channel-specific transform prior to modifying the pixel intensity values.

17. evaluating the template image in an upsampled sub-pixel domain to identify sub-pixels that include part of any cluster; 10. The computer-implemented method of claim 9, further comprising: assigning a background intensity to sub-pixels identified in the template image as not contributing to any cluster.

18. Evaluating the template image in an upsampled sub-pixel domain includes:

20. The computer-implemented method of claim 17, further comprising: calculating how many sub-pixels in at least one pixel include part of any cluster; and calculating per-sub-pixel region weighting factors for the sub-pixels in the at least one pixel.

19. The method further comprising: processing one or more initial image sets, each generated in one or more initial sequencing cycles of a plurality of sequencing cycles, through the first neural network to generate the template image at the upsampled sub-pixel resolution, each image set including one or more images, each of the images showing intensity radiation of the cluster and a background surrounding the cluster in a respective one of one or more imaging channels captured at an optical pixel resolution, the template image classifying sub-pixels into classes including cluster center, background, and cluster interior; upsampling each of the images captured at the optical pixel resolution to a sub-pixel domain; and assigning a background intensity to each sub-pixel of the images that is identified in the template image as not contributing to any cluster; processing the upsampled image through the second neural network to generate an alternative representation of the upsampled image; and b. base calling a plurality of said clusters using said alternative representations.

20. upsampling each of the images 20. The computer-implemented method of claim 19, further comprising: distributing intensities of particular pixels among first sub-pixels of the particular pixels identified in the template image as contributing to any cluster by applying the sub-pixel-wise regional weighting factor; and assigning a background intensity to second sub-pixels of the particular pixels identified in the template image as not contributing to any cluster.

21. Prior to the upsampling, the method further comprises: a current image set generated in a current one of the plurality of sequencing cycles, one or more previous image sets generated in one or more of the plurality of sequencing cycles that precede the current one of the plurality of sequencing cycles, respectively; and accessing one or more images at the optical pixel resolution in each of one or more subsequent image sets generated in one or more of the plurality of sequencing cycles respectively subsequent to the current one of the plurality of sequencing cycles, wherein after upsampling, the method further comprises: extracting image patches from each upsampled image, such that each image patch has an array of sub-pixels; convolving image patches extracted from the upsampled image through a convolutional neural network of the second neural network to generate convolved representations of the image patches; processing the convolutional representation through an output layer to generate, for each subpixel in the array, a likelihood of a base being incorporated in the current one of the plurality of sequencing cycles that are A, C, T, and G; classifying the base as A, C, T, or G based on the likelihood; and base calling each one of the plurality of clusters based on a base classification assigned to a respective subpixel that comprises a center of a corresponding cluster.

22. 22. The computer-implemented method of claim 20 or 21, further comprising aligning each of the images captured at the optical pixel resolution with the template image using cycle-specific and imaging channel-specific transforms prior to the upsampling.

23. a receptacle coupled to a biosensor system, the biosensor system configured to include an array of photodetectors, the biosensor system including a biosensor, the biosensor including a reaction site configured to include a cluster; an illumination system configured to direct excitation light to the biosensor and illuminate the clusters within the reaction sites, the illumination system providing a luminescence signal when at least some of the clusters are illuminated; a system controller coupled to the receiver and comprising an analysis module, the analysis module comprising: acquiring image data from the photodetector at each of a plurality of sequencing cycles, the image data being derived from the luminescence signal detected by the photodetector; a system controller configured to process the image data for each of the plurality of sequencing cycles through a neural network and generate base calls for at least some of the clusters in each of the plurality of sequencing cycles.

Images