Polymer analysis using transformer neural network

Abstract

A method of estimating a sequence of polymer units of a polymer comprising measurements of a signal generated during translocation of the polymer with respect to a nanopore includes determining a sequence of hidden states associated with the signal using a transformer encoder network, and processing the sequence of hidden states using a network head to estimate a sequence of polymer units of the polymer. The processing by the network head includes generating a respective array of values for each hidden state indicating an estimated likeliness of the hidden state being associated with each of a set of symbols, where the set of symbols includes symbols associated with respective types of canonical polymer unit, and analysing the generated arrays of values to estimate the sequence of polymer units of the polymer.

Description

[0001]POLYMER ANALYSIS USING TRANSFORMER NEURAL NETWORKTechnical Field The present disclosure relates to estimating a sequence of polymer units of a polymer. The disclosure has particular, though not exclusive, relevance to basecalling a polynucleotide. Background Various biochemical analysis systems provide measurements of polymer units for the purpose of determining the sequence. One such type of analysis system uses a nanopore. Biochemical analysis systems that use a nanopore have been the subject of much recent development. Typically, successive measurements of a polymer are taken from a sensor element comprising a nanopore during translocation of the polymer through the nanopore. Some property of the system depends on the polymer units in the nanopore, and measurements of that property are taken. This type of measurement system using a nanopore has considerable promise, particularly in the field ofsequencing a polynucleotide such as Deoxyribonucleic acid (DNA) or Ribonucleic acid(RNA), sometimes referred to as “basecalling”. Biochemical analysis systems using nanopores can provide long continuous reads of polymers, for example in the case of polynucleotides ranging in length from 20 to >1Mb. The data gathered in this way comprises measurements, such as measurements of ion current, where the translocation of the polymer through thenanopore results in a change in the measured property indicative of the sequence ofpolymer units. Some biochemical analysis systems use machine learning models to estimate a sequence of polymer units from data comprising sequences of measurements collected as described above. Suitable machine learning models include Hidden Markov Models (HMMs), neural networks including recurrent neural network (RNN) models, such asthose comprising long short-term memory (LSTM) units, and combinations of theabove. However, the inventors have recognised that such models have limitations interms of accuracy and scalability, and the performance of such models can deterioratefor very long sequences of measurements as may be needed to sequence polymers withhundreds of thousands, or more, polymer units.Summary According to aspects of the present disclosure, there are provided a computer-implemented method of estimating a sequence of polymer units of a polymercomprising measurements of a signal generated during translocation of the polymer with respect to a nanopore, a data processing system comprising means for carrying out the method, and a computer program product (such as one or more non-transitory computer-readable storage media) comprising instructions which, when executed by a computer, cause the computer to carry out the method. The method includes determining a sequence of hidden states associated with the signal using a transformer encoder network, and processing the sequence of hiddenstates using a network head to estimate a sequence of polymer units of the polymer.The processing by the network head includes generating a respective array of values for each hidden state indicating an estimated likeliness of the hidden state being associated with each of a set of symbols, where the set of symbols includes symbols associatedwith respective types of canonical polymer unit, and analysing the generated arrays ofvalues to estimate the sequence of polymer units of the polymer.The inventors have found that using the transformer encoder network andnetwork head enables more accurate sequencing or basecalling at a given number offloating point operations per second (FLOPS) when compared with other types ofmachine learning model for processing sequential data. For example, the inventors wereable to achieve around 1.5 Q higher accuracy than a state-of-the-art basecalleremploying a long short-term memory (LSTM) architecture. Furthermore, the accuracyof the proposed architecture is found to be more scalable with model size (e.g. depth,size of hidden layer, length of training runs, context length etc.) than existingapproaches, leading to the possibility of even better performance as improvements inhardware and techniques such as quantisation enable larger models to be realised.The transformer encoder network may use sliding window self-attention, whereby each hidden state depends on a respective portion of the signal. The respective portion may be a contiguous portion or may include gaps, for example in the case of diluted sliding window self-attention or cross-attention between concatenated signal portions as described hereinafter. By using sliding window self-attention, the memoryfootprint of the method becomes independent of the length of the signal while thecompute cost scales linearly with the length of the signal, in contrast with the quadraticscaling exhibited by global self-attention mechanisms. This may enable very longpolymers (for example, more than 100,000, more than 1,000,000, or more than10,000,000 polymer units) to be sequenced or basecalled in a single pass, potentially insubstantially real-time as the polymer is translocated with respect to the nanopore. By contrast, the performance of methods based on recurrent neural networks such as LSTMs tends to decline with very long sequence lengths, for example sequences with more than 100,000 polymer units. The inventors have discovered that the accuracy of the method may begin tosaturate beyond a certain window size, for example at a window size of around 64, 128,256, or 512, meaning that most of the benefits of the transformer encoder architecture may be reaped using a window size of between 64 and 512, without being subject tothe quadratic scaling of compute costs (this may be in contrast with for examplelanguage modelling applications of transformers, in which long-range relationshipsbetween tokens may be critical and therefore global attention may be needed to achievepeak performance). Determining the sequence of hidden states may include processing the signal to generate a sequence of feature vectors using one or more neural network layers to generate a sequence of feature vectors, the sequence of feature vectors, and processing the sequence of feature vectors using the transformer encoder network to determine the sequence of hidden states. The one or more neural network layers may include one or more convolutional layers. The polymer units of the polymer may pass through the nanopore at an averagetranslocation rate, and the signal may have a first frequency that is greater (for example,at least 3 times greater) than the average translocation rate, thereby enabling multiple measurements to be made in the vicinity of each polymer unit. The sequence of featurevectors may have a second frequency that is lower than the first frequency. Processingthe sequence of feature vectors may include generating a sequence of intermediatestates having the second frequency as an output of the transformer encoder network, and upsampling the sequence of intermediate states to determine the sequence of hidden states. The sequence of hidden state may have a third frequency that is at no less than the average translocation rate. By generating a sequence of hidden states with a lower frequency than thesignal, the compute cost and memory footprint of the subsequent processing by thetransformer encoder may be reduced significantly without reducing accuracy. By usingone or more neural network layers (which may include one or more convolutionallayers) to achieve this reduction in frequency, the feature vectors may neverthelesscapture substantially all of the useful information within the signal. The inventors have found that frequency reduction by factor of between 5 and 20 may be achieved without reducing accuracy. The frequency may nevertheless be at least partially restored upon generating the hidden states, otherwise it may not be possible for the network head to resolve all of the polymer units of the polynucleotide. To enable full resolution of the polynucleotide, the third frequency may for example be between 1.5 and 5 times the average translocation rate. The upsampling of the sequence of intermediate states may be performed forexample using a linear upsampling layer, or a convolutional upsampling layer, or anattention upsampling layer. The linear upsampling may achieve at least comparable performance with other upsampling methods, while being less computationally expensive and not significantly increasing the number of trainable parameters of the machine learning model. Analysing the generated arrays of values may include applying a search algorithm to the generated arrays of values. For example, the set ofsymbols may include a blank symbol, and analysing the generated arrays of values mayinclude determining an intermediate sequence of symbols based on the generated arrays of value, and collapsing the intermediate sequence of symbols by merging repeated symbols and then removing instances of the blank symbol. In this way, the search algorithm may correspond to a connectionist temporal classification (CTC) topology. Providing a blank symbol and an algorithm for collapsing the intermediatesequence of symbols may enable the sequence of polymer units to be estimated withoutrelying on a known alignment between measurements in the signal and the polymer units in the polymer. In this way, configuring the apparatus may be greatly simplifiedwhile also providing robustness against variations in translocation rate. The intermediate sequence of symbols may be determined using a greedy searchtechnique in which the most likely symbol indicated by each array of values is selected,which in many cases will result in the most likely sequence of polymer units being returned. However, due to the possible lack of alignment, multiple intermediatesequences may map to a given final estimate after the collapsing algorithm is applied.In that case, a better approach may be to identify the estimate whose corresponding intermediate sequences have the greatest cumulative posterior likelihood. To account for this situation, determining the intermediate sequence of symbols may include performing a beam search based on the generated arrays of values. Using a wider beammay result increase the probability of the best estimate being identified, at the expenseof a higher compute cost. The network head may be configured (for example, trained) to generate the arrays of values as a conditional random field (CRF), which advantageously may allow for conditional dependence of the arrays of values which may result in improved accuracy compared with methods that assume conditional independence. When thearrays of values are generated as a CRF and then the collapse algorithm described aboveis applied to arrive at the final estimate of the sequence of polymer units, the resultingnetwork head may be referred to as a CTC-CRF head, though it will be appreciated that other types of network head may be used, such as a CTC head, an autoregressive transformer decoder, or any other form of network head or decoder that can estimate or classify polymer units from the series of hidden states. The polymer may be a first polymer, and the signal may further comprise measurements of a second polymer by the sensor element during translocation of the second polymer with respect to the nanopore, the first polymer and the second polymerbeing a complementary pair. The transformer encoder network may include anattention mechanism arranged to capture relationships between measurements associated with mutually corresponding segments of the first polymer and the second polymer. For example, the signal may include a first signal portion comprising measurements of the first polymer concatenated with a second signal portion comprising measurements of the second polymer. The attention mechanism may thenbe a global self-attention mechanism which learns to cross-attend betweencorresponding parts of the first and second signal portions, which advantageously mayobviate the need to align the first and second signal portions. Alternatively, theattention mechanism may use two local windows respectively centred on(approximately) corresponding parts of the first and second signal portions. This approach may result in a more scalable compute cost as the size of the sliding windows can remain constant, but may need at least partial alignment of the signal portions. The transformer encoder may include between 8 and 64 transformer encoderblocks. For example, the transformer encoder may include between 12 and 20transformer encoder blocks, which the inventors have found to provide a good balancein the trade-off between accuracy and speed. The polymer may be a polynucleotide such as DNA or RNA, and the polymer units may be nucleotides. A further aspect of the present disclosure include apparatus comprising a sensor element comprising a nanopore, means for translocating a polymer with respect to the nanopore, and a data processing system comprising means for carrying out the above computer-implemented method. The means may include a polymer binding enzyme for controlling the translocation of the polymer with respect to the nanopore. A still further aspect includes a method comprising translocating a polymer with respect to a nanopore, generating a signal comprising measurements of the polymer by a sensor element comprising the nanopore, and analysing the signal using the above computer-implemented method. According to further aspects of the present disclosure, there are provided acomputer-implemented method of training a neural network model comprising atransformer encoder network and a network head, a data processing system comprising means for carrying out the method, and a computer program product (such as one or more non-transitory computer-readable storage media) comprising instructions which, when executed by a computer, cause the computer to carry out the method. The method may include providing a signal comprising measurements of a polymer by a sensor element comprising a nanopore during translocation of the polymer with respect to the nanopore, the signal being associated with a target sequence ofpolymer units, determining a sequence of hidden states associated with the signal usingthe transformer encoder network, processing the sequence of hidden states using the network head to generate a respective array of values indicating an estimated likeliness of each hidden state being associated with each of a set of symbols comprising symbols associated with respective types of canonical polymer unit and updating parameters of the neural network based on a loss function depending on the generated arrays of values and the target sequence of polymer units. The loss function may include at least one of a connectionist temporal classification (CTC) loss and a connectionist temporal classification-conditional random field (CTC-CRF) loss, which may enable training on signals where alignment between the measurements and the polymer units is not known a priori. Further features and advantages will become apparent from the following description of preferred embodiments, given by way of example only, which is made with reference to the accompanying drawings. Brief Description of the Drawings Fig.1 shows an example of apparatus for estimating a sequence of polymer unitsof a polymer. Fig. 2 shows schematically an example of a neural network for analysing asignal comprising measurements of a polymer.Fig. 3 is a flow diagram representing a method of analysing a signal accordingto an aspect of the present disclosure. Fig. 4 is a flow diagram representing a method of training a neural network according to an aspect of the present disclosure. Figs.5A and 5B illustrate methods of analysing a composite signal comprising measurements of complementary strands of a polymer. Detailed Description Details of systems and methods according to examples will become apparent from the following description with reference to the figures. In this description, for the purposes of explanation, numerous specific details of certain examples are set forth. Reference in the specification to ‘an example’ or similar language means that a feature, structure, or characteristic described in connection with the example is included in at least that one example but not necessarily in other examples. It should be further noted that certain examples are described schematically with certain features omitted and / or necessarily simplified for the ease of explanation and understanding of the concepts underlying the examples. Embodiments of the present disclosure relate to sequencing polymers such as polynucleotides. In particular, embodiments described herein address challenges involved in accurately and efficiently sequencing long-chain polymers. Fig. 1 shows apparatus 100 for estimating a sequence or series of polymer unitsof a polymer 102. The polymer 102 may be a polynucleotide strand of a polynucleotide(or nucleic acid), and the polymer units may be nucleotides. However, in general thepolymer 102 may be of any type, for example a polypeptide such as a protein, or apolysaccharide. The polymer 102 may be natural or synthetic. In the case of apolynucleotide or nucleic acid, the polymer units are nucleotides. The nucleic acid is typically deoxyribonucleic acid (DNA), ribonucleic acid (RNA), or a synthetic nucleic acid known in the art, such as peptide nucleic acid (PNA), glycerol nucleic acid (GNA), threose nucleic acid (TNA), locked nucleic acid (LNA) or other synthetic polymers with nucleotide side chains. The nucleic acid may be single-stranded, be double-stranded or comprise both single-stranded and double-stranded regions. The nucleic acid may comprise one strand of RNA hybridised to one strand of DNA. Typically cDNA, RNA, GNA, TNA or LNA are single stranded. The polymer units may be any type of nucleotide. The nucleotide can be naturally occurring or artificial. For instance, the method may be used to verify the sequence of a manufactured oligonucleotide. The polymer units may be canonical polymer units. For example in the case that the polymer 102 is a DNA polynucleotide, the canonical bases are adenine (A), cytosine (C), guanine (G), and thymine (T). By contrast ribonucleic acid (RNA) comprises the canonical bases A, C and G, with uracil (U) in place of thymine. A nucleotide may also lack a nucleobase and a sugar. The nucleotide may be modified such as 5mC, 5hmC and N1-methylpseudouridine. Machine learning methods to detect modified bases are disclosed in WO23094806. The apparatus 100 comprises a sensor element 104 comprising a nanopore 106situated in a membrane 108, and a sensor device 110. Although only a single nanopore106 is shown in Fig. 1, the apparatus 100 may employ many nanopores, for example arranged in an array, to provide parallelised collection of information. The measurement system may comprise at least 10 nanopores, at least 100 nanopores, or at least 1,000 nanopores. The nanopore 106 is a pore or hole, typically having a size of the order ofnanometres, which may allow the passage of polymers therethrough. The nanopore may be a protein pore or a solid state pore. The nanopore 106 may, for example, be a protein pore such as a polypeptide or a collection of polypeptides. Alternatively, the nanopore may be composed of any other such molecules that allow the nanopore 106 to function as an aperture in the membrane 108. The biological pore may be a transmembrane protein pore. Transmembrane protein pores for use in accordance with the invention can be derived from ^-barrel pores or α-helix bundle pores. ^-barrel pores comprise a barrel or channel that is formed from ^-strands. Suitable ^-barrel pores include, but are not limited to, ^-toxins, such as α-hemolysin, anthrax toxin and leukocidins, and outer membrane proteins / porins of bacteria, such as Mycobacterium smegmatis porin (Msp), for example MspA, MspB, MspC or MspD, lysenin, outer membrane porin F (OmpF), outer membrane porin G (OmpG), outer membrane phospholipase A and Neisseria autotransporter lipoprotein (NalP). α-helix bundle pores comprise a barrel or channel that is formed from α-helices. Suitable α-helix bundle pores include, but are not limited to, inner membrane proteins and α outer membrane proteins, such as WZA and ClyA toxin. The transmembrane pore may be derived from Msp or from α-hemolysin (α-HL). The transmembrane pore may be derived from lysenin. Suitable pores derived from lysenin are disclosed in WO 2013 / 153359. Suitable pores derived from MspA are disclosed in WO-2012 / 107778. The pore may be derived from CsgG, such as disclosed in WO-2016 / 034591 and WO2019 / 002893, both herein incorporated by reference in their entirety. The pore may be a DNA origami pore. A protein pore may be a naturally occurring pore or may be a mutant pore. A protein pore may be inserted into an amphiphilic layer such as a biological membrane, for example a lipid bilayer. An amphiphilic layer is a layer formed from amphiphilic molecules, such as phospholipids, which have both hydrophilic and lipophilic properties. The amphiphilic layer may be a monolayer or a bilayer. The amphiphilic layer may be a co-block polymer such as disclosed in Gonzalez-Perez et al., Langmuir, 2009, 25, 10447-10450, WO2014 / 064444, or US6723814 herein incorporated by reference in its entirety. Alternatively, a protein pore may be inserted into an aperture provided in a solid state layer, for example as disclosed in WO2012 / 005857. A suitable apparatus for providing an array of nanopores is disclosed in WO 2014 / 064443. The nanopores may be provided across respective wells wherein electrodes are provided in each respective well in electrical connection with an ASIC for measuring current flow through each nanopore. A suitable current measuring apparatus may comprise the current sensing circuit as disclosed in WO-2016 / 181118. The nanopore 106 may comprise an aperture formed in a solid state layer, which may be referred to as a solid state pore. The aperture may be a well, gap, channel, trench or slit provided in the solid state layer along or into which analyte may pass. Such a solid-state layer is not of biological origin. In other words, a solid state layer is not derived from or isolated from a biological environment such as an organism or cell, or a synthetically manufactured version of a biologically available structure. Solid state layers can be formed from both organic and inorganic materials including, but not limited to, microelectronic materials, insulating materials such as Si3N4, A1203, and SiO, organic and inorganic polymers such as polyamide, plastics such as Teflon® or elastomers such as two-component addition-cure silicone rubber, and glasses. The solid state layer may be formed from graphene. Suitable graphene layers are disclosed in WO-2009 / 035647, WO-2011 / 046706 or WO-2012 / 138357. Suitable methods to prepare an array of solid state pores is disclosed in WO-2016 / 187519. Such a solid state pore is typically an aperture in a solid state layer. The aperture may be modified, chemically, or otherwise, to enhance its properties as a nanopore. A solid state pore may be used in combination with additional components which provide an alternative or additional measurement of the polymer such as tunnelling electrodes (Ivanov AP et al., Nano Lett. 2011 Jan 12;11(1):279-85), or a field effect transistor (FET) device (as disclosed for example in WO-2005 / 124888). Solid state pores may be formed by known processes including for example those described in WO-00 / 79257. The nanopore 106 may be a hybrid of a solid state pore with a protein pore. The polymer 102 may be a single strand of a dual-stranded polymer 112. Thenanopore 106 may be constructed to allow the polymer 102 to be moved or translocatedthrough the nanopore 106.The sensor device 110 may takes a series or sequence of measurements of aproperty that depends on the polymer units of the polymer 102 being translocated withrespect to the nanopore 106. The series of measurements may form a measurementsignal. The property that is measured may be associated with an interaction betweenthe polymer 102 and the nanopore 106. Such an interaction may occur at a constrictedregion of the pore. In one type of sensor device 110, the property that is measured maybe the ion current flowing through a nanopore. These and other electrical propertiesmay be measured using standard single channel recording equipment as describe inStoddart D et al., Proc Natl Acad Sci, 12;106(19):7702-7, Lieberman KR et al, J AmChem Soc. 2010;132(50):17961-72, and WO-2000 / 28312. Alternatively, measurements of electrical properties may be made using a multi-channel system, for example as described in WO-2009 / 077734, WO-2011 / 067559 or WO-2014 / 064443. The property that is measured by the sensor device 110 may not necessarily be ion current. Some examples of alternative types of property include electrical properties and optical properties. A suitable optical method involving the measurement of fluorescence is disclosed by J. Am. Chem. Soc. 2009, 131 1652-1653. Possible electrical properties include: ionic current, impedance, a tunnelling property, for example tunnelling current (for example as disclosed in Ivanov AP et al., Nano Lett. 2011 Jan 12;11(1):279-85), and a FET (field effect transistor) voltage (for example as disclosed in WO2005 / 124888). One or more optical properties may be used, optionally combined with electrical properties (Soni GV et al., Rev Sci Instrum. 2010 Jan;81(1):014301). The property may be a transmembrane current, such as ion current flow through a nanopore. The ion current may typically be the DC ion current, although in principle an alternative is to use the AC current flow (i.e. the magnitude of the AC current flowing under application of an AC voltage). Ionic solutions may be provided on either side of the membrane 108 or solidstate layer, which ionic solutions may be present in respective compartments. A sample containing the polymer analyte of interest may be added to one side of the membrane and allowed to move with respect to the nanopore, for example under a potential difference or chemical gradient. The measurement signal may be derived during themovement of the polymer 102 with respect to the nanopore 106, for example takenduring translocation of the polymer 102 through the nanopore 106. The polymer maypartially translocate the nanopore 106.In order to allow measurements to be taken as the polymer 102 passes throughthe nanopore 106, the rate of translocation can be controlled by a polymer bindingmoiety. Typically the moiety can move the polymer 102 through the nanopore with oragainst an applied field. The moiety can be a molecular motor using for example, in the case where the moiety is an enzyme, enzymatic activity, or as a molecular brake. Where the polymer is a polynucleotide there are a number of methods proposed for controlling the rate of translocation including use of polynucleotide binding enzymes. Suitable enzymes for controlling the rate of translocation of polynucleotides include, but are not limited to, polymerases, helicases, exonucleases, single stranded and double stranded binding proteins, and topoisomerases, such as gyrases. The helicase may be any of the helicases, modified helicases or helicase constructs disclosed in WO2013 / 057495, WO 2013 / 098562, WO2013098561, WO 2014 / 013259; WO 2014 / 013262 and WO 2014013260. The helicase may be added to the polynucleotide during sample preparation and stalled by one or more spacers as disclosed in WO2014135838. For other polymer types, moieties that interact with that polymer type can be used. The polymer interacting moiety may be any disclosed in WO-2010 / 086603, WO- 2012 / 107778, and Lieberman KR et al, J Am Chem Soc.2010;132(50):17961-72), and for voltage gated schemes (Luan B et al., Phys Rev Lett. 2010;104(23):238103). The rate of translocation of the polymer through the nanopore may be controlled by a voltage control pulse to step the polymer through the nanopore such as disclosed in WO2019 / 006214. Translocation of the polymer may be controlled by a molecular hopper such as disclosed by WO2020 / 016573. The polynucleotide may comprise a polymer leader sequence which preferentially threads into the pore. The leader is preferably negatively charged and may be a polynucleotide, such as DNA or RNA, a modified polynucleotide (such as abasic DNA), PNA, LNA, polyethylene glycol (PEG) or a polypeptide. The leader sequence may form part of a Y adaptor typically comprising (a) a double stranded region and (b) a single stranded region or a region that is not complementary at the other end. Leader sequences and Y adaptors suitable for use are disclosed for example in WO2017149316. The effective surface concentration at the membrane surface can be enhanced by coupling the polynucleotide to the membrane. Suitable coupling moieties are disclosed for example in WO2017149316 and WO12164270. The polymer binding moiety can be used in a number of ways to control the polymer motion. The moiety can move the polymer through the nanopore with or against the applied field. The polynucleotide binding enzyme does not need to display enzymatic activity as long as it is capable of binding the target polynucleotide and controlling its movement through the pore. For instance, the enzyme may be modified to remove its enzymatic activity or may be used under conditions which prevent it from acting as an enzyme. Such conditions are discussed in more detail below. The polynucleotide binding enzyme may be a Dda helicase such as disclosed in WO2015055981, hereby incorporated by reference in its entirety. Translocation of the polymer 102 through the nanopore may occur, either cis totrans or trans to cis, either with or against an applied potential, applied by an electroniccircuit (not shown) as described hereafter. The translocation may occur under anapplied potential which may control the translocation. The binding enzyme is typically held against the cis or trans opening of the nanopore during translocation of the polynucleotide through the nanopore under an applied potential. The electronic circuit may be an electronic circuit as disclosed inWO2016059427, incorporated herein by reference in its entirety. The electronic circuitmay be arranged to control the application of bias voltages across each sensor elementof the sensor device 110. During normal operation, the bias voltage is selected to enable translocation of a polymer through the pore of a sensor element. Such a bias voltage may typically be of a level up to -200 mV. The bias voltage supplied by the electronic circuit may also be selected so that it is sufficient to eject the translocating polymer from the pore. By causing the electronic circuit to supply such a bias voltage, the sensor element is operable to eject a polymer that is translocating through the pore. To ensure reliable ejection, the bias voltage is typically a reverse bias, although that is not alwaysessential. The electronic circuit may be connected to electrodes on either side of themembrane 108. The electronic circuit controls the application of bias voltages togenerate a bias between the electrodes to control translocation of the polymer 102 asdescribed above. In some examples, exonucleases that act progressively or processively on double stranded DNA can be used on the cis side of the nanopore 106 to feed the remaining single strand through under an applied potential or the trans side under a reverse potential. Likewise, a helicase that unwinds the double stranded DNA can also be used in a similar manner. There are also possibilities for sequencing applications that require strand translocation against an applied potential, but the DNA must be first “caught” by the enzyme under a reverse or no potential. With the potential then switched back following binding the strand will pass cis to trans through the pore and be held in an extended conformation by the current flow. The single strand DNA exonucleases or single strand DNA dependent polymerases can act as molecular motors to pull the recently translocated single strand back through the pore in a controlled stepwise manner, trans to cis, against the applied potential. Alternatively, the single strand DNA dependent polymerases can act as a molecular brake slowing down the movement of a polynucleotide through the pore. Any moieties, techniques or enzymes described in WO-2012 / 107778 or WO-2012 / 033524 could be used to control polymer motion. Measurement signals generated by the sensor device 110 may be provided to a data processing system 114 for analysis. Variations of the property measured by thesensor device 110 may be extremely small. For example, measurements of ionic currentmay be on the order of picoamps. Measurements of the property may therefore byamplified before being digitised to generate the measurement signal.The data processing system 114 may be a desktop computer, a laptop computer,a tablet computer, a server computer, or any combination thereof. In some examples,the data processing system 114 may be a dedicated device for sequencing polymers, such as a device produced by Oxford Nanopore Technologies (RTM). The data processing system 114 includes one or more processors 116 and memory 118. Additionally, the data processing system 114 includes a power supply 120, which may include a mains power supply, a battery, a solar power supply, and / or the like, and the data processing system 114 may also include one or more interfacedevices 122. The interface devices 122 may include input devices such as a keyboards,a touch screen, a touch pad, a mouse, a microphone, etc. to enable a user to control thedata processing device 114, along with output devices such as a display, a speaker, etc.The interface devices 122 may also include network interface devices for example to enable data or information generated by the data processing system 114 to be transmitted to other devices or systems. The processor(s) 116 may include one or more of each of a central processing unit (CPUs), graphics processing unit (GPU), neural processing unit (NPU), neural network accelerator (NNA), tensor processing unit (TPU), application-specific integrated circuit (ASIC), application-specific standard product (ASSP), digital signal processor (DSP), field programmable gate array (FPGA), system-on-chip (SoC), any other suitable form of integrated circuit. In the present disclosure, the term memory is used to encapsulate both volatile and non-volatile working memory, as well as non-volatile storage. The memory 118 isconfigured to store measurement signals generated by the sensor device 110, as well asprogram code and a machine learning model for implementing the methods describedhereinafter. The program code may include source code, object code, firmware, etc. in any suitable language. For example, the source code may be written in Python, C, C++, Rust, Julia, etc and may use specific development frameworks, libraries, or packages, including PyTorch, TensorFlow, Keras, CUDA, etc. Fig. 2 shows an example of a machine learning model for use in analysing asignal 202 such as a measurement signal determined using the apparatus of Fig. 1. Thesignal 202 may include raw measurements of a polymer by a sensor element asdescribed above. The signal 202 may include a sequence of raw measurements or mayhave undergone pre-processing such as normalisation, outlier removal, amplificationetc. The signal 202 may have a first frequency corresponding to the number ofmeasurements taken by the sensor element per unit time. The frequency may for example be of the order of thousands of measurements per seconds or Hertz, for example 1kHz, 2kHz, 5kHz, 10kHz or any other frequency sufficiently high multiple measurements to be taken as each polymer unit passes the nanopore. The average rate at which polymer units pass the nanopore may for example be of the order of hundreds of polymer units per second, such as 100 units / s, 200 units / s, 400 units / s, 800 units / s.For a given rate, the measurement frequency may be chosen to be high enough tocapture a maximal degree of useful information about the polymer units beforesaturation. The frequency of the signal 202 may for example be in the region of ten or more times higher than the average rate at which the polymer units pass the nanopore. In a particular example, the measurement frequency is 5kHz and the average rate of polymer units passing the nanopore is 400 units / s. The signal 202 in this example is processed by one or more neural networklayers 204, which in this example include ^ one-dimensional convolutional layers eachapplying one or more kernels to generate a respective sequence of feature vectors 206. The kernels may be applied sequentially to the signal 202 resulting in the sequence of feature vectors 206 being generated sequentially, for example in substantially real time as the signal 202 is generated, or alternatively the processing at different kernel positions may be parallelised across multiple processor cores. At least one of the convolutional layers may use a stride greater than one. Thewidth of the kernel and the stride at each layer may be chosen such that portions of thesignal processed at successive positions of the kernel overlap, meaning that every measurement in the signal 202 is processed by the kernel in at least one position. The number of feature vectors 206 may be less than the number of measurements in thesignal 202, or in other words the downsampled signal may have a lower frequency thanthe signal 202. Successive convolutional layers may incrementally reduce thefrequency, for example by each having a stride greater than one. The frequency of thesequence of feature vectors 206 may be between 5 and 20 times less than the frequencyof the signal 202. For example, the frequency of the sequence of feature vectors 406 may be 12 times less than the frequency of the signal 202, which may be achieved usingtwo convolutional layers with stride two and one convolutional kernel with stride three(in any order), or alternatively using one convolutional layer with stride four and oneconvolutional layer with stride three. It will be appreciated that other combinations may be used to achieve other degrees of downsampling, and stride-one convolutionsmay also be included to increase the expressiveness of the feature representationsencoded by the feature vectors 206. The kernel used in a given convolutional layer mayfor example have a depth of 2, 4, 816, or any other suitable depth. Other types of layersor operations may additionally, or alternatively, be used in the downsampling processand / or otherwise to generate the sequence of feature vectors 206, such as pooling layersor predetermined filters. In some examples, the feature vectors 206 may be generatedwithout any degree of downsampling. The sequence of feature vectors 206 is processed by a transformer encoder network comprising one or more transformer encoder blocks 208. In the example ofFig. 2, the transformer encoder network includes ^ transformer encoder blocks 208each including an attention layer 210 and a feed forward network 212. The number ^of transformer encoder blocks may be chosen to balance possibly competing requirements such as accuracy, speed, memory footprint, compute, and need for highvolumes of training data. In some examples, ^ may be between 8 and 64 (inclusive),or between 12 and 20 (inclusive), though it will be appreciated that more or fewertransformer encoder blocks may be appropriate in particular settings, such as when very large or small volumes of training data are available, or when accuracy is paramount irrespective of compute cost, or when sequencing is to be performed on a mobile device and / or substantially in real time. The purpose of the attention layer 210 is to capture relationships betweenvectors in a sequence. More precisely, for a given input vector the attention layer 210may generate a corresponding output vector that capture relationships between thegiven input vector and other input vectors within a receptive field of the correspondingoutput vector. The corresponding output vector may be said to attend to the inputvectors within its receptive field. For example, the attention layer 210 of a giventransformer encoder block 208 may be configured to process a sequence of ^ inputvectors ^^ to generate a sequence of ^ output vectors ^^ of dimension ^. The outputvectors ^^ may be computed using a scaled dot-product attention kernel according tothe following equation: ^^ = softmax where with ^^ denoting the receptive field of the attention layer 210 for the output vector ^^.The matrices ^^ , ^^ , ^^ are learned weight matrices arranged to transform a giveninput vector ^^ into a query, key, or value respectively, while the constant ^ is the innerdimension of the queries and keys. The output vector ^^ may be a sum of the valuesweighted by the scaled dot-product similarity of the keys and queries.Prior to the processing by the attention layer 210, a positional encodingtechnique may be applied to encode positional information about the respective sequence positions of the input vectors. For example, fixed or learned positional encodings, such as sinusoidal positional encodings, may be added to the input vectors. As another example, rotary position embeddings (RoPE) may be applied, in which thequeries and keys in the attention layer 210 are rotated, with each position in thesequence receives a different rotation. This way, the dot-product between queries andkeys may diminish for tokens that are distant from one another in the sequence,providing an effective means of encoding relative positions. Compared with additivepositional encodings, RoPE has been found to maintain a greater degree of the originaltoken information while still providing the model with an effective way to understand sequence positions. In another example, attention with linear biases (ALiBi) may be applied, in which fixed sequence of bias values are added to the key-query dot-product within the attention layer, or adaptive ALiBi which replaces the fixed sequence with learned additive biases. The attention layer 210 may be a multi-headed attention layer which appliesmultiple attention functions (or “heads”) in parallel, with different learned weightmatrices, and combines the resulting vectors to give a final output. For example, theoutput of the attention layer 210 may be determined by concatenating the outputs of the individual attention functions and projecting them to an output matrix comprising the sequence of output vectors. For global attention, the receptive field for each input vector may be the entiresequence, such that ^^ = ^ With global attention, the attention layer 210may in principle be able to capture all possible relationships between the input vectorsof the transformer encoder block 208, including possible long-range correlations orpatterns. However, global attention leads to a computational complexity that growsquadratically with the number of input vectors (i.e. ^(^^)), which may be inefficientand possibly prohibitive for sequencing very long polymers. To mitigate such issues,alternative topologies may be used for the attention layer 210, such as sparse topologies. A particularly advantageous implementation of the attention layer 210 for the present purpose is sliding window self-attention, in which the receptive field for a givenoutput vector is a contiguous sequence of inputs vectors of fixed size. For example, thereceptive field for an output vector ^^ may be given by ^^ = where the constants ^^and ^^denote the extent of the sliding window on either sideof the ^^^ input vector. A symmetric sliding window of size ^ may have ^^ = ^^ =^ / 2. Sliding window attention has the advantage that the computational complexitygrows only linearly with the length of the sequence, ^(^^), while the memory footprintremains constant, because in effect the transformer encoder processes only a fixed- length sequence of the input signal at a time. As a result, long range correlations and patterns within the signal 202 may not be captured, but the inventors have found that a window of sufficient width, such as between 64 and 512, may be sufficient to capture the relevant information for sequencing or basecalling, whilst performance improvements may saturate beyond such window sizes. By using sliding window self- attention, arbitrarily long polymers may be sequenced without performance deterioration or prohibitive compute scaling. This contrasts with existing applications of transformers, such as natural language processing, in which performance can suffer significantly if long-range correlations are neglected. It will be appreciated that different window sizes or topologies may be used for the attention layer 210 in different transformer encoder blocks 208, or for differentheads within a given transformer encoder block 208. For example, diluted slidingwindow self-attention may be used in some blocks / heads, in which gaps are introducedinto the windows, with different gaps being used for different blocks / heads to enabledifferent correlations to be captured at different blocks / heads. The sequence of output vectors generated by the attention layer 210 may beprocessed by a feed forward network 212. The feed forward network 212 may apply anon-linear function independently to each output vector ^^ generated by the attentionlayer 210. The non-linear function may for example be composed of two learned lineartransformations, with a rectified-linear unit (ReLU) activation function applied betweenthe two learned linear transformations, as follows:FFN^^^^(^^, ^^, ^^, ^^, ^^) = max(0, ^^^^ + ^^) ^^ + ^^,where ^^, ^^are learned weight matrices, and ^^, ^^are (optional) learned bias vectors. Depending on the dimensions of these vectors and matrices, the feed forward network 212 may effectively expand and contract the size of the internal state. Expanding the internal state can enable the transformer encoder block 208 to be more expressive, at the cost of a greater number of parameters and therefore higher compute and memory requirements, while returning the internal state to its original dimensions enables arbitrary numbers of transformer encoder blocks 208 to be stacked. Theexpansion factor, which may for example be defined as the ratio |^^| / |^^| ofdimensions between the hidden vectors, may be in the region of the order of 2, 5, 10 or more. Other implementations of the feed forward network 212 are possible, for example in which the ReLU activation function is replaced by a Gaussian error linear unit (GeLU) or the Swish function. Other implementations are based on the Gated Linear Unit (GLU), which employs a component-wise product of two linear transformations of the input, one of which is sigmoid-activated. The sigmoid activation may also be omitted, resulting in a so-called bilinear feed forward network. In aparticularly preferred example, the feed forward network 212 comprises a sigmoid-weighted gated linear unit (SwiGLU), which employs a component-wise product of two linear transformations of the input, one of which is swish-activated, as shown below: FFN^^^^^^(^^, ^^, ^, ^^) = (Swish^(^^ ^) ⊗ ^^^)^^,where ^^, ^^, ^ are learned weight matrices and the Swish function is defined asSwish ^(^) = It will be appreciated that variations on these functions arepossible, for example by adding learned bias vectors or changing the value of theparameter ^ of the SwiGLU function.The transformer encoder block 208 may perform additional operations. Forexample, the transformer encoder block 208 may apply residual connection(s) inparallel with the attention layer 210 and / or the feed forward network 212, in which theinput vector is added to the output of the attention layer 210 and / or feed forwardnetwork 212. Residual connections mainly help mitigate vanishing gradients duringtraining, whilst also encouraging the feed forward network 212 to learn representations that retain local information, which may be particularly valuable in the present settingwhere local effects may be more important than long-range effects. Additionally, oralternatively, the transformer encoder block 208 may include one or more normalisationfunctions, which may reduce training time and improve stability by preventingexploding parameter values. Examples of normalisation functions include batchnormalisation and layer normalisation. Normalisation functions may be applied aftereach of the attention layer 210 and the feed forward network 212 (so-called post-norm),or before each of the attention layer 210 and the feed forward network 212 (so-calledpre-norm). The output of the final transformer encoder block 208 is a sequence of intermediate hidden states 214. The number (or frequency) of intermediate hidden states 214 may be equal to the number (or frequency) of feature vectors 206 generated by the downsampling neural network layers 204. Depending on the level ofdownsampling applied by the neural network layers 204, the frequency of theintermediate hidden states 214 may be comparable to, or even less than, the rate atwhich polymer units are translocated relative to the nanopore. It may therefore not bepossible to map individual intermediate hidden states to polymer units for classification without the possibility of some polymer units being missed, despite the hidden states 214 encoding sufficient information to identify all of the polymer units. To mitigate this issue, the machine learning model may apply an upsampling layer 216 to obtain asequence of upsampled hidden states 218 which may have a sufficiently high frequencythat at least one hidden state will map to each individual polymer unit. The frequencyof the sequence of upsampled hidden states 218 may be greater than the averagetranslocation rate, for example between 1.5 and 5 times the average translocation rate. The upsampling layer may increase the frequency by a factor of 2, 5, 8, or any othersuitable upsampling factor depending on the frequency of the sequency of intermediatehidden states and the average (or minimum) translocation rate. The upsampling layer 216 may be any layer or functional capable of increasing the frequency or sampling rate of a sequence while maintaining information within the sequence. For example, the upsampling layer 216 may be a convolutional upsamplinglayer, an attention upsampling layer, or a linear upsampling layer. Linear upsamplingmay be performed by increasing the dimension of an intermediate hidden state 214 ofshape (1, ^) to have a stretched shape of (1, ^^), where ^ is the upsampling factor, thenreshaping the resulting vector from shape (1, ^^) to (^, ^). The upscaling factor maybe, for example 2, 3, 5, or any other value to achieve a suitable frequency for the upsampled hidden states 218. Linear upsampling has the advantage of beingcomputationally less expensive than convolutional upsampling or attention upsampling.The upsampled hidden states 218 are processed using a network head 220 toestimate a sequence of polymer units of the polymer. The network head 220 in this example includes a projection layer 222 for projecting each upsampled hidden state 218to a respective array of values 224 indicating an estimated likeliness of the hidden state218 being associated with each of a set of symbols, the set of symbols comprising symbols associated with respective types of canonical polymer unit. The likeliness arrays may be vectors. Within the meaning of the present disclosure, the term likeliness may refer to a measure of probability or any other quantity representing how likely an event is to happen (possibly, but not necessarily, normalised within a range of zero to one). The set of symbols may correspond to canonical nucleotide bases such as adenine(A), cytosine (C), guanine (G) and thymine (T) for DNA basecalling, or adenine (A),cytosine (C), guanine (G) and uracil (U) for RNA basecalling. For other types ofpolymer, other symbols may be included. The network head 220 analyses the arrays of values 224, for example byapplying a search algorithm to the arrays of values 224, to estimate the sequence ofpolymer units of the polymer. As explained above the frequency of the hidden state218 arriving at the network head may be higher than the translocation rate of thepolymer units, meaning that there is not a one-to-one mapping of arrays 224 to polymer units. To address this issue, the network head may adopt a connectionist temporal classification (CTC) topology, and may therefore be referred to as a CTC head or a conditional random field-CTC (CRF-CTC) head, depending on the arrays 224 are generated and how the network head 220 is trained. As discussed in more detail hereinafter, using a CTC-CRF head may lead to improved accuracy as it may better capture conditional dependencies between hidden states 218 than a “vanilla” CTC head.According to the CTC topology, an additional “blank” symbol (e.g., isintroduced to the set of symbols, and the algorithm includes determining and collapsingan intermediate sequence of symbols to arrive at a final estimate. For example, thesearch algorithm may collapse the intermediate sequence symbols by (1) mergingrepeated symbols and then (2) removing instances of the blank symbol. In the exampleof Fig. 2, an intermediate sequence 226 of symbols corresponding to DNA nucleotidebases is given by CGG-A-TT. Repeated symbols (i.e. neighbouring symbols that areidentical to one another) are merged, resulting in a partially collapsed sequence 228 of symbols given by CG-A-T. Instances of the blank symbol are then removed to determine a fully collapsed sequence 230, which may be returned as the estimated sequence of polymer units. The network head 220 may determine the intermediate sequence 226 of symbolsby select the most likely symbol from each array 224, which may result in theintermediate sequence with the highest possible posterior probability. This may in turn result in the estimated nucleotide sequence with the highest posterior probability, whichis the desired behaviour of the network head 220. However, in some cases multipleintermediate sequence may map to a common fully collapsed sequence (for example,intermediate sequences CGG-A-TT and CCGA-TT both map to a fully collapsed sequence CGAT). Accordingly, the intermediate sequence with the highest posterior probability may not correspond to the fully collapsed sequence with the highestposterior probability. The relevant metric is the cumulative posterior probability for allintermediate sequences corresponding to a given collapsed sequence. However, the computational complexity of any algorithm capable of finding the highest cumulative posterior probability results in a combinatoric explosion which prohibits use of suchalgorithms for long sequence of polymers. As an alternative, a heuristic known as abeam search may be used, which moves along the sequence of arrays 224 in a sequentialmanner, computing an updated set of hypotheses at each step. The updated set ofhypotheses at each step is derived from the set of hypotheses at the previous set byextending each hypothesis with all possible keeping only the top ^ candidates, where ^is a hyperparameter referred to as the beam width. The beam width ^ may for examplebe 2, 4, 8, 16, 32, 64, or any other suitable size. With the CTC topology, a score (e.g.,probability) is determined at each step after collapsing repeated symbols and removingblank characters. In some examples, separate scores may be tracked for hypotheses thatend in the blank symbol and do not end in the blank symbol, and these separate scores may be combined to determine which hypothesis branches to maintain. In the case of a beam search, the intermediate sequence 226 may correspond to the remaining branch after the algorithm has run through all of the arrays 224. It will be appreciated that other network heads may be used in place of the CTChead or CTC-CRF head described above, such as a transformer decoder head whichmay be arranged to generatively predict the sequence of polymers using the set ofhidden states 218 as context. However, compared with a CTC-based head or anothertype of relatively light-weight head, a transformer decoder head would includeadditional blocks of attention layers and feed forward networks, which may significantly increase the computationally complexity of performing inference, while the additional trainable parameters may result in more training runs being needed to achieve a comparable level of accuracy. To summarise, Fig. 3 shows an example of a computer-implemented method300 of analysing a signal comprising measurements of a polymer by a sensor elementcomprising a nanopore during translocation of the polymer with respect to thenanopore. The method 300 may use a neural network model comprising a transformer encoder network and a network head, as described above with reference to Fig.2. The method 300 includes determining, at 302, a sequence of hidden states associated withthe signal using the transformer encoder network, and processing the sequence ofhidden states using the network head to estimate a sequence of polymer units of thepolymer. The processing by the network head includes generating, at 304, a respectivearray of values for each hidden state indicating an estimated likeliness of the hidden state being associated with each of a set of symbols, where the set of symbols includessymbols associated with respective types of canonical polymer unit, and analysing thegenerated arrays of values to estimate, at 306, the sequence of polymer units of thepolymer. Fig. 4 shows an example of a computer-implemented method 400 of training aneural network model comprising a transformer encoder network and a network head for use in the computer-implemented method 300 of Fig.3. The method 400 includesproviding, at 402, a training signal comprising measurements of a polymer by a sensor element comprising a nanopore during translocation of the polymer with respect to thenanopore. The training signal being associated with a target sequence of polymer units.The method 400 includes determining, at 404, a sequence of hidden states associatedwith the signal using the transformer encoder network, processing, at 406, the sequenceof hidden states using the network head to generate a respective array of values indicating an estimated likeliness of each hidden state being associated with each of a set of symbols comprising symbols associated with respective types of canonicalpolymer unit, and updating, at 406, parameters of the neural network based on a lossfunction that depends on the generated arrays of values and the target sequence ofpolymer units. The training signal and associated target sequence may be obtained by measuring a polymer with a known sequence using the sensor element, or by measuring a polymer with an unknown sequence using the sensor element and using an alternative sequencing method to determine the target sequence. In either case, the training signal and the target sequence may not necessarily be aligned, meaning that there may not be a known correspondence between positions in the target sequence and sections of the training signal. The training signal may be part of a dataset comprising a large number of training signals each having respective target sequences. The dataset may for example include hundreds, thousands, tens of thousands, hundreds of thousands, or millions of training signals. The dataset may include polymers of a given class, such as polynucleotides, DNA, RNA, animal DNA, plant DNA, animal RNA, plant RNA, etc. or may include polymers of a wide range of classes. The updating of the parameters may be performed using a gradient-based optimisation method, in which one or more loss values corresponding to evaluations of the loss function a batch of one or more training signals are backpropagated through the neural network to determine the gradient of the loss (or losses) with respect to the parameters of the neural network. Values of the parameters (e.g., weights, biases) may be updated using a gradient-based optimiser, such as stochastic gradient descent (SDG), Adam, RMSProp, or AdamW. The optimiser may have one or more hyperparameterssuch as learning rate, which may be held constant or may be varied according to apredetermined or learned schedule. For example, learning rate warm-up may be applied in which the learning rate increases gradually from a very small value in order to mitigate issues associated with exploding gradients in early phases of the training. Techniques such as batch normalization may be used to improve the efficiency of the training process. This process may be repeated over many iterations covering one or more passesthrough the dataset (epochs), for example until a stopping condition is satisfied, such asa convergence condition or after a predetermined number of iterations or epochs (suchas 10, 30, 50 or 100 epochs), at which point the neural network may be validated and tested using validation and test datasets before being deployed for use. For a signal ^ and associated target sequence ^ within the training dataset, theloss function may include a CTC loss corresponding to a negative log-likelihood of thetarget sequence ^ given the signal ^, i.e. ^^^^(^, ^) = − log^^(^|^)^. For longsequences, this term (and its gradient) can be prohibitively expensive to compute usingthe naïve approach of computing and summing the probabilities of each possiblealignment. However, the CTC loss can be calculated efficiently using a dynamicprogramming algorithm referred to as a forward-backward algorithm, in which aforward variable and backward variable are recursively calculated during forward andreverse passes through the sequence of hidden states and then used to determine the loglikelihood. Apossible drawback of using the CTC loss is that it works on the basis thatevery symbol is conditionally independent of every other symbol given the sequence ofhidden states, and may therefore fail to leverage information pertaining to commonlyoccurring patterns or correlations in the symbols. To alleviate this issue, a CTC-CRFloss may be used, which may result in the arrays of values being determined as a CRFwhich can capture conditional dependencies between symbols. The CRF models theconditional probability of a hidden state sequence ^ given an input sequence ^ asfollows: where ^(^, ^) = ∑^ log ^(^^|^) + log ^(^) is a potential function.The hidden state sequence ^ is connected to the target sequence ^ by a mapping^: ^^ → ^^ where ^^ and ^^ are the symbol sets for the hidden states and targetsequence respectively. The CTC-CRF loss may then be given by: ^^^^^^^^ = − log ^ ^(^, ^|^) = − log^ The CTC-CRF loss can also be computed using a forward-backward algorithm. Insome examples, the loss function may include a weighted linear combination of theCTC loss and the CTC-CRF loss (for example, with the CTC loss having a relatively lower weight such as 0.01 or 0.1 times the weight of the CTC-CRF loss). Includingboth loss terms may help with convergence during training.Regularisation and other techniques may be applied during training of the neuralnetwork model, for example to improve the convergence properties, stability and speedof the training. Examples include layer drop in which certain layers are randomly dropped with a predetermined probability during training, and dropout in which certain parameters such as elements of the learned matrices are randomly zeroed with a predetermined probability during training. A further option is batch normalisation. Suitable batch sizes for training may be in the region of 32, 64, 128, or 256 training signals. The above embodiments are to be understood as illustrative examples of the invention. Further embodiments of the invention are envisaged. For example, while the methods described above may be applied to single strand of a polynucleotide, in other examples a signal or signals may include measurements of multiple polymer strands, for example two complementary strands of a polynucleotide. In such cases, thetransformer encoder network may be adapted to process measurements of both signals.By leveraging the information from both complementary strands, accuracy may befurther improved as the effects of measurement errors, noise, and other sources of error may be mitigated by the inherent redundancy of information between the strands. Fig. 5A shows an example of a composite signal formed of a first signal portion502 concatenated with a second signal portion 504. The first signal portion 502 comprises measurements of a first polynucleotide strand and the second signal portion 504 comprises measurements of a second polynucleotide strand, where the first and second polynucleotide strands are complementary strands. The start of the first signalportion 502 is indicated by a first dashed line 506, while the start of the second signalportion 504 is indicated by a second dashed line 508. The two signal portions are at least partially aligned in the sense that the start of the first signal portion 502 and the start of the second signal portion 504 correspond to approximately equal locations within the respective polynucleotide strands. The concatenated signal portions 502, 504 may be processed according to the methods described above to generate a sequence of feature vectors 510. In this example, the sequence of feature vectors 510 is processed using one or more transformer encoder blocks having a sliding window cross- attention layer 512. The attention layer 512 in this example has a receptive field corresponding to two sliding windows 514 and 516. For feature vectors 510 derived from the first signal portion 502, the receptive field may be given by ^(^) ^ = the number offeature vectors derived from each signal portion. For feature vectors 510 derived from the second signal portion 504, the receptive field may be given by = In this way, output of theattention layer 512 attends to feature vectors within both sliding windows, allowing relationships between the first signal portion 502 and the second signal portion 504 to be captured. This dual sliding window approach may benefit from the advantages of sliding window attention described elsewhere in the present disclosure, provided the degree of misalignment is sufficiently small that corresponding parts of the first signal portion 502 and the second signal portion 504 concurrently fall within the first and second sliding windows 514, 516. Fig. 5B illustrates an alternative method of analysing the composite signal ofFig. 5A. In this example, one or more transformer encoder blocks use a global self- attention layer 518 with a receptive field covering the feature vectors 510 from bothsignal portions, such that ^^ (assuming ^^feature vectors are derived from each of the signal portions 502, 504). The global self-attention layer 518 advantageously may not require any alignment between the first signal portion 502 and the second signal portion 504, so may be applicable in settings where such alignment cannot be readily obtained. Still further embodiments are envisaged. For example, further modifications to the neural network architectures discussed above may be employed, for exampleconfiguring one or more of the transformer encoder blocks as a mixture of experts toachieve more efficient training, and / or quantising one or more parts of the neuralnetwork model. Furthermore, additional steps may be carried out in dependence on the estimated sequence of polymer units of the polymer. For example, the estimatedsequence may be output to a display or to a file / memory, or to another softwareapplication. Additionally, or alternatively, the estimated sequence may be used todetermine whether to eject the polymer being sequenced, and / or to translocate thepolymer at the same or a faster rate without making further electrical measurements,for example to mitigate use of computational resources on h It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

Claims

CLAIMS1. A computer-implemented method of estimating a sequence of polymer units ofa polymer comprising measurements of a signal generated during translocation of thepolymer with respect to a nanopore, comprising: determining, using a transformer encoder network, a sequence of hidden states associated with the signal; processing the sequence of hidden states using a network head to estimate asequence of polymer units of the polymer, the processing comprising: generating, for each hidden state, a respective array of values indicating an estimated likeliness of the hidden state being associated with each of a set of symbols, the set of symbols comprising symbols associated with respective types of canonical polymer unit; and analysing the generated arrays of values to estimate the sequence ofpolymer units of the polymer.

2. The computer-implemented method of claim 1, wherein the transformerencoder network uses sliding window self-attention, whereby each hidden state depends on a respective portion of the signal.

3. The computer-implemented method of claim 2, wherein the polymer comprisesat least 100,000 polymer units.

4. The computer-implemented method of claim 2 or claim 3, wherein the slidingwindow attention uses a window size of between 64 and 512.

5. The computer-implemented method of any preceding claim, whereindetermining the sequence of hidden states comprises: processing, using one or more neural network layers, the signal to generate asequence of feature vectors; andprocessing, using the transformer encoder network, the sequence of feature vectors to determine the sequence of hidden states.

6. The computer-implemented method of claim 5, wherein:polymer units of the polymer pass through the nanopore at an averagetranslocation rate;the signal has a first frequency that is greater than the average rate; the sequence of feature vectors has a second frequency that is lower than thefirst frequency; andprocessing the sequence of feature vectors comprises: generating, as an output of the transformer encoder network, a sequence of intermediate states having the second frequency; and upsampling the sequence of intermediate states to determine the sequence of hidden states, wherein the sequence of hidden state has a third frequency that is at no less than the average translocation rate.

7. The computer-implemented method of claim 6, wherein the third frequency isless than the first frequency.

8. The computer-implemented method of claim 6 or claim 7, wherein the thirdfrequency is between 1.5 and 5 times the average translocation rate.

9. The computer-implemented method of any of claims 6 to 8, wherein the firstfrequency is at least 3 times greater than the average translocation rate.

10. The computer-implemented method of any of claims 6 to 9 wherein a ratiobetween the first frequency and the second frequency is between 5 and 20.

11. The computer-implemented method of any of claims 6 to 10, wherein theupsampling is linear upsampling.

12. The computer-implemented method of any of claims 5 to 11, wherein the oneor more neural network layers comprise one or more convolutional layers.

13. The computer-implemented method of any preceding claim, wherein:the set of symbols comprises a blank symbol; and analysing the generated arrays of values comprises:determining an intermediate sequence of symbols based on the generated arrays of values; and collapsing the intermediate sequence of symbols by merging repeated symbols and then removing instances of the blank symbol.

14. The computer-implemented method of claim 13, wherein determining theintermediate sequence of symbols comprises performing a beam search based on the generated arrays of values.

15. The computer-implemented method of any preceding claim, wherein thenetwork head applies a linear projection to the sequence of hidden states to generate thearrays of values.

16. The computer-implemented method of any preceding claim, wherein thenetwork head is configured to generate the arrays of values as a conditional random field (CRF).

17. The computer-implemented method of any preceding claim, wherein:the polymer is a first polymer; the signal further comprises measurements of a second polymer by the sensor element during translocation of the second polymer with respect to the nanopore, the first polymer and the second polymer being a complementary pair; and the transformer encoder network comprises an attention mechanism arranged to capture relationships between measurements associated with mutually corresponding segments of the first polymer and the second polymer.

18. A computer-implemented method of training a neural network modelcomprising a transformer encoder network and a network head, the method comprising: providing a signal comprising measurements of a polymer by a sensor element comprising a nanopore during translocation of the polymer with respect to the nanopore, the signal being associated with a target sequence of polymer units; determining, using the transformer encoder network, a sequence of hidden states associated with the signal; processing the sequence of hidden states using the network head to generate, for each hidden state, a respective array of values indicating an estimated likeliness of the hidden state being associated with each of a set of symbols, the set of symbols comprising symbols associated with respective types of canonical polymer unit; and updating parameters of the neural network based on a loss function depending on the generated arrays of values and the target sequence of polymer units.

19. The computer-implemented method of claim 19, wherein the loss functioncomprises at least one of a connectionist temporal classification (CTC) loss and a connectionist temporal classification-conditional random field (CTC-CRF) loss.

20. The computer-implemented method of any preceding claim, wherein thepolymer is a polynucleotide, and the polymer units are nucleotides.

21. The computer-implemented method of any preceding claim, wherein thesignal is an electrical signal.

22. The computer-implemented method of any preceding claim, wherein thesignal is indicative of ion flow through the nanopore.

23. A computer program product comprising instructions which, when the programis executed by a computer, cause the computer to carry out the method of any preceding claim.

24. A data processing system comprising means for carrying out the method of anyof claims 1 to 22.

25. Apparatus comprising:a sensor element comprising a nanopore; means for translocating a polymer with respect to the nanopore; and a data processing system comprising means for carrying out the method of any of claims 1 to 17 to analyse a signal comprising measurements of a polymer by the sensor element during translocation of the polymer with respect to the nanopore.

26. A method comprising:translocating a polymer with respect to a nanopore; generating a signal comprising measurements of the polymer by a sensor element comprising the nanopore; and analysing the signal using the computer-implemented method of any of claims 1 to 17.

Images