Method and system for recognizing nanopore sequencing signal based on transformer

Abstract

The application is suitable for the field of gene sequencing technology, and provides a nanopore sequencing signal recognition method and system based on a Transformer, which comprises: normalizing nanopore sequencing signals and dividing the signals into fixed-length signal windows; extracting signal features through a Transformer encoder comprising a convolution layer and a multi-head self-attention module; adopting an autoregressive Transformer decoder to combine the signal features and context information of a decoded base sequence, and generating prediction probability of a current base through a cross-attention mechanism; iteratively generating an optimal base sequence based on a beam search algorithm, and optimizing decoding efficiency in combination with a key-value cache; and fusing and quality-controlling segmented decoding results, and outputting a final sequence. The application significantly improves the accuracy and reliability of nanopore sequencing in complex biological sequence analysis, and the recognition accuracy on various gene sequencing data sets is higher than that of existing CTC-based models.

Description

Technical Field

[0001] This invention relates to the field of gene sequencing technology, specifically to a method and system for identifying nanopore sequencing signals based on Transformer. Background Technology

[0002] Early base identification tools (such as Nanocall) used Hidden Markov Models (HMMs) to match base sequences through event segmentation and state transition probabilities, but these relied on manual feature engineering and were inefficient for processing long-read signals. In recent years, deep learning-based Connectionist Temporal Classification (CTC) models (such as Chiron and Guppy) have become mainstream. These models extract signal features through an encoder (usually CNN+RNN) and use the CTC loss function to achieve end-to-end sequence alignment, avoiding explicit segmentation steps. However, existing CTC-based schemes have inherent drawbacks: Conditional Independence Assumption: CTC assumes that the predictions at each time step are independent of each other, ignoring the biological associations between bases (such as codon preference and homopolymer continuity), which leads to a significant increase in the error rate of identifying complex regions (such as long homopolymers and repetitive sequences). However, some advanced commercial base calling tools, such as Guppy, Chiron, and Bonito, while improving performance through modifications to the encoder architecture, are limited by the inherent limitations of the CTC framework. RODAN, a successful third-party open-source base calling tool, employs a fully convolutional architecture and a CTC decoder, ensuring efficient parallel processing, but still faces the problem of signal-sequence context fragmentation. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for identifying nanopore sequencing signals based on Transformer, in order to solve the problems mentioned in the background art.

[0004] This invention is implemented as follows: a nanopore sequencing signal recognition method based on Transformer, which includes the following steps: Step 1: Normalize the nanopore sequencing signal and divide it into signal windows of fixed length; Step 2: Extract signal features using a Transformer encoder that includes convolutional layers and a multi-head self-attention module. The Transformer encoder uses rotational position encoding to embed temporal information. Step 3: Using an autoregressive Transformer decoder, the predicted probability of the current base is generated by combining signal features with the contextual information of the decoded base sequence through a cross-attention mechanism. Step 4: Iteratively generate the optimal base sequence based on the beam search algorithm, and optimize decoding efficiency by combining key-value caching; Step 5: Perform overlapping region fusion and quality control on the segmented decoding results, and output the final sequence.

[0005] A Transformer-based nanopore sequencing signal recognition system is used to perform the Transformer-based nanopore sequencing signal recognition method as described above. The system includes: Signal preprocessing module: used to normalize nanopore sequencing signals and segment overlapping windows; CNN-Transformer encoder module: used to extract local features of the signal through convolutional layers, and combine multi-head self-attention and rotation position encoding to model the global context information of the signal; Transformer decoder module: used to generate base prediction probabilities through self-attention and cross-attention mechanisms; Beam search decoding module: used to combine key value caching and dynamic masking to generate the optimal base sequence; The result fusion module is used to perform overlapping region fusion and quality control on the segmented decoding results.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method and system for identifying nanopore sequencing signals based on Transformer, which significantly improves the accuracy of base identification by combining the dual context of sequence and signal. This method achieves higher recognition accuracy than existing CTC-based models on various gene sequencing datasets; This invention significantly improves the accuracy and reliability of nanopore sequencing in the analysis of complex biological sequences and is applicable to long-read sequencing devices such as Oxford Nanopore Technologies (ONT). Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the overall structure of the CNN-Transformer signal encoder network in this invention.

[0008] Figure 2 This is a schematic diagram of the Transformer coding layer network structure of the signal encoder in this invention.

[0009] Figure 3 This is a schematic diagram of the overall structure of the Transformer decoder network in this invention.

[0010] Figure 4This is a schematic diagram of the self-attention layer network structure in the Transformer decoder of this invention.

[0011] Figure 5 This is a schematic diagram of the cross-attention layer network structure in the Transformer decoder of this invention.

[0012] Figure 6 This is a flowchart of the training process for the base prediction model in this invention.

[0013] Figure 7 This is a graph showing the cross-entropy loss curve during training of the base prediction model in this invention.

[0014] Figure 8 The flowchart shows the transfer learning process for adding monotonic alignment constraint loss to the base prediction model in this invention.

[0015] Figure 9 This is a preliminary diagram of the diagonally aligned cross-attention probability matrix established in this invention.

[0016] Figure 10 This is a graph of the monotonic alignment constraint loss function during the early training phase of this invention.

[0017] Figure 11 This is a flowchart of the training process for the k-mer prediction model in this invention.

[0018] Figure 12 This is a schematic diagram of the main process of sequencing signal recognition in this invention.

[0019] Figure 13 This is a schematic diagram of signal segmentation using the overlapping window strategy in this invention.

[0020] Figure 14 This is a diagram of the attention mask matrix of the decoder's self-attention layer and the last cross-attention layer in this invention.

[0021] Figure 15 This is a diagram of the beam search process in this invention.

[0022] Figure 16 This is a diagram illustrating the beam search path update process in this invention. Detailed Implementation

[0023] The present invention will be further explained below with reference to specific embodiments.

[0024] Please see Figures 1-16 The present invention provides a method for identifying nanopore sequencing signals based on Transformer, characterized in that the method includes the following steps: Step 1: Normalize the nanopore sequencing signal and divide it into signal windows of fixed length; Step 2: Extract signal features using a Transformer encoder that includes convolutional layers and a multi-head self-attention module. The Transformer encoder uses rotational position encoding to embed temporal information. Step 3: Using an autoregressive Transformer decoder, the predicted probability of the current base is generated by combining signal features with the contextual information of the decoded base sequence through a cross-attention mechanism. Step 4: Iteratively generate the optimal base sequence based on the beam search algorithm, and optimize decoding efficiency by combining key-value caching; Step 5: Perform overlapping region fusion and quality control on the segmented decoding results, and output the final sequence.

[0025] A Transformer-based nanopore sequencing signal recognition system is used to execute the above-described Transformer-based nanopore sequencing signal recognition method. The system includes: Signal preprocessing module: used to normalize nanopore sequencing signals and segment overlapping windows; CNN-Transformer encoder module: used to extract local features of the signal through convolutional layers, and combine multi-head self-attention and rotation position encoding to model the global context information of the signal; Transformer decoder module: used to generate base prediction probabilities through self-attention and cross-attention mechanisms; Beam search decoding module: used to combine key value caching and dynamic masking to generate the optimal base sequence; The result fusion module is used to perform overlapping region fusion and quality control on the segmented decoding results.

[0026] In embodiments of the present invention, the core innovations include: overcoming the limitations of traditional CTC models that rely solely on signal information and the assumption of sequence conditional independence, constructing a Transformer-based encoder-decoder network architecture, and introducing dual contextual dependencies of signals and sequences in recognition, specifically including: 1. Hybrid Feature Signal Encoder: Employs a CNN-Transformer hybrid architecture, where the CNN captures local signal features and the Transformer models global signal context information; 2. Context-Aware Decoder: This decoder obtains the contextual feature representation of the sequence through a self-attention mechanism, explicitly establishes the alignment relationship between the signal and the sequence through a cross-attention mechanism, and dynamically fuses the temporal context of the sequencing signal with the biological context of the base sequence. During autoregressive decoding, the prediction probability of each base is determined by the joint distribution of the signal feature vector and the generated sequence. 3. Beam search algorithm and parallel attention computation further improve the decoding efficiency of the model.

[0027] The overall process of this invention includes: 1. Details of Transformer-based encoder-decoder network structure 1.1 CNN-Transformer Signal Encoder: Figure 1 This is a schematic diagram of the overall structure of the CNN-Transformer signal encoder network, which consists of a convolutional downsampling module, 18 Transformer coding layers, and a final upsampling module.

[0028] 1.1.1 Convolutional Downsampling Module: The convolutional module consists of one-dimensional convolution, batch normalization, and the Swish activation function. It is used to extract local features of the signal while reducing the signal sampling length to reduce the amount of computation.

[0029] 1.1.2 Transformer Encoding Layer: The Transformer encoding layer consists of a Multiple-Head Self-Attention Module with residual connections and layer normalization, and a Multilayer Perceptron (MLP) Module, as follows... Figure 2 As shown.

[0030] The multi-head self-attention module is a key module for extracting context-dependent information from sequencing signals. Specifically, rotational position encoding embeds signal position information into signal features, and the self-attention weight matrix... , and Signal feature sequence Convert to query sequence Keyword sequence Sum sequence By calculating the cosine similarity (dot product) between each pair of query vectors and keyword vectors, the attention matrix between signal features can be obtained. , and through The normalized function is multiplied by the value vector to obtain the output signal context features. ,in Dimensions of signal features.

[0031] 1.1.3 Upsampling Module: The upsampling module uses a weight matrix... ( (as the dimension of signal features) divides the signal feature sequence from Turn to This can be further expressed as a length of Dimensions Feature sequences ( (representing the set of real numbers), thereby ensuring that the length of the signal feature sequence is strictly greater than the length of the corresponding base sequence, so as to have sufficient resolution for decoding.

[0032] 1.2 Transformer Decoder Network 1.2.1 Overall Structure: The network structure of the Transformer autoregressive decoder is as follows: Figure 3 As shown, the system comprises a word embedding network that encodes the input symbols, a self-attention layer that extracts sequence context features, and a cross-attention layer that models the alignment information between the sequence and signal context, totaling 12 Transformer decoding layers. Layers 3, 6, 9, and 12 are cross-attention layers, while the others are self-attention layers. The final decoder layer outputs the predicted probabilities of bases through a linear transformation and a softmax function.

[0033] 1.2.2 Token Embedding Network: The decoder token embedding network encodes the input base sequence (integer representation) using one-hot encoding and then uses weights... Convert it into a multidimensional feature sequence, where It is the size of the character table.

[0034] 1.2.3 Self-attention layer: The decoder's self-attention layer consists of a multi-head self-attention module with residual connections and layer normalization, and a multilayer perceptron module, such as... Figure 4 As shown. The self-attention described here is basically the same as the signal self-attention calculation in 1.1.2 above, the difference being that the self-attention matrix needs to be adjusted here. ( Add a causal mask to the base signature sequence length. ,satisfy To mask the current base characteristics For undecoded base positions Attention calculation.

[0035] 1.2.4 Cross Attention Layer: The decoder's cross attention layer includes a fusion layer and a self-attention layer, such as... Figure 5 As shown, the fusion layer consists of a multi-head cross-attention module with residual connections and layer normalization, and a multilayer perceptron module. The output of each module is scaled by a Tanh gating function before the residual connections to allow for more precise control when fusing base and signal information. The self-attention layer is the same as the modules described above.

[0036] This invention uses a soft alignment method to calculate cross-attention, performing attention calculations between base features at any position and signal features at all positions. Considering the sequential nature of sequencing, the cross-attention probability matrix between bases and signal features should exhibit a monotonically aligned "diagonal" form. Specifically, the alignment positions of consecutive bases need to satisfy a certain interval (let's say...). The monotonic alignment constraint loss function is ( For the first (The alignment position is obtained for each head, and the loss is calculated separately for each cross-attention head. Finally, the average value is taken.)

[0037] in, For the base i alignment position of the o-th attention head, Let ε be the base i+1 alignment position of the o-th attention head, ε be the alignment interval threshold, T be the signal feature length, and L be the length of the base sequence.

[0038] 2. Training dataset and training process 2.1 Training Set Construction: This invention uses nanopore sequencing data from multiple species as the training set, including synthetic data and sequencing data from multiple real species, covering animals, plants, and microorganisms. The sequencing data is re-identified using commercial software (such as Guppy), and aligned to reference sequences to determine the base sequences corresponding to the signals. Signal segmentation is then performed, and low-quality fragments are filtered out, ultimately constructing millions of signal fragment data as the training dataset. Specifically, in this embodiment, this invention uses a cross-species mixed RNA sequencing dataset to improve model generalization, including synthetic data from *Human sapiens*, *Arabidopsis*, *Caenorhabditis elegant*, *Escherichia coli*, and Curlcake. Guppy v3.4.5 is used to call the bases of the sequencing reads, and then the resquiggle command in Tombo v1.5.1 is used to align them to their respective transcriptome reference sequences to filter out low-quality reads. Subsequently, the filtered reads were processed using Taiyaki to generate an HDF5 file containing 116,072 reads, including 24,370 from Arabidopsis thaliana, 30,048 from Homo sapiens, 29,728 from synthetic constructs, 24,192 from nematodes, and 7,734 from Escherichia coli. In this embodiment, to further enhance the model's ability to identify more species, training datasets for four additional species—Mouse, Yeast (Saccharomyces cerevisiae S288C), Poplar, and Zebrafish—were added. These, along with the aforementioned training sets, form an extended training set, covering most of the iconic biological data. More details and data links for the training sets are shown in Table 1.

[0039] Table 1. Data sources for the training dataset

[0040] 2.2. Model Training Configuration The model was trained using the AdamW optimizer, an initial learning rate of 0.0002, and a batch size of 128 for a total of ten epochs. More details of the training configuration are listed in Table 2.

[0041] Table 2 Training Configuration

[0042] 2.3. Model Training Process This invention employs a cross-entropy loss function and a monotonic alignment constraint loss function to optimize parameters by minimizing the loss function. Simultaneously, it uses transfer learning to train a model with k-mer as the prediction unit. Specifically, the training process of the Nanopore direct-RNA sequencing signal recognition model provided in this embodiment includes: Step 1: Based on the signal fragments in the training set and their corresponding base sequences, train a base prediction model, such as... Figure 6 As shown. Specifically, in this embodiment, the training signal segment is a one-dimensional 32-bit floating-point sequence of length 4096, which is normalized using the MAD normalization method. The normalized signal segment is input into the CNN-Transformer signal encoder to extract the signal feature sequence. In this embodiment, the encoder first uses a convolutional module to model the local features of the signal. The convolutional module contains five convolutional layers with kernel sizes of 5, 5, 9, 9, and 5, and channel numbers of 64, 64, 128, 128, and 512, respectively. The kernel strides are 1, 1, 3, 2, and 2, respectively. The length of the feature sequence output by the convolutional module is 1 / 12 of the original signal length. Then, 18 Transformer encoding layers are used to model the contextual dependencies of the signal. Each Transformer encoding layer uses an 8-head self-attention structure, with each attention head having a dimension of 64, and positional encoding is used to introduce the temporal information of the signal. Finally, the signal feature sequence output by the upsampling module has a length of 684 and a dimension of 512. The base sequence corresponding to the training signal needs to be encoded as an integer sequence. In this encoding, the base symbols A, C, G, and T are encoded as 1, 2, 3, and 4, respectively, with 0 representing the start symbol and 5 representing the stop symbol. Taking the sequence "ACAATG" as an example, its encoding result is 01211435. During training, the decoder's input sequence is 0121143, and the prediction target is 1211435, i.e., predicting the next base symbol. The cross-entropy loss function curve during training is shown below. Figure 7 As shown.

[0043] Step 2: Based on the base prediction model obtained in Step 1, fine-tune the model by adding monotonic alignment constraint loss, such as... Figure 8 As shown. In this embodiment, the word embedding network, self-attention layer, and linear network of the signal encoder and decoder are initialized using the model parameters from step 1. The parameters of the decoder's cross-attention layer are randomly initialized. In this embodiment, only the cross-entropy loss function is used to optimize the model parameters in the first 3000 training steps. After observing a trend of the alignment constraint loss first increasing and then decreasing, confirming that the model has established a preliminary "diagonal" form of the cross-attention probability matrix (as shown). Figure 9 As shown in the figure), the alignment loss function is then introduced into the training. The monotonic alignment constraint loss function curve in the early stage of training is shown in the figure. Figure 10 As shown.

[0044] Step 3: Based on the model obtained in Step 2, fine-tune the model using k-mer as the basic symbolic unit, such as... Figure 11 As shown. In this embodiment, the model structure remains basically unchanged; only the symbol set needs to be expanded. Specifically, taking 2-mer (k=2) as an example, the symbols AA, AC, AG, AT, CA, CC, CG, CT, GA, GC, GG, GT, TA, TC, TG, and TT are encoded from 1 to 16, and A, C, G, and T are encoded from 17 to 20. 0 represents the start symbol, and 21 represents the end symbol. For example, the base sequence ACGTT can be represented as a sequence of three symbols AC, GT, and T, encoded as the integer sequence 0, 2, 12, 20, 21.

[0045] 3. Identification process and parameter settings Figure 12 This embodiment provides a basic flowchart for sequencing signal recognition, which includes the following steps: Step 1, Sequencing signal preprocessing. Nanopore sequencing electrical signals are typically saved in Fast5 file format, and their field descriptions are shown in Table 3.

[0046] Table 3 Fast5 File Format

[0047] Parse the Fast5 file to obtain the Signal, offset, range, and digitization fields, and convert the 16-bit digital signal into an electrical signal using a digital-to-analog converter.

[0048] in It is a digital signal. The offset value from the reference signal, The total span of electrical signals that can be measured within the quantization range of digital signals. The quantization order of the digital-to-analog converter. This is the final output electrical signal; Finally, the signal is normalized using the median absolute difference (MAD).

[0049] Step 2: Segment the one-dimensional signal into fixed-length signal segments. After signal segmentation, the information at the segment boundaries may be incomplete. Therefore, it is necessary to ensure that there is a certain overlap between the segments, such as... Figure 13 As shown. In this embodiment, the original signal length is The signal is divided into segments of length 4096. The overlap length between the preceding and following segments is 1200 ( If the original signal length is less than 4096, recognition will not be performed. The starting position for signal segmentation is... If this value is greater than 0, the first 4096 units of the signal are taken as the first segment. For example, given an original signal of length 102500... The starting position for the segmentation is 2836. It was divided into 34 segments, each with a length of 4096 and an overlap of 1200, and then... As the first segment.

[0050] Step 3: Obtain the feature sequence of the signal through the CNN-Transformer signal encoder.

[0051] Step 4: Decoding is performed using the beam search algorithm of the Transformer autoregressive decoder. Simultaneously, a key-value cache (KV Cache) is used to store the calculated base symbols' key and value vectors in the decoder's self-attention and cross-attention layers to avoid secondary computation. The beam search decoding steps are as follows: Step 401: Initialize the variables required for beam search decoding; the maximum decoding length is... The beam search width is That is, retain each time Find the optimal candidate sequences. Initialize the prediction result matrix. Alignment path Path score matrix Predicting sequence length The initial symbol is 0, that is... Based on the maximum decoding length, a decoder self-attention mask matrix can be constructed. It is a lower triangular matrix (all beams are the same), such as Figure 14 As shown on the left, self-attention computation for undecoded locations is masked. The mask for the last cross-attention layer is... ( (The length of the signal feature sequence). Initially ,like Figure 14 The first row of the matrix on the right is shown.

[0052] Step 402: Enter the autoregressive beam search loop for each position. Predict symbols ranging from 1 to the maximum decoding length.

[0053] Step 403: In this embodiment, the attention key-value vectors of the base symbols already calculated during the decoding process are cached using KV-cache. Therefore, in each loop, only the symbols decoded in the previous step need to be used as the input to the decoder. If the symbols predicted by some beams in the previous step are terminating symbols or k-mer symbols of length less than k (only for k-mer prediction models), their corresponding beam indices are recorded as follows: The predicted symbol from the previous step. As input to the decoder, the signal passes sequentially through a word embedding network, a self-attention layer, a cross-attention layer, and finally through a linear network and the log_softmax function to obtain the log-likelihood probability matrix of the successor symbols in each beam. , Where is the size of the character table. For stopped beams, their likelihood probabilities need to be updated to: This ensures that all subsequent decoded symbols are terminating symbols.

[0054] In this embodiment, masks need to be used for all self-attention layers. (This mask represents a length of) The vector is used to mask the first... Position symbols for undecoded position symbols (range in) (Self-attention calculation); a mask needs to be used on the last cross-attention layer. And calculate beam At the current location The alignment position of the predicted symbols in the signal feature sequence (in It is a two-dimensional matrix created initially to store alignment position information; here it is used... Distinguishing different beam sequences The The alignment position of the predicted symbol in the signal feature sequence is the minimum alignment position among all cross-attention heads, and its value ranges from [value missing]. , (where the length of the signal feature sequence is used), and finally the cross-attention mask is updated. ; ( (the length of the signal feature sequence), such as Figure 14 ( Figure 14 The second, third, and fourth rows of the matrix on the right (where black positions are 1 and white positions are 0) are shown. This mask allows for identical alignment of consecutive symbols. In certain special cases (such as homopolymer signals or noise signals), multiple identical symbols may be repeated in one position, typically at the end alignment. Therefore, this embodiment adds a threshold. If the current symbol alignment position exceeds Then the alignment interval of subsequent symbols must be at least 1, and the mask update rule is as follows: .

[0055] Step 404: Calculate the expansion path score for all beams. (here) This represents the sum of the log-likelihood probabilities of the decoded paths for each beam, plus the log-likelihood probability of the next predicted base for each path. Add them together to get the score of the extended path. The extended path size is ),from B optimal paths are selected from the given paths as new beams, and the corresponding prediction symbols are stored in [the specified path]. and update the path score. (This indicates the score from the extended path) Select the top B largest values ​​(i.e., perform the topk operation, where k is B) as the score of the new beams decoding path. Finally, record the parent node beams index of the new path. ,based on The key-value pairs stored in the KV-cache are rearranged to match the current path order. Figure 15 and Figure 16 For example ( Figure 16 The parent node beam number is marked in square brackets, and the path probability is marked on the side. It shows the search process using 2-beam and its path update process. The earlier the beam number, the higher the path probability. When decoding the second base, the final path order is: GC, GT; when decoding the third base, the new path is: GTC, GCG, and the corresponding parent node beam numbers are 1 and 0. At this time, the key-value order stored in KV-cache needs to be reversed.

[0056] Step 405: If some beams currently predict symbols that are terminating symbols or k-mer symbols of length less than k; or if the alignment position of some beams currently predicting symbols reaches the end of the signal feature sequence; record the corresponding beam number as... Update its sequence length If the sequence length of all beams has been updated, or the decoding loop has reached the maximum decoding length, then decoding stops. If, when the maximum decoding length is reached, there are still some signals whose optimal beams have not had their sequence lengths updated, then the maximum decoding length is increased and they are decoded a second time.

[0057] Step 5: Perform quality control on the identification results of the sequencing signal fragments. Calculate the average Phred quality score based on the symbol prediction probability output by the model. If the quality score is lower than 10, discard the identification result of the fragment.

[0058] Step 6: Merge the recognition results of all segments in the original signal, retaining only the recognition results of overlapping regions with high confidence during the merging process, such as... Figure 13 The double-arrow region shown provides more complete information about the preceding and following signals during decoding. In this embodiment, if the overlap length of preceding and following segments A and B is 1200, the recognition results corresponding to the first 600 signal points of the overlap interval of segment A and the last 600 signal points of the overlap interval of segment B are merged.

[0059] 4. Test Dataset Construction and Performance Evaluation The test set encompasses six species to validate cross-species robustness: Homo sapiens, Arabidopsis thaliana, mice, Saccharomyces cerevisiae S288C (yeast), poplar, and zebrafish (Danio rerio) samples. The test dataset was generated through a series of preprocessing steps. First, base calls were performed on the sequencing reads using Guppy v4.4.0, except for the zebrafish samples, which used Guppy v6.5.7 for base calls. Detailed sequencing configurations and data links for the test dataset are listed in Table 4.

[0060] Table 4. Sequencing configuration and links for the test dataset

[0061] Read segment accuracy is a key indicator for evaluating the performance of a base recaller.

[0062] in, , , and These represent the number of matching bases, mismatches, insertions, and deletions in the read segment, respectively. This information can be extracted from the CIGAR string in the SAM file. The error rate (mismatch / insertion / deletion) of the read segment is defined as:

[0063] Table 5 shows the comparison results of the read accuracy of each recognition model on the test data. Among them, our-transformer-base-12 is a base prediction model trained on the training set of five species using a 12-layer Transformer decoder. It outperforms third-party recognition models on all test datasets, only falling short of the official Oxford Nanopore Dorado model in sequencing accuracy for poplar, yeast, and zebrafish, species outside the training species distribution. our-transformer-base-12[ext] is a model trained on an extended training set, achieving the best recognition accuracy on all sequencing datasets.

[0064] Table 5 Comparison of Reading Accuracy of Test Datasets

[0065] Table 6 shows the accuracy comparison results of the CTC decoder and the Transformer decoder using the same CNN-Transformer signal encoder. Our-CTC is a model trained on a training set of five species; Our-CTC[ext] is a model trained on an expanded dataset. Replacing the CTC decoder with the Transformer decoder of this patent demonstrates better recognition accuracy on training data of different scales, and this advantage becomes more pronounced as the scale of the training data increases.

[0066] Table 6 Accuracy Comparison of CTC Decoder and Transformer Decoder

Claims

1. A method for identifying nanopore sequencing signals based on Transformer, characterized in that, The method includes the following steps: Step 1: Normalize the nanopore sequencing signal and divide it into signal windows of fixed length; Step 2: Extract signal features using a Transformer encoder that includes convolutional layers and a multi-head self-attention module. The Transformer encoder uses rotational position encoding to embed temporal information. Step 3: Using an autoregressive Transformer decoder, combining signal features with contextual information of the decoded base sequence, the predicted probability of the current base is generated through a cross-attention mechanism; Step 4: Iteratively generate the optimal base sequence based on the beam search algorithm, and optimize decoding efficiency by combining key-value caching; Step 5: Perform overlapping region fusion and quality control on the segmented decoding results, and output the final sequence; The Transformer decoder includes: Self-attention layer: Uses causal masking to shield future location information, ensuring the temporal consistency of autoregressive decoding; Cross-attention layer: Using signal features as keys and base features as queries, it calculates the alignment probability matrix to capture the global dependency between signals and bases; Monotonic alignment constraint: By minimizing the monotonic alignment constraint loss function of bases in the cross-attention layer at the signal feature alignment positions, the physical rationality of decoding is ensured; The monotonic alignment constraint loss function of the cross-attention layer for: ； in, For the first The base of each attention head Alignment position, For the first The base of each attention head +1 alignment position, For alignment interval threshold, The signal characteristic length, is the length of the base sequence.

2. The method for identifying nanopore sequencing signals based on Transformer according to claim 1, characterized in that, The Transformer encoder includes: Convolutional downsampling module: Consists of multiple one-dimensional convolutions, batch normalization, and the Swish activation function, used to compress signal length and extract local features; Multi-layer Transformer coding block: Each layer contains a multi-head self-attention module and a gated feedforward network, and gradient propagation is optimized through residual connections and layer normalization; Upsampling module: Recovers the signal length through linear transformation to ensure input and output length matching.

3. The method for identifying nanopore sequencing signals based on Transformer according to claim 1, characterized in that, The beam search algorithm includes: Key-value caching optimization: Cache the computed key and value vectors in the decoder's self-attention layer and cross-attention layer to avoid duplicate computation; Phased decoding control: Based on the target sequence length distribution, the maximum number of initial decoding iterations and the final decoding length threshold are set to prioritize the fast decoding of short sequences; Dynamic mask update mechanism: based on the current base alignment position Update the cross-attention mask vector to restrict the range of signals accessed in subsequent decoding. To the end of the signal; Secondary decoding optimization: For candidate sequences that were not fully decoded in the first decoding, a second iteration is performed using the maximum decoding length threshold to ensure that long sequences are output completely.

4. The method for identifying nanopore sequencing signals based on Transformer according to claim 1, characterized in that, The nanopore sequencing signal was normalized and divided into fixed-length signal windows, specifically including: Robust normalization based on absolute median difference: The 16-bit digital signal is converted into an electrical signal using a digital-to-analog conversion formula. ； in It is a digital signal. The offset value from the reference signal, The total span of electrical signals that can be measured within the quantization range of digital signals. The quantization order of the digital-to-analog converter. This is the final output electrical signal; Normalization was performed using the absolute median difference. ； The signal segmentation employs an overlapping window strategy to avoid the loss of boundary information.

5. The method for identifying nanopore sequencing signals based on Transformer according to claim 1, characterized in that, The method also includes a training process, specifically: Construct a cross-species mixed RNA sequencing dataset as a training set; Optimize model parameters using cross-entropy loss function and monotonic alignment constraint loss function; A transfer learning approach was adopted, first training the base prediction model and then fine-tuning the k-mer prediction model.

6. The method for identifying nanopore sequencing signals based on Transformer according to claim 1, characterized in that, The method also includes an identification process, specifically: Parse the Fast5 file, obtain the signal data, and perform normalization processing; The signal is divided into fixed-length segments, and base sequences are generated by an encoder and a decoder. Perform quality control and merging on the segmented results, and output the final sequence.

7. A Transformer-based nanopore sequencing signal recognition system, characterized in that, The system for performing the Transformer-based nanopore sequencing signal recognition method as described in claim 1, the system comprising: Signal preprocessing module: used to normalize nanopore sequencing signals and segment overlapping windows; CNN-Transformer encoder module: used to extract local features of the signal through convolutional layers, and combine multi-head self-attention and rotation position encoding to model the global context information of the signal; Transformer decoder module: used to generate base prediction probabilities through self-attention and cross-attention mechanisms; Beam search decoding module: used to combine key value caching and dynamic masking to generate the optimal base sequence; The result fusion module is used to perform overlapping region fusion and quality control on the segmented decoding results.

8. The Transformer-based nanopore sequencing signal recognition system according to claim 7, characterized in that, The CNN-Transformer encoder module includes a convolutional downsampling module, 18 Transformer encoding layers, and an upsampling module; the Transformer decoder module contains 12 decoding layers, of which layers 3, 6, 9, and 12 are cross-attention layers, and the rest are self-attention layers. Each layer adopts an 8-head attention structure with a feature dimension of 512.

Images