Base recognition method and device for nanopore sequencing and storage medium

Abstract

The invention relates to the technical field of bioinformatics, and discloses a base recognition method and device for nanopore sequencing and a storage medium. The method comprises the following steps: inputting current data obtained by nanopore sequencing into a pre-trained convolutional neural network model to obtain current feature data; inputting the current feature data into a pre-trained encoder model to determine an attention score of the current feature data, and determining a semantic vector based on the attention score; the encoder model has a multi-head self-attention structure to calculate an attention score, and the multi-head self-attention structure fuses the rotation position codes to fuse the position dependency relationship between different positions in the attention score; inputting a semantic vector output by the encoder model into a CTC decoder model to obtain a base recognition result; the method not only can be applied to identification of DNA molecules, but also can be applied to identification of RNA molecules, and the universality of the base identification method can be improved while the accuracy of base identification is improved.

Description

Technical Field

[0001] This disclosure relates to the field of bioinformatics technology, and in particular to a base identification method, apparatus and storage medium for nanopore sequencing. Background Technology

[0002] Nanopore sequencing is a single-molecule sequencing technology that uses nanopores (tiny protein channels) to detect changes in electrical current as DNA or RNA molecules pass through. Based on these changes in current, the specific nucleotide sequence of the DNA or RNA can be deduced, enabling base recognition. Nanopore sequencing technology allows for real-time, long-read sequencing and can be performed directly from raw samples without pre-amplification.

[0003] Traditional base recognition methods include the following:

[0004] The first type is based on Hidden Markov Models (HMMs), which achieve base identification by segmenting current signals into events and calculating transition probabilities, such as the base identification methods in nanopore sequencing data analysis tools like Metrichor and Nanocall.

[0005] The second type is base recognition methods based on recurrent neural networks (RNNs), such as the Chiron method based on long short-term memory (LSTM) networks, the Deepnano method based on two-layer LSTM networks, and the Guppy method that uses proprietary neural network architectures and training datasets for base recognition.

[0006] The third type is end-to-end base recognition methods such as Causalcall and RODAN, which are based on the Convolutional Neural Network (CNN) architecture.

[0007] The fourth method is the SACall method, which introduces a self-attention mechanism into nanopore signal decoding.

[0008] However, except for RODAN, the methods mentioned above all focus on base recognition in DNA molecules. Compared to the pore speed of 450 bps for DNA molecules, the pore speed for RNA is only 70 bps. Therefore, the signal patterns of RNA sequencing signals and DNA sequencing signals are completely different. Base recognition models used for DNA molecules cannot function properly during the decoding process of RNA signals, resulting in poor accuracy in RNA base recognition. Summary of the Invention

[0009] In view of this, this disclosure proposes a base recognition method, device and storage medium for nanopore sequencing, which can combine the positional dependence between different element positions during the base recognition process to perform base recognition, so that the base recognition method can be applied not only to the recognition of DNA molecules, but also to the recognition of RNA molecules, thereby improving the accuracy of base recognition and the versatility of the base recognition method.

[0010] According to one aspect of this disclosure, a base recognition method for nanopore sequencing is provided, the method comprising:

[0011] Obtain current data from nanopore sequencing;

[0012] The current data is input into a pre-trained convolutional neural network model to downsample and extract features from the current data, thereby obtaining current feature data.

[0013] The current feature data is input into a pre-trained encoder model, so that the encoder model determines the attention score of the current feature data based on the current feature data, and determines the semantic vector corresponding to the current feature data based on the attention score; wherein, the encoder model has a multi-head self-attention structure to calculate the attention score, and the multi-head self-attention structure integrates rotational position encoding to integrate the positional dependencies between different positions in the attention score;

[0014] The semantic vector output by the encoder model is input into a pre-trained CTC decoder model to decode the semantic vector and obtain the base recognition result.

[0015] In one possible implementation, the method is used in a base recognition device, the memory hierarchy of which includes high-bandwidth memory (HBM) and static random access memory (SRAM); accordingly,

[0016] The step of inputting the current feature data into a pre-trained encoder model to determine the semantic vector corresponding to the current feature data based on the attention score of the current feature data includes:

[0017] The HBM divides the input data to be input to the multi-head self-attention structure into blocks to obtain the input vector;

[0018] The input vector is loaded from the HBM into the SRAM, and the multi-head self-attention structure is run in the SRAM to obtain the attention score.

[0019] The attention score of the input vector is updated to the HBM.

[0020] In one possible implementation, the multi-head self-attention structure incorporates rotational position encoding, including:

[0021] For each self-attention structure in the multi-head self-attention structure, based on a set of self-attention weights of the self-attention structure, the query vector, key vector, and value vector corresponding to the input vector are determined; wherein, the self-attention weights include a first weight for calculating the query vector, a second weight for calculating the key vector, and a third weight for calculating the value vector;

[0022] Determine the rotation matrix corresponding to each vector element at each position in the input vector;

[0023] For each vector element, a set of query vector elements and key vector elements are corresponding to the same set of query vector elements and key vector elements. The set of query vector elements and key vector elements are rotated and encoded based on the rotation matrix to obtain the encoded query vector elements and the encoded key vector elements.

[0024] The position dependency model is obtained by performing a dot product between the encoded query vector elements and the encoded key vector elements.

[0025] Based on the position dependency model and the value vector, the self-attention score of the self-attention structure is determined;

[0026] The attention scores are obtained by fusing the self-attention scores corresponding to the self-attention structures of each head based on a preset parameter matrix.

[0027] In one possible implementation, the encoder model further includes: a depthwise separable convolutional network structure based on a gating mechanism, wherein the depthwise separable convolutional network structure is connected to the multi-head self-attention structure; the depthwise separable convolutional network structure includes at least:

[0028] The first pointwise convolutional layer is used to perform interactive calculations on data from different input channels to obtain channel interaction information.

[0029] A gated layer is used to select information based on the channel interaction information and transmit the selected first feature map to the deep convolutional layer.

[0030] The deep convolutional layer is used to independently perform convolution operations on each channel of the input first feature map to obtain the feature map corresponding to each channel;

[0031] The second pointwise convolutional layer is used to combine and transform the different channel data of the input second feature map to obtain a new feature map. The second feature map is determined based on the feature maps corresponding to each channel output by the deep convolutional layer.

[0032] In one possible implementation, the depthwise separable convolutional network structure further includes:

[0033] The first normalization layer is used to normalize the attention score output by the multi-head self-attention structure and output the first normalization result to the first pointwise convolutional layer. The normalization result includes the data of the different input channels.

[0034] The batch normalization layer is used to normalize the feature map corresponding to each channel output by the deep convolutional layer, and output the obtained second normalization result to the activation layer.

[0035] The activation layer is used to adjust the activation level based on the second normalization result to obtain the second feature map, which includes the feature maps of each channel after activation.

[0036] In one possible implementation, the base recognition model includes the convolutional neural network model, the encoder model, and the CTC decoder model, which is trained in conjunction with a bidirectional decoder; for each sequence output position, the bidirectional decoder is used to determine the decoding result of the output position by combining the context information of the output position.

[0037] In one possible implementation, the training process of the base recognition model includes:

[0038] Obtain a training set, which includes multiple sets of training data, each set of training data including sample current data and base sequence labels corresponding to the sample current data;

[0039] The sample current data is input into a pre-created neural network model to obtain the model output; the neural network model includes an initial convolutional neural network model, an initial encoder model, an initial CTC decoder connected to the initial encoder model, and the bidirectional decoder; the model output includes a first output result from the initial CTC decoder and a second output result from the bidirectional decoder.

[0040] The first output result and the base sequence tag are input into a preset CTC loss function to obtain the CTC loss value;

[0041] The second output result and the base sequence tag are input into a preset KL divergence loss function to obtain the KL divergence loss value;

[0042] The neural network model is iteratively trained by combining the CTC loss value and the KL divergence loss value to obtain the base recognition model, which includes a trained convolutional neural network model, a trained encoder model, and a trained CTC decoder.

[0043] In one possible implementation, the encoder model further includes a first forward propagation layer, a depthwise separable convolutional network structure based on a gating mechanism, a second forward propagation layer, and a second normalization layer.

[0044] The output features of the first forward propagation layer and the input features of the first forward propagation layer are merged, and the merged first feature is output to the multi-head self-attention structure; the first forward propagation layer is used to extract features from the input features;

[0045] After the first merged feature is merged with the attention score output by the multi-head self-attention structure, the merged second merged feature is output to the depthwise separable convolutional network structure.

[0046] The new feature map output by the depthwise separable convolutional network structure is merged with the second merged feature, and the resulting third merged feature is output to the second forward propagation layer; the second forward propagation layer is used to extract features from the third merged feature.

[0047] After the output feature of the second forward propagation layer is merged with the third merged feature, the merged fourth merged feature is output to the second normalization layer for normalization processing to obtain the third normalization result.

[0048] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0049] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0050] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0051] This disclosure involves acquiring current data from nanopore sequencing; inputting the current data into a pre-trained convolutional neural network model to downsample and extract features, resulting in current feature data; inputting the current feature data into a pre-trained encoder model to determine an attention score for the current feature data, and then determining the corresponding semantic vector based on the attention score; wherein the encoder model has a multi-head self-attention structure to calculate the attention score, and the multi-head self-attention structure incorporates rotational position encoding to integrate positional dependencies between different positions in the attention score; the semantic vector output by the encoder model is input into a pre-trained CTC decoder model to decode the semantic vector and obtain the base recognition result; this can solve the problem of inaccurate RNA molecule recognition in traditional base recognition methods used for nanopore sequencing; because the multi-head self-attention structure incorporates rotational position encoding, base recognition can be performed by combining positional dependencies between different element positions during the base recognition process, making this base recognition method applicable not only to DNA molecules but also to RNA molecules, improving both the accuracy and versatility of base recognition methods.

[0052] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0053] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0054] Figure 1 A flowchart illustrating a base recognition method for nanopore sequencing according to an embodiment of the present disclosure is shown;

[0055] Figure 2 A schematic diagram of the structure of a base recognition model according to an embodiment of the present disclosure is shown;

[0056] Figure 3 A schematic diagram of a depthwise separable convolutional network structure based on a gating mechanism according to an embodiment of the present disclosure is shown.

[0057] Figure 4 The diagram illustrates the loss curves during training of a base recognition model, a rotation position encoding removal model, and a depthwise separable convolutional network structure with a gating mechanism removal model according to an embodiment of the present disclosure.

[0058] Figure 5 A block diagram of a base recognition device for nanopore sequencing according to an embodiment of the present disclosure is shown;

[0059] Figure 6 A block diagram of a base recognition device for nanopore sequencing according to another embodiment of the present disclosure is shown. Detailed Implementation

[0060] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0061] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0062] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0063] First, let me introduce some of the terms used in this application.

[0064] Transformer architecture: It is a deep learning model based on the self-attention mechanism, mainly used to process sequence data. In this application, the Transformer architecture is used to process the current data corresponding to the base sequence.

[0065] The Transformer architecture primarily relies on a self-attention mechanism to capture global dependencies in the input sequence. This allows the Transformer to process all elements in the sequence in parallel, thereby significantly improving training speed.

[0066] The self-attention mechanism is used to filter out important information and prioritize less important information. It can simultaneously focus on all positions in the sequence, thus capturing global dependencies. Furthermore, this mechanism allows the model to automatically calculate the importance weight of each position in the input sequence. This enables the model to focus on important information relevant to the current task, placing more attention on meaningful locations. Compared to traditional fixed-weight mechanisms, self-attention is more flexible in adapting to different tasks and inputs.

[0067] The self-attention mechanism uses three input vectors: Q (query vector), K (key vector), and V (value vector). These three vectors are obtained from the original input vector (usually represented by word embeddings) through a linear transformation. In the self-attention mechanism, based on the query vector Q, a weight distribution is obtained by calculating the similarity between the query vector and all key vectors K, which is then used to weight and sum the associated value vector V.

[0068] The Transformer architecture consists of an encoder and a decoder. The encoder processes the input sequence, transforming it into a continuous representation that includes both semantic and structural information. An encoder typically consists of multiple identical network units stacked together. Each network unit contains two main parts: a multi-head self-attention structure and position-wise feed-forward networks.

[0069] Among them, the multi-head self-attention structure is used to learn multiple different attention representations simultaneously. Each attention head can focus on different parts of the sequence, thus providing multiple independent expressive capabilities. Multi-head attention can capture semantic information at different levels, enabling the model to better understand the input sequence and improve representation and generalization capabilities.

[0070] Forward propagation layer: Used to perform independent nonlinear transformations on the output at each location, enhancing the expressiveness of the model. Illustratively, the forward propagation layer can be implemented based on a fully connected network.

[0071] The decoder is used to predict the decoding result at the next position or generate the entire output sequence based on the encoder's output and the already generated sequence. Generally, the decoder is also composed of multiple identical network units stacked together. Each network unit, in addition to containing the multi-head self-attention mechanism and forward propagation layer from the encoder, adds another multi-head self-attention mechanism to focus on the encoder's output.

[0072] In this application, a base recognition model is constructed based on an encoder with a Transformer architecture. This encoder's multi-head self-attention structure incorporates rotational position encoding, which can integrate positional dependencies between different positions during base recognition, thereby improving the accuracy of molecular base recognition. Therefore, due to the improved base recognition accuracy, the base recognition method proposed in this application can be applied not only to the base recognition of DNA molecules but also to the base recognition of RNA molecules.

[0073] The base recognition method for nanopore sequencing proposed in this application will be described in detail below.

[0074] Figure 1 A flowchart illustrating a base identification method for nanopore sequencing according to an embodiment of this disclosure is shown. This embodiment describes the method in a computationally oriented base identification device, which can be a user terminal or a server. The user terminal includes, but is not limited to, nanopore sequencers, computers, mobile phones, or tablet computers. This embodiment does not limit the type of base identification device or user terminal. Figure 1 As shown, the method includes:

[0075] Step 101: Obtain the current data obtained from nanopore sequencing.

[0076] The current data can be collected by a base recognition device or sent by other devices; this embodiment does not limit the process of acquiring the current data. The current data is the blocking current collected when a single nucleotide chain of a molecule passes through a nanopore; the molecule can be a DNA molecule or an RNA molecule; this embodiment does not limit the type of molecule.

[0077] Step 102: Input the current data into a pre-trained convolutional neural network model to downsample and extract features from the current data to obtain current feature data.

[0078] In one example, the convolutional neural network model includes multiple convolutional units, each comprising a one-dimensional convolutional layer, a batch normalization layer, and an activation layer connected in sequence. The one-dimensional convolutional layer performs convolution operations on the input data to extract local features; the batch normalization layer normalizes the local features output by the one-dimensional convolutional layer to improve the model's training and convergence speed; and the activation layer performs non-linear transformations on the feature data output by the batch normalization layer to improve the generalization ability of the convolutional neural network model.

[0079] refer to Figure 2 Taking a convolutional neural network model consisting of 3 convolutional units (201) as an example ( Figure 2 (represented by "×3"), each convolutional unit includes a one-dimensional convolutional layer 2011, a batch normalization layer 2012 connected to the one-dimensional convolutional layer 2011, and an activation layer 2013 connected to the batch normalization layer. This activation layer uses the activation function Sigmoid Linear Unit (SiLU). Figure 2 This example illustrates the use of a Sigmoid linear unit for nonlinear transformation in the activation layer. In actual implementation, the activation function used in the activation layer can be other functions; this embodiment does not limit the implementation method of the activation function. (Illustratively,) Figure 2 In the diagram, the configuration of each convolutional unit in the three convolutional units 201 is as follows:

[0080] The first convolutional unit has 4 output channels, a kernel size of 5, and a stride of 1.

[0081] The second convolutional unit has 6 output channels, a kernel size of 5, and a stride of 1.

[0082] The third convolutional unit has 512 output channels, a kernel size of 19, and a stride of 10.

[0083] The number of output channels, also known as the number of convolutional kernels or filters, determines how many different features a convolutional unit will learn. For example, the first convolutional unit has 4 output channels, meaning it will learn 4 different features. The kernel size determines the size of the input region covered by each convolution operation. For example, the first convolutional unit has a kernel size of 5, meaning each convolution operation considers 5 consecutive elements of the input data. The stride determines the distance the convolutional kernel moves across the input data. For example, the first convolutional unit has a stride of 1, meaning the kernel moves one element at a time. The third convolutional unit has a stride of 10, a larger stride than the first two, resulting in smaller output feature sizes. This reduces the spatial dimensionality of the data, facilitating subsequent model processing.

[0084] The number and configuration parameters of the convolutional units mentioned above are only illustrative. In actual implementation, the convolutional neural network model can have more or fewer convolutional units, and the configuration parameters of each convolutional unit can also be set to other values. This embodiment does not limit this.

[0085] Step 103: Input the current feature data into the pre-trained encoder model so that the encoder model can determine the attention score of the current feature data based on the current feature data, and determine the semantic vector corresponding to the current feature data based on the attention score; wherein, the encoder model has a multi-head self-attention structure to calculate the attention score, and the multi-head self-attention structure integrates rotation position encoding to integrate the positional dependencies between different positions in the attention score.

[0086] In this embodiment, the multi-head self-attention structure fusion rotational position encoding includes the following steps 1-6:

[0087] Step 1: For each self-attention structure in a multi-head self-attention structure, determine the query vector, key vector, and value vector corresponding to the input vector based on a set of self-attention weights of the self-attention structure.

[0088] The self-attention weights include a first weight for calculating the query vector, a second weight for calculating the key vector, and a third weight for calculating the value vector.

[0089] Assuming the input vector is X, the multi-head self-attention structure includes h self-attention structures. Each self-attention structure has a set of self-attention weights, including: the first weight... Second weight and third weight Where i is a positive integer in the range [1, h], and h is a positive integer. For the i-th self-attention structure, the query vector, key vector, and value vector corresponding to the input vector are determined by a set of self-attention weights based on the self-attention structure, which can be expressed by the following formula:

[0090]

[0091] Wherein, the input vector X is a vector matrix containing multiple vector elements. and These are the trainable weight matrices, Q. i Let K be the query vector corresponding to the i-th self-attention structure, K be the key vector corresponding to the i-th self-attention structure, and V be the value vector corresponding to the i-th self-attention structure.

[0092] Step 2: Determine the rotation matrix corresponding to each element of the input vector.

[0093] Rotation matrix R d Θ,m can be expressed by the following formula:

[0094]

[0095] in, d represents the spatial dimension of the rotation encoding, and d is a preset value; i represents the grouping index, which ranges from [0, d / 2), corresponding to θ. i It is a monotonically decreasing function, with values ​​in the range [1, 10]. -4 Between ), m is a vector element x m Index the position of the input vector.

[0096] Step 3: For each vector element, a set of query vector elements and key vector elements are rotated and encoded based on the rotation matrix to obtain the encoded query vector elements and the encoded key vector elements.

[0097] Schematic representation of the encoded query vector element f Q (x m (m) and the encoded key vector element f K (x m (m) can be expressed by the following formula:

[0098] f Q (x m ,m)=Rd Θ,mW Q x m ;

[0099] f K (x m ,m)=R d Θ,mW K x m ;

[0100] Among them, W Q x m W represents the query vector element at position index m. K x m The key vector element representing the position index m, R d Θ,m represents the rotation matrix corresponding to position index m.

[0101] Step 4: Perform a dot product between the encoded query vector elements and the encoded key vector elements to obtain the position dependency model.

[0102] Schematic, the process of performing a dot product between the encoded query vector elements and the encoded key vector elements can be represented by the following formula:

[0103]

[0104] in, K represents the transpose of the encoded query vector element corresponding to position index m. n W represents the encoded key vector element corresponding to position index n. Q x m W represents the query vector element at position index m. K x n The key vector element representing the position index n, R d Θ,m represents the rotation matrix corresponding to position index m, R d Θ,n represents the rotation matrix corresponding to position index n, W Q W represents the first weight. K This represents the second weight, where m is a vector element x. m In the input vector, n is the index of the vector element x. n Index the position of the input vector.

[0105] Step 5: Determine the self-attention score of the self-attention structure based on the position dependency model and the value vector corresponding to the input vector.

[0106] The self-attention score head(Q,K,V) of each head's self-attention structure can be represented by the following formula:

[0107]

[0108] Where, d k Let K be the dimension of matrix K, V be the value vector corresponding to the input vector, and Q be the dimension of matrix K. T K represents the position dependency model, where Q represents the encoded query vector composed of the elements of each encoded query vector, and K represents the encoded key vector composed of the elements of each encoded key vector.

[0109] Step 6: Based on the preset parameter matrix, fuse the self-attention scores corresponding to the self-attention structures of each head to obtain the attention score.

[0110] The process of fusing the self-attention scores corresponding to each head's self-attention structure can be represented by the following formula:

[0111] MultiHead(Q,K,V)=Concat(head1,...,head h W O ;

[0112] Where MultiHead(Q,K,V) represents the attention score output by the multi-head self-attention structure, and the Concat function is used to concatenate the self-attention scores output by the multi-head self-attention structure. O is a trainable parameter matrix.

[0113] In one example, the multi-head self-attention structure involves a large number of matrix multiplication and addition operations. Therefore, to improve the efficiency of base recognition, in this embodiment, the memory hierarchy of the base recognition device running the multi-head self-attention structure includes High Bandwidth Memory (HBM) and Static Random-Access Memory (SRAM). Correspondingly, current feature data is input into a pre-trained encoder model to determine the semantic vector corresponding to the current feature data based on the attention score of the current feature data. This includes: dividing the input data to be input to the multi-head self-attention structure into blocks in the HBM to obtain an input vector; loading the input vector from the HBM to the SRAM and running the multi-head self-attention structure in the SRAM to obtain an attention score; and updating the attention score of the input vector to the HBM.

[0114] Loading the input vector from HBM into SRAM and running a multi-head self-attention structure in SRAM yields an attention score that can be expressed as follows:

[0115]

[0116] Among them, Q i K i and Vi It is an input vector obtained by partitioning, d k For matrix K i The dimension size, Attention represents the computation process of the multi-head self-attention structure.

[0117] Since SRAM has a faster access speed than HBM but a lower storage capacity, computational efficiency can be improved by performing multi-head self-attention calculations in SRAM and storing the output in HBM, without increasing the storage capacity of SRAM.

[0118] Optionally, to further improve the computational efficiency of the encoder model, the encoder model also includes a depthwise separable convolution (DSC) network structure based on a gating mechanism, which is connected to a multi-head self-attention structure. Depthwise separable convolution aims to reduce the number of model parameters and computational cost; it consists of depthwise convolution and pointwise convolution. In this embodiment, a gating mechanism is also incorporated into the depthwise separable convolution network structure, which can further save computational cost and improve the computational efficiency of the encoder model.

[0119] refer to Figure 3 As shown, the depth-separable convolutional network structure includes at least: a first pointwise convolutional layer 31, a gated layer 32, a depthwise convolutional layer 33, and a second pointwise convolutional layer 34.

[0120] The first pointwise convolutional layer 31 is used to perform interactive calculations on data from different input channels to obtain channel interaction information.

[0121] Compared to ordinary convolution, the first pointwise convolutional layer has a 1×1 kernel and is used for interactive computation of different input channels. In this embodiment, by setting the first pointwise convolutional layer 31, the interaction information between channels can be obtained before the depthwise convolutional layer 33, so as to reduce the amount of computation while achieving the same computational effect as conventional convolution; on the other hand, it can also provide context information for the gating layer 32, thereby improving the accuracy of the gating layer 32's computation.

[0122] The gate layer 32 is used to select information based on channel interaction information and transmit the selected first feature map to the deep convolutional layer 33.

[0123] The gating layer controls the flow of information based on the Gated Linear Unit (GLU) mechanism, deciding which information should be retained or discarded. Introducing a gating mechanism into a depthwise separable convolutional network can further improve the model's flexibility and efficiency. The computation process of the GLU can be represented by the following formula:

[0124] a,b=Chunk(hidden_state,dim=-1)

[0125]

[0126] Where, hidden_state represents the channel interaction information received by the gating layer, Chunk represents dividing the channel interaction information hidden_state into two equal parts a and b along the last dimension (dim=-1); GLU(a,b) represents the first feature map output by the gating layer, and σ represents the sigmoid function.

[0127] The deep convolutional layer 33 is used to independently perform convolution operations on each channel of the input first feature map to obtain the feature map corresponding to each channel.

[0128] The deep convolutional layer 33 performs high-dimensional modeling of the first feature map of the input. Compared with traditional convolution, each convolutional kernel of the deep convolution operates on only one input channel to extract the spatial features of the input feature map.

[0129] The second pointwise convolutional layer 34 is used to combine and transform the different channel data of the input second feature map to obtain a new feature map. The second feature map is determined based on the feature maps corresponding to each channel output by the deep convolutional layer.

[0130] The second pointwise convolutional layer 34 uses a 1×1 kernel to perform convolution, thereby changing the number of channels in the second feature map and fusing the second feature maps from different channels.

[0131] The input of the depthwise separable convolutional network structure is merged with the output of the second pointwise convolutional layer 34 and then output to the next layer of the encoder model to fuse the features before and after convolution for processing, thereby improving model performance.

[0132] Optionally, to improve the training efficiency of depthwise separable convolutional network structures, refer to Figure 3 The depthwise separable convolutional network structure also includes: a first normalization layer 35, a batch normalization layer 36, and an activation layer 37.

[0133] The first normalization layer 35 is used to normalize the attention score output by the multi-head self-attention structure and output the first normalization result to the first pointwise convolutional layer. The normalization result includes data from different input channels input to the first pointwise convolutional layer 31.

[0134] Layer normalization is used to normalize each feature channel of the attention score output by each group of multi-head self-attention structures, making the mean close to 0 and the variance close to 1. In this embodiment, by setting a first normalization layer 35 after the multi-head self-attention structure, the problems of gradient vanishing or gradient exploding during training can be reduced, thereby making the training process of the model more stable and faster.

[0135] Batch normalization layer 36 is used to normalize the feature map corresponding to each channel of the deep convolutional layer output, and output the obtained second normalization result to activation layer 37.

[0136] Batch normalization is used to normalize each feature channel in each training batch, making the mean close to 0 and the variance close to 1. In this embodiment, by setting a batch normalization layer after the deep convolutional layer, training convergence can be further accelerated and the vanishing / exploding gradient problem can be reduced.

[0137] Activation layer 37 is used to adjust the activation level based on the second normalization result to obtain a second feature map, which includes the feature maps of each channel after activation.

[0138] Activation layer 37 is used to introduce non-linearity into the depthwise separable convolutional network structure, improving the model's generalization ability. Schematic, activation layer 37 adjusts the activation level using the self-gated Swish activation function. This self-gating mechanism allows the model to adaptively adjust the activation strength during training. In this embodiment, by setting the activation layer after the batch normalization layer, the input data to the activation layer can be distributed within a certain range, improving the effectiveness of the activation function.

[0139] In this embodiment, the second feature map is obtained by batch normalizing and activating the feature maps corresponding to each channel output by the deep convolutional layer. In other embodiments, the second feature map can also be the feature maps corresponding to each channel output by the deep convolutional layer. That is, batch normalization and activation are not performed on the feature maps corresponding to each channel. In this case, the deep convolutional layer is directly connected to the second pointwise convolutional layer. This embodiment does not limit the implementation method of the depth-separable convolutional network structure.

[0140] Optionally, the encoder model also includes a first forward propagation layer, a depthwise separable convolutional network structure based on a gating mechanism, a second forward propagation layer, and a second normalization layer.

[0141] The first forward propagation layer is used to extract features from the input features. Specifically, the first forward propagation layer, connected to the convolutional neural network, extracts features from the current feature data output by the convolutional neural network model. The output features and input features of the first forward propagation layer are merged, and the merged first feature is output to the multi-head self-attention structure. The input vector of the multi-head self-attention structure can be the vector obtained after dividing the first merged feature into blocks.

[0142] Furthermore, after merging the first merged feature with the attention score output from the multi-head self-attention structure, the resulting second merged feature is output to the depthwise separable convolutional network structure (e.g., input to the aforementioned first normalization layer). In this way, the depthwise separable convolutional network structure can combine the attention score and the output features of the first forward propagation layer for feature extraction, thereby improving the accuracy of feature extraction.

[0143] The new feature map output by the depthwise separable convolutional network structure is merged with the second merged feature, and the resulting third merged feature is output to the second forward propagation layer; the second forward propagation layer is used to extract features from the third merged feature.

[0144] The implementation principle of the second forward propagation layer is the same as that of the first forward propagation layer, and will not be elaborated further in this embodiment. After the output feature of the second forward propagation layer is merged with the third merged feature, the merged fourth merged feature is output to the second normalization layer for normalization processing to obtain the third normalized result. At this time, the third normalized result of the second normalization layer connected to the CTC decoder model is the semantic vector.

[0145] Indicatively, for reference Figure 2 The encoder model includes multiple coding units 202. Figure 2 The encoder model in China includes 8 coding units 202 as an example ( Figure 2 (represented by "×8" in Chinese). Figure 2It can be seen that each coding unit 202 includes, in sequence along the information propagation direction: a first forward propagation layer 2021, a multi-head self-attention structure 2022, a depthwise separable convolutional network structure 2023, a second forward propagation layer 2024, and a second normalization layer 2025. The outputs of different network layers in the coding unit 202 are combined with the inputs of that network layer and then transmitted to the next network layer. That is, the output of the first forward propagation layer 2021 is combined with the input of the first forward propagation layer 2021 to obtain the first merged feature, which is then transmitted to the multi-head self-attention structure 2022; the output of the multi-head self-attention structure 2022 (i.e., the attention score) is combined with the input of the multi-head self-attention structure 2022 (i.e., the first merged feature) to obtain the second merged feature, which is then transmitted to the depthwise separable convolutional network structure 2023; the output of the depthwise separable convolutional network structure 2023 (i.e., the new feature map) is combined with the input of the depthwise separable convolutional network structure 2023 (i.e., the second merged feature) to obtain the third merged feature, which is then transmitted to the second forward propagation layer 2024; the output of the second forward propagation layer 2024 is combined with the input of the second forward propagation layer 2024 (i.e., the third merged feature) to obtain the fourth merged feature, which is then transmitted to the second normalization layer 2025 to obtain the third normalized result output by the second normalization layer 2025.

[0146] In this design, the first forward propagation layer 2021 in the first coding unit 202 is connected to the convolutional neural network model. Specifically, the input feature of the first forward propagation layer 2021 in the first coding unit 202 is the current feature data output by the convolutional neural network model. The input feature of the first forward propagation layer 2021 in each coding unit 202 after the first coding unit 202 is the third normalized result output by the second normalization layer 2025 of the previous coding unit 202. The third normalized result output by the second normalization layer 2025 of the eighth coding unit 202 (i.e., the last coding unit 202) is the semantic vector output by the encoder model.

[0147] Step 104: Input the semantic vector output by the encoder model into the pre-trained CTC decoder model to decode the semantic vector and obtain the base recognition result.

[0148] Connectionist Temporal Classification (CTC) decoders are a type of decoder used to map input sequences to output sequences, for example: [reference needed] Figure 2In this embodiment, the CTC decoder model 203 is connected to the decoder model 202 and is used to map the semantic vector output by the decoder model to a base sequence. Since the CTC decoder can introduce a whitespace topological state, it can solve the problem of inconsistency between the current signal and the base sequence length, thus ensuring the accuracy of the base recognition result.

[0149] In summary, the base recognition method for nanopore sequencing provided in this embodiment acquires current data obtained from nanopore sequencing; inputs the current data into a pre-trained convolutional neural network model to downsample and extract features from the current data, obtaining current feature data; inputs the current feature data into a pre-trained encoder model, enabling the encoder model to determine the attention score of the current feature data based on the current feature data, and to determine the semantic vector corresponding to the current feature data based on the attention score; wherein, the encoder model has a multi-head self-attention structure to calculate the attention score, and the multi-head self-attention structure integrates rotational position encoding to incorporate different... The method identifies positional dependencies between elements; it inputs the semantic vector output by the encoder model into a pre-trained CTC decoder model to decode the semantic vector and obtain the base recognition result; it can solve the problem of inaccurate RNA molecule recognition in traditional base recognition methods used for nanopore sequencing; due to the fusion of rotational position encoding in the multi-head self-attention structure, the method can combine the positional dependencies between different element positions during base recognition, making it applicable not only to DNA molecules but also to RNA molecules, thus improving both the accuracy and versatility of base recognition methods.

[0150] In addition, by loading the input vector from HBM to SRAM and running a multi-head self-attention structure in SRAM, attention scores can be obtained; updating the attention scores of the input vector to HBM can improve computational efficiency without increasing the storage capacity of SRAM.

[0151] In addition, by setting depthwise separable convolutions based on gating mechanisms in the encoder model, the number of model parameters and computational cost can be reduced, thereby improving the training efficiency of the model.

[0152] Optionally, based on the above embodiments, the base recognition model is trained in conjunction with a bidirectional decoder. This base recognition model includes the convolutional neural network model, encoder model, and CTC decoder model described in the above embodiments. Specifically, for each sequence output position, the bidirectional decoder is used to determine the decoding result of that output position by combining the context information of that output position.

[0153] Accordingly, the training process of the base recognition model includes the following steps:

[0154] Step 1: Obtain the training set. The training set includes multiple sets of training data. Each set of training data includes sample current data and the base sequence label corresponding to the sample current data.

[0155] The method for acquiring sample current data is the same as that for acquiring current data, and will not be repeated here. The base sequence tags of the sample current data are pre-labeled to indicate the actual base sequence corresponding to the sample current data.

[0156] Step 2: Input the sample current data into the pre-created neural network model to obtain the model output results; the neural network model includes an initial convolutional neural network model, an initial encoder model, an initial CTC decoder and a bidirectional decoder connected to the initial encoder model respectively; the model output results include the first output result of the initial CTC decoder and the second output result of the bidirectional decoder.

[0157] Optionally, the bidirectional decoder can be a bidirectional mask decoder; or, it can include a forward Transformer decoder and a backward Transformer decoder, in which case the output of the initial encoder model is connected to the forward Transformer decoder and the backward Transformer decoder, respectively. Figure 2 In this example, the bidirectional decoder 204, which includes a forward Transformer decoder and a reverse Transformer decoder, will be used for illustration.

[0158] Step 3: Input the first output result and the base sequence label into the preset CTC loss function to obtain the CTC loss value.

[0159] Step 4: Input the second output result and the base sequence label into the preset KL divergence loss function to obtain the KL divergence loss value.

[0160] Step 5: Iteratively train the neural network model using the combined CTC loss value and KL divergence loss value to obtain the base recognition model. The base recognition model includes the trained convolutional neural network model, the trained encoder model, and the trained CTC decoder.

[0161] In this embodiment, by simultaneously using the CTC loss function and the KL divergence loss function to train the base recognition model, the model convergence speed can be accelerated and the model accuracy can be improved.

[0162] Optionally, since retaining the bidirectional decoder has no impact on the actual accuracy when using the base recognition model for base recognition after training, but retaining the decoder will reduce the inference speed, the bidirectional decoder is no longer used to calculate the base sequence during the base recognition process.

[0163] To better understand the performance of the base recognition model provided in this application, the base recognition method proposed in this application is compared with traditional base recognition methods below.

[0164] Specifically, Table 1 below shows the base recognition accuracy of this application compared to the latest versions of Guppy and RODAN in five species (human, mouse, Arabidopsis thaliana, poplar, and yeast) on the test set. All base recognition results were aligned with reference genome sequences using a sequence alignment program (such as minimap2), and only the best alignment results were retained. The decoding accuracy is shown in Table 1 below, revealing that the base recognition accuracy of this application is superior to traditional base recognition algorithms in all five species. Regarding inference speed, although Guppy with 2.2M parameters has the shortest inference time, the emergency recognition model of this application, with over 65M parameters, only lags behind Guppy by 0.03s in inference time and is faster than RODAN with 10M parameters.

[0165] Table 1: Performance comparison of different RNA base recognition methods.

[0166]

[0167] To further illustrate the model structure affecting the base recognition accuracy of this invention, the following two sets of ablation experiments are referenced, including: 1. Replacing rotational position encoding with absolute position encoding; 2. Removing the depthwise separable convolutional network structure from the base recognition model. The two sets of ablation experiments are described below.

[0168] 1. To investigate the impact of rotation position encoding embedding on model performance, the model with rotation position encoding removed was trained for 12 epochs on the same training set. The test results are shown in Table 2 below. Compared with the original model, the decoding performance decreased after removing rotation position encoding. Therefore, Table 2 shows that rotation position encoding embedding can enhance the robustness of multi-head self-attention mechanism to sequence position representation.

[0169] 2. To investigate the impact of the depthwise separable convolutional network structure on model performance, the model after removing the depthwise separable convolutional network structure was trained for 12 epochs. The test results are shown in Table 2 below. Compared with the original model and the model after removing the rotation position encoding, the decoding performance of the model after removing the depthwise separable convolutional network structure further deteriorated. Therefore, Table 2 can show that the depthwise separable convolutional network structure based on the gating mechanism in this application plays an important role in enhancing the model's local dependency modeling.

[0170] Table 2: Comparison of model accuracy under different ablation experiments.

[0171]

[0172] Furthermore, the loss curves during training of the base recognition model, the removal of rotation position encoding, and the removal of the depthwise separable convolutional network structure based on gating mechanism proposed in this application are as follows: Figure 4 As shown, according to Figure 4 It is evident that removing the rotational position encoding has little impact on convergence during training; its main effect is to enhance the model's generalization ability to decode sequences of different lengths. Conversely, models that remove the depthwise separable convolutional network structure converge more slowly during training and converge to higher loss values. Therefore, setting a depthwise separable convolutional network structure can improve model convergence speed and accuracy.

[0173] Figure 5 This is a block diagram of a base recognition device for nanopore sequencing provided in one embodiment of this application. The device includes: an acquisition module 510, a convolution module 520, an encoding module 530, and a decoding module 540.

[0174] The acquisition module 510 is used to acquire current data obtained from nanopore sequencing.

[0175] The convolution module 520 is used to input the current data into a pre-trained convolutional neural network model to downsample and extract features from the current data to obtain current feature data.

[0176] The encoding module 530 is used to input the current feature data into a pre-trained encoder model, so that the encoder model determines the attention score of the current feature data based on the current feature data, and determines the semantic vector corresponding to the current feature data based on the attention score; wherein, the encoder model has a multi-head self-attention structure to calculate the attention score, and the multi-head self-attention structure integrates rotational position encoding to integrate the positional dependencies between different positions in the attention score;

[0177] The decoding module 540 is used to input the semantic vector output by the encoder model into a pre-trained CTC decoder model to decode the semantic vector and obtain the base recognition result.

[0178] For relevant details, please refer to the above method implementation examples.

[0179] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0180] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0181] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions stored in the memory. The electronic device includes the base recognition device from the above embodiments; examples of the electronic device can be found in 6.

[0182] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0183] Figure 6 This application provides a block diagram of a base recognition device for nanopore sequencing according to one embodiment. For example, device 1900 can be provided as a server or terminal device. (Refer to...) Figure 6 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0184] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0185] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0186] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A base recognition method for nanopore sequencing, characterized in that, The method includes: Obtain current data from nanopore sequencing; The current data is input into a pre-trained convolutional neural network model to downsample and extract features from the current data, thereby obtaining current feature data. The current feature data is input into a pre-trained encoder model, so that the encoder model determines the attention score of the current feature data based on the current feature data, and determines the semantic vector corresponding to the current feature data based on the attention score; wherein, the encoder model has a multi-head self-attention structure to calculate the attention score, and the multi-head self-attention structure integrates rotational position encoding to integrate the positional dependencies between different positions in the attention score; The semantic vector output by the encoder model is input into a pre-trained CTC decoder model to decode the semantic vector and obtain the base recognition result.

2. The method according to claim 1, characterized in that, The method is used in a base recognition device, wherein the memory hierarchy of the base recognition device includes high-bandwidth memory (HBM) and static random access memory (SRAM); accordingly, The step of inputting the current feature data into a pre-trained encoder model to determine the semantic vector corresponding to the current feature data based on the attention score of the current feature data includes: The HBM divides the input data to be input to the multi-head self-attention structure into blocks to obtain the input vector; The input vector is loaded from the HBM into the SRAM, and the multi-head self-attention structure is run in the SRAM to obtain the attention score. The attention score of the input vector is updated to the HBM.

3. The method according to claim 1, characterized in that, The multi-head self-attention structure fusion rotational position encoding includes: For each self-attention structure in the multi-head self-attention structure, based on a set of self-attention weights of the self-attention structure, the query vector, key vector, and value vector corresponding to the input vector are determined; wherein, the self-attention weights include a first weight for calculating the query vector, a second weight for calculating the key vector, and a third weight for calculating the value vector; Determine the rotation matrix corresponding to each vector element at each position in the input vector; For each vector element, a set of query vector elements and key vector elements are corresponding to the same set of query vector elements and key vector elements. The set of query vector elements and key vector elements are rotated and encoded based on the rotation matrix to obtain the encoded query vector elements and the encoded key vector elements. Perform a dot product between the encoded query vector elements and the encoded key vector elements to obtain a position dependency model; Based on the position dependency model and the value vector, the self-attention score of the self-attention structure is determined; The attention scores are obtained by fusing the self-attention scores corresponding to the self-attention structures of each head based on a preset parameter matrix.

4. The method according to claim 1, characterized in that, The encoder model further includes: a depthwise separable convolutional network structure based on a gating mechanism, wherein the depthwise separable convolutional network structure is connected to the multi-head self-attention structure; the depthwise separable convolutional network structure includes at least: The first pointwise convolutional layer is used to perform interactive calculations on data from different input channels to obtain channel interaction information. A gated layer is used to select information based on the channel interaction information and transmit the selected first feature map to the deep convolutional layer. The deep convolutional layer is used to independently perform convolution operations on each channel of the input first feature map to obtain the feature map corresponding to each channel; The second pointwise convolutional layer is used to combine and transform the different channel data of the input second feature map to obtain a new feature map. The second feature map is determined based on the feature maps corresponding to each channel output by the deep convolutional layer.

5. The method according to claim 4, characterized in that, The depthwise separable convolutional network structure also includes: The first normalization layer is used to normalize the attention score output by the multi-head self-attention structure and output the first normalization result to the first pointwise convolutional layer. The normalization result includes the data of the different input channels. The batch normalization layer is used to normalize the feature map corresponding to each channel output by the deep convolutional layer, and output the obtained second normalization result to the activation layer. The activation layer is used to adjust the activation level based on the second normalization result to obtain the second feature map, which includes the feature maps of each channel after activation.

6. The method according to claim 1, characterized in that, The base recognition model includes the convolutional neural network model, the encoder model, and the CTC decoder model, which is trained in conjunction with a bidirectional decoder. For each sequence output position, the bidirectional decoder is used to determine the decoding result of the output position by combining the context information of the output position.

7. The method according to claim 6, characterized in that, The training process of the base recognition model includes: Obtain a training set, which includes multiple sets of training data, each set of training data including sample current data and base sequence labels corresponding to the sample current data; The sample current data is input into a pre-created neural network model to obtain the model output; the neural network model includes an initial convolutional neural network model, an initial encoder model, an initial CTC decoder connected to the initial encoder model, and the bidirectional decoder; the model output includes a first output result from the initial CTC decoder and a second output result from the bidirectional decoder. The first output result and the base sequence tag are input into a preset CTC loss function to obtain the CTC loss value; The second output result and the base sequence tag are input into a preset KL divergence loss function to obtain the KL divergence loss value; The neural network model is iteratively trained by combining the CTC loss value and the KL divergence loss value to obtain the base recognition model, which includes a trained convolutional neural network model, a trained encoder model, and a trained CTC decoder.

8. The method according to any one of claims 1 to 7, characterized in that, The encoder model also includes a first forward propagation layer, a depthwise separable convolutional network structure based on a gating mechanism, a second forward propagation layer, and a second normalization layer. The output features of the first forward propagation layer and the input features of the first forward propagation layer are merged, and the merged first feature is output to the multi-head self-attention structure; the first forward propagation layer is used to extract features from the input features; After the first merged feature is merged with the attention score output by the multi-head self-attention structure, the merged second merged feature is output to the depthwise separable convolutional network structure. The new feature map output by the depthwise separable convolutional network structure is merged with the second merged feature, and the resulting third merged feature is output to the second forward propagation layer; the second forward propagation layer is used to extract features from the third merged feature. After the output feature of the second forward propagation layer is merged with the third merged feature, the merged fourth merged feature is output to the second normalization layer for normalization processing to obtain the third normalization result.

9. A base recognition device for nanopore sequencing, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the base recognition method for nanopore sequencing as described in any one of claims 1 to 8 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the base recognition method for nanopore sequencing as described in any one of claims 1 to 8.

Images