An RNA sequence classification method based on manba model and semi-supervised learning

Abstract

The application provides an RNA sequence classification method based on a Mamba model and semi-supervised learning, and belongs to the field of bioinformatics. First, multi-scale sparse features are extracted from the RNA sequence, high-dimensional features are compressed into low-dimensional potential space vectors through an encoder network, and L2 normalization is performed to enhance feature separability. Second, a Mamba model based on a selective state space model is introduced in a residual structure to efficiently model long-range dependencies with low complexity. Finally, a semi-supervised learning of unsupervised feature reconstruction and supervised classification is realized by constructing an encoder-decoder structure, and a weighted loss function is used to improve the generalization performance of the model under small sample conditions. Experimental results show that the method significantly improves the F1 score in the multi-class RNA sequence classification task. The application reduces the cost of long sequence modeling and labeling, and is suitable for the fields of RNA function prediction, biomedical research and disease diagnosis.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of bioinformatics, and particularly relates to an RNA sequence classification method based on a Mamba model and semi-supervised learning, which can be used for the identification and classification of non-coding RNA and messenger RNA such as circRNA, IncRNA and mRNA, and includes technologies such as deep learning, semi-supervised learning and state space model. BACKGROUND

[0002] RNA sequence classification is of great significance in bioinformatics and clinical medical research, and plays a key role in non-coding RNA function identification, disease-related RNA marker screening and drug target cell point discovery. RNA sequences are relatively easy to obtain, but high-quality labeled data usually requires professional experiments and expert interpretation, resulting in high labeling cost and limited quantity, which seriously restricts the performance improvement of fully supervised learning classification models.

[0003] At present, mainstream deep learning methods such as attention mechanism-based Transformer perform excellently in sequence modeling, but have high training complexity, which leads to poor processing of high-throughput long sequence RNA data. In addition, RNA sequences are often represented by k-mer features, and the dimension of k-mer increases exponentially with the increase of k value, resulting in high and sparse input feature dimension. Traditional dense storage and processing methods have high memory occupation and low computational efficiency, which reduces the utilization of features in deep model training. Therefore, there is an urgent need for a new RNA sequence classification method that can make full use of unlabeled data, effectively capture long sequence dependencies, and optimize computing resources. SUMMARY

[0004] To overcome the problems of poor sequence modeling ability and low unlabeled RNA utilization of existing technologies, the present application proposes an RNA sequence classification method based on a Mamba model and semi-supervised learning. By constructing a semi-supervised framework based on an encoder-decoder structure and integrating a selective state space model Mamba module, long dependency feature extraction and unsupervised latent representation learning of RNA sequences are effectively realized, and the performance and generalization ability of the classification task are improved. The specific technical scheme of the present application includes 4 steps:

[0005] Step 1, feature extraction and latent space coding. Extract k-mer frequency features from RNA sequences to generate high-dimensional sparse feature vectors. Store the k-mer features in Compressed Sparse Row (CSR) format, then use an encoder network to compress the 1344-dimensional high-dimensional input to a 256-dimensional latent space vector, and perform L2 normalization on the latent space features to enhance the class discrimination.

[0006] Step 2, fusion of Mamba module and attention mechanism. Introduce Mamba module in residual block, dynamically select traditional multi-head attention module and state space Mamba module in the third and fourth residual blocks of the encoder, when the sequence length is less than 300, still use traditional multi-head attention module, when the sequence length is greater than 300, capture long-range dependencies through state transition matrix (complexity O(L)), the module is based on selective state space model SSM theory, generates discrete parameters A, B, C through linear projection, realizes selective state update through gating mechanism, efficiently captures long sequence context information with low complexity; keep residual connection, LayerNorm and Dropout mechanism to ensure the stability and generalization ability of network training.

[0007] Step 3, semi-supervised joint training and loss design. Build a double-path structure to realize unsupervised feature learning by constructing an encoder-decoder path, the encoder maps the input to a 256-dimensional latent space, the decoder reconstructs the original input from the latent space, and the classification path is used for supervised learning tasks, including two-layer perceptron classification head; the encoder serves both the reconstruction task and the classification task, and the latent space features are used for reconstruction and classification after L2 normalization;

[0008] The loss function is designed as the weighted sum of reconstruction loss and classification loss:

[0009]

[0010] Among them, For cross-entropy loss, is the mean square error loss, and is the hyperparameter of the adjustment weight, take 5, take 1, to balance the classification performance and feature reconstruction quality; 128 samples per batch, 20% and 80% of labeled samples and unlabeled samples, respectively, and the batch is constructed by dynamically adjusting the proportion according to the formula In the training process, the gradients of labeled data and unlabeled data are separated, Gaussian noise is added to the unlabeled data to enhance the reconstruction strength, consistency regularization is used to calculate the mean square error of two different dropouts of the same sample; introduce double early stopping mechanism, the main stopping condition is that the F1 score of the validation set does not improve for 15 consecutive rounds, and the auxiliary condition is that the ratio of reconstruction loss to classification loss deviates from the benchmark for more than 10 rounds; strict training monitoring ensures the stability of semi-supervised learning.

[0011] Step 4, classification prediction and feature application. The RNA sequence to be predicted is also input into the trained model after k-mer feature extraction and sparse coding; the model outputs multi-class classification probability and 256-dimensional latent space vector, realizes function prediction, and can be used for downstream function analysis, clustering and visualization, and assists biological research. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 Fig. 1 is a schematic diagram of the overall process of an RNA sequence classification method based on the Mamba model and semi-supervised learning.

[0013] Figure 2 Fig. 2 is a framework diagram of a semi-supervised learning network with an encoder-decoder structure.

[0014] Figure 3 Fig. 3 is a schematic diagram of a network structure integrating a residual block into a Mamba module.

[0015] Figure 4 Fig. 4 is a data flow processing flowchart for model training.

[0016] Figure 5 Fig. 5 is a data flow processing flowchart for the model inference stage. DETAILED DESCRIPTION

[0017] The present application is described in detail below in conjunction with the drawings and examples.

[0018] Feature extraction and latent space coding. Figure 2 The pre-processing of RNA sequence data, k-mer feature construction and latent space vector generation process are shown; data preparation: three types of RNA data in public databases are selected: mRNA, IncRNA and circRNA, with a total of about 61888 samples; the RNA sequence length ranges from 200 to 2000 nt, and sequences containing more than 10% of N are excluded; then the data is cleaned, uracil in the sequence is replaced with thymine, and DNA notation is used uniformly, then sequences containing more than 3 consecutive unknown bases N are deleted and sequences with more than 10% of N bases are excluded; sampling stratification divides the training set, validation set and test set in a ratio of 14:3:3, wherein the labeled samples in the training set account for 15%, and the rest are unlabeled data; k-mer feature extraction: set k = 4, divide the sequence into 4-mer fragments with a step of 1 and a non-overlapping sliding window, a total of 4^4 = 256 possibilities; for each k-mer, count the frequency of all possible k-mers, then normalize the frequency by dividing by the effective sequence length; to enhance feature richness, extract k = 3 and k = 2 features as well. k k k k ​​​​=5 features, and use the horizontal splicing operation of sparse matrix to splice the three types of k-mer features into a sparse matrix with a dimension of 1344; the generated sparse features are stored in CSR format, and the single batch memory occupancy is reduced by about 70%; the latent space encoding uses an improved UnsupervisedResNet architecture, first, the input layer receives the 1344-dimensional k-mer sparse feature vector and performs dense mapping; second, the 1344-dimensional feature is compressed to a 1024-dimensional dense vector through the input projection layer; then, the data passes through four residual blocks composed of multi-head attention and Mamba, of which the first two blocks are composed of multi-head attention, and the last two blocks are composed of Mamba modules containing linear layers, Batch, and GELU activation functions; next, the 1024-dimensional feature is compressed to a 256-dimensional latent space vector; finally, the output is connected to the downstream task, which is used for classification tasks and reconstruction tasks respectively.

[0019] Mamba module integration and long sequence modeling. Figure 3 Demonstrates how to integrate the Mamba module into the encoder to improve long sequence modeling capability and optimize GPU usage.

[0020] Mamba module position and structure: in the third and fourth residual blocks of the encoder, when the sequence length is greater than 300, replace the traditional multi-head attention with the Mamba module dynamically, which first receives the feature vector from the previous residual block and inputs it into the normalization layer unit for standardization processing; then, the standardized feature is mapped to the parameters required by the state update distribution and branch through linear projection, and one-dimensional causal convolution is performed on the input branch to capture the context information; then, the state is updated according to the selected state space model, the time evolution of the hidden state is realized through the state transition matrix, and the new input is injected combined with the gate control mechanism; then, the updated state passes through the output projection generation module to generate the output feature; finally, the output and input features are stacked through the residual link, and Dropout is performed to enhance the stability and generalization ability of network training, obtaining the output of the Mamba module; parameter configuration: input / output dimension: 1024, state dimension: 16, convolution kernel size: 4, expansion factor: 2; the Mamba module is based on the selective state space model SSM, which captures long-range dependencies through the state transition matrix, and the input projection maps the input x to two branches (x, z) through a linear layer, one-dimensional causal convolution is performed on the x branch: kernel_size=4, padding=3, state parameters B: 16-dimensional and C: 32-dimensional are generated through x_proj, and time step parameters dt: 1024-dimensional are generated through dt_proj, then discrete calculation is performed, and the state transition matrix A is the core component of the Mamba module, which is used to control the time evolution characteristics of the hidden state, and its initialization is ; the state update is realized through the following discrete equation:

[0021] wherein corresponding memory decay, denotes new input injection, , t is dynamically generated by input features; to optimize memory, gradient checkpointing technique is used to reduce activation storage and mixed precision training is adopted; each residual block contains convolution units, LayerNorm, Dropout of 0.1 and residual connection to ensure gradient stability.

[0022] Semi-supervised joint training and loss design. Figure 4 Show how to improve the performance of the model under the condition of small sample by jointly optimizing the classification task and the reconstruction task.

[0023] Decoder and classifier design: the decoder uses a symmetric 4-layer convolutional network to reconstruct the 256-dimensional latent vector layer by layer into a 1344-dimensional dense k-mer vector for unsupervised reconstruction task; the classification head is a two-layer fully connected network, which outputs a three-class probability distribution.

[0024] Loss function design: the overall loss function is ,

[0025] wherein is the cross-entropy loss; is the mean square error loss; the weight is taken to ensure that the classification performance is prioritized; training strategy: the optimizer uses AdamW, the initial learning rate is 1e-5, the weight decay is 1e-5, the parameters are (0.9, 0.999); the batch size is 128, each batch contains about 26 labeled samples, accounting for 20% of the 1400 randomly selected samples, and 102 unlabeled samples; labeled samples are used to calculate classification loss and reconstruction loss, while unlabeled samples are only used for reconstruction loss calculation; by setting the classification loss weight and the reconstruction loss weight respectively, the model can prioritize classification performance while learning more general latent feature representation using unlabeled data; train for 150 epochs; in terms of learning rate scheduling, use the loss monitoring method, when the loss stagnates for 4 epochs, the learning rate is halved.

[0026] Classification prediction and downstream feature application. Figure 5 Show the inference process: input RNA sequence through the same k-mer feature extraction and sparse coding; only use the encoder and classification head for forward inference, without the participation of the decoder; output multi-class classification probability and 256-dimensional latent space vector; the input RNA sequence is first converted from uracil to thymine, and then filtered for sequences containing more than 3 consecutive N bases or N accounting for >10%; extract k=3, 4, 5 k-mer features, and use the horizontal splicing operation of the sparse matrix to splice the three types of k-mer features into a sparse matrix with a dimension of 1344, load the pre-trained model, disable gradient calculation for model inference; the output result probs represents the probability distribution of mRNA, IncRNA, and circRNA, latent is 256-dimensional floating point data, norm = 1; the downstream application of the feature vector includes t-SNE visualization of the latent vector to realize RNA family clustering; the high-confidence prediction result is used for disease marker screening or functional annotation research; the latent vector can be fused with other multi-omics data for multi-modal biological information analysis.

[0027] The present application uses the low coverage benchmark data set AttenRNA to test the performance of the proposed RNA sequence classification method based on the Mamba model and semi-supervised learning. The data set contains 61888 different types of biological sequence RNA. After the sample of the data set is subjected to the k-mer feature extraction, compressed sparse row storage, encoder network compression and L2 normalization processing of step 1, the feature distribution is on the same hypersphere, then the normalized feature is input into the residual block stacking structure built in step 2, the first two residuals retain the traditional multi-head attention, extract the context features, the third and fourth residual blocks are replaced by the Mamba module, and the long sequence dependence is processed through the selective state space mechanism, finally the hybrid representation of the fused local and global features is output, providing a more discriminative feature basis for downstream tasks; the model is trained on the training set and the validation set of the sample data set, and is tested with the test set, and finally the f1 score of the model is calculated. The method proposed in the present application is trained and tested, and the f1 score value on the data set AttenRNA is 0.9192, which is improved by 2% compared with the AttenRNA target accuracy prediction model with the best performance at present; the present application builds a hybrid framework that combines convolution and Mamba, and designs a semi-supervised loss function for the characteristics of biological sequences, so the performance is higher than that of other existing methods, and the classification ability of different RNA biological sequences can be effectively improved.

[0028] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the present application.

Claims

1. A method for RNA sequence classification based on Manba model and semi-supervised learning, characterized in that, A semi-supervised framework based on encoder-decoder structure is constructed, which integrates the selective state space model Mamba module, including the following steps: When the length of the RNA sequence is greater than 300, the RNA sequence is preprocessed, uracil is replaced with thymine, and sequences containing more than three consecutive N bases or N accounting for more than 10% are removed; k-mer features of k=3, 4, 5 are extracted respectively with a sliding window step of 1, and the horizontal splicing operation of the sparse matrix is used to splice the three types of k-mer features into a sparse matrix with a dimension of 1344, which is compressed into a 256-dimensional latent vector by the encoder and normalized by L2; The encoder input layer receives the 1344-dimensional k-mer sparse feature vector and performs dense mapping; secondly, the 1344-dimensional feature is compressed to a 1024-dimensional dense vector through the input projection layer; then, the data passes through the 4 residual blocks composed of multi-head attention and Mamba, of which the first two blocks are composed of multi-head attention, and the last two blocks are composed of Mamba modules containing linear layers, Batch, and GELU activation functions; next, the 1024-dimensional feature is compressed to a 256-dimensional latent space vector; finally, the output is connected to the downstream task, which is used for classification task and reconstruction task respectively; The Mamba module captures long-range dependencies through a state transition matrix, updates the state with a gating mechanism, linearly projects to generate discrete parameters A, B, C and time step Δt, and retains residual connection, LayerNorm and Dropout to ensure training stability; The decoder uses a symmetric 4-layer convolutional network to reconstruct the 256-dimensional latent vector into a 1344-dimensional dense k-mer vector layer by layer for unsupervised reconstruction task; the classification head is a two-layer fully connected network, which outputs a three-class probability distribution, and the encoder-decoder is used to realize semi-supervised joint training, and the three-class probability refers to the probability of RNA belonging to mRNA, IncRNA and circRNA; The input RNA sequence to be classified is processed and outputs multi-class classification probability and 256-dimensional latent feature, which is used for RNA family clustering, function prediction and downstream visualization analysis. 2.The RNA sequence classification method based on the Manba model and semi-supervised learning according to claim 1, wherein, The double early stopping mechanism includes: the main stopping condition is that the F1 score of the validation set does not improve for 15 consecutive rounds, and the auxiliary condition is that the ratio of reconstruction loss to classification loss deviates from the benchmark for more than 10 rounds.

3. The RNA sequence classification method based on the Manba model and semi-supervised learning according to claim 1, characterized in that, The state update of the mamba module is achieved by the following discretization equation: the state transition matrix A is the core component of the mamba module, which is used to control the time evolution characteristics of the hidden state, and the initialization is divided into ; the state update is achieved by the following discretization equation: ; wherein corresponds to memory decay, represents new input injection, , t is dynamically generated by input characteristics; parameter configuration: input / output dimension: 1024, state dimension: 16, convolution kernel size: 4, expansion factor:

2.

4. The RNA sequence classification method based on the Manba model and semi-supervised learning according to claim 1, characterized in that, The semi-supervised joint training loss function is designed as a weighted sum of reconstruction loss and classification loss: ; wherein, is a cross-entropy loss, is a mean square error loss, and is a hyperparameter to adjust the weight, takes 5, takes 1 to balance the classification performance and the feature reconstruction quality.

Images