MRNA codon sequence language modeling method based on bidirectional Mama architecture

Abstract

The invention relates to an mRNA (messenger Ribonucleic Acid) codon sequence language modeling method based on a bidirectional Mama framework, which comprises the following steps of: acquiring an mRNA coding sequence, performing sequence tokenization on the mRNA coding sequence, and querying by adopting a word list to obtain an embedding characteristic; the embedded features are sequentially input into a convolution feature extraction module and a state space module of a bidirectional Mamba model to generate collaborative representation, and meanwhile, each time step serves as an affine pair and is used for carrying out inter-block synthesis on block-level pairs to obtain a synthesis result; and inputting the collaborative representation and synthesis result into a bidirectional branch and linear fusion module to obtain a linear transformation fusion result, projecting the linear transformation fusion result to a word list dimension, obtaining a cross entropy and a model parameter used for training a bidirectional Mama model, and generating a target dimension vector of each sequence by using the trained bidirectional Mama model. According to the invention, the accuracy and expandability of mRNA sequence analysis and design can be improved.

Description

Technical Field

[0001] This invention relates to the fields of bioinformatics and biological sequence modeling, and in particular to a method for modeling mRNA codon sequence language based on a bidirectional Mamba architecture. Background Technology

[0002] Messenger RNA (mRNA) vaccines have significantly accelerated vaccine development and immunization in recent years, with their rapid deployment and remarkable efficacy highlighting the importance of this technology. Compared to traditional vaccine technology platforms, mRNA vaccines do not require cell culture, can rapidly adapt to emerging pathogens, and possess advantages such as inducing potent immune responses and good safety profiles.

[0003] However, optimizing mRNA sequences to achieve higher expression levels, greater stability, and lower non-specific immunogenicity remains a core challenge in achieving large-scale application and improving vaccine efficacy. One fundamental reason for this challenge is the degeneracy of the genetic code. For the same protein amino acid sequence, there are often multiple synonymous coding DNA sequences (CDS). Although these different CDS are completely equivalent at the protein level, they may exhibit drastically different translational kinetics, transcript stability, and protein yield in actual expression. Therefore, the rational selection and optimization of codons has become a crucial step in the mRNA vaccine design process.

[0004] Existing optimization methods, such as heuristics based on codon fitness index (CAI), often struggle to capture the complex patterns hidden in synonym codon selection and fail to balance local preferences with overall structural features. In recent years, inspired by the rapid development of Natural Language Processing (NLP), large-scale language models have been increasingly introduced into biological sequence analysis, such as protein and nucleic acid sequence modeling. Studies have shown that language models based on codon-level and mRNA-specific approaches can simultaneously characterize local synonym codon usage preferences and global long-program sequence dependencies. However, existing models generally suffer from limitations in input sequence length during training, making it difficult to directly process complete mRNA sequences containing thousands of nucleotides. They also suffer from excessive computational / memory overhead and difficulty in capturing global dependencies, thus limiting their applicability in downstream tasks. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide an mRNA codon sequence language modeling method based on a bidirectional Mamba architecture. This method models large-scale mRNA transcripts using a state space model (SSM) of the bidirectional Mamba model and combines codon-level characterization with bidirectional Mamba modules. This allows for the modeling and optimization of mRNA sequences while preserving the protein amino acid sequence, effectively improving translation efficiency, transcript stability, and immunogenicity control. Furthermore, by utilizing state space recursion and bidirectional linear fusion for joint modeling, it retains local synonymous codon biases while efficiently capturing global long-range dependencies, supporting downstream tasks such as mRNA stability, expression levels, and immunogenicity.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for language modeling of mRNA codon sequences based on a bidirectional Mamba architecture includes: Obtain the mRNA coding sequence, tokenize the mRNA coding sequence, and use a vocabulary lookup to obtain the embedding features; The embedded features are sequentially input into the convolutional feature extraction module and the state space module of the bidirectional Mamba model to generate a collaborative representation. At the same time, each time step is used as an affine pair to synthesize the block-level pairs between blocks and obtain the synthesis result. The collaborative representation and the synthesis result are input into the bidirectional branching and linear fusion module to obtain the linear transformation fusion result. The linear transformation fusion result is then projected onto the vocabulary dimension to obtain the cross-entropy, which is used to train the model parameters of the bidirectional Mamba model. Using the trained bidirectional Mamba model, the target dimension vector of each sequence is generated.

[0007] Optionally, sequence tokenization of the mRNA coding sequence includes: The mRNA coding sequence is divided into codon sequences according to the triplet, and a vocabulary is constructed; the vocabulary includes: standard codons and special tokens.

[0008] Optionally, obtaining the embedding features includes: Based on the vocabulary, the embedding features corresponding to the codon sequence are obtained and normalized position by position to obtain the embedding features: ; in, For embedding features, To achieve position-by-position normalization, The embedded features are to be normalized.

[0009] Optionally, generating the collaborative representation includes: The embedded features are input into the convolutional feature extraction module, which applies depth-separable one-dimensional convolution and pointwise convolution to the embedded features to generate intermediate features. The intermediate features are input into the state space module, which sets a hidden state of the target dimension for each channel to discretize the state space and process the intermediate features. The output results of each channel are stacked and normalized by residuals and layers to generate the collaborative representation.

[0010] Optionally, generating the intermediate features includes: ; in, As an intermediate feature, , These are the pointwise convolution weights and biases, respectively. This is the two-dimensional feature matrix output by the convolutional feature extraction module. Let it be its row vector at time step t.

[0011] Optionally, discretizing the state space includes: ; ; in, Let be the hidden state of channel c at time step t. Here is the state transition matrix. This is the hidden state from the previous time step. Let c be the input matrix. For the input vector, For output, To read the matrix, It is a through matrix.

[0012] Optionally, generating the collaborative representation includes: ; in, For collaborative characterization, For embedding features, The output results are stacked for each channel.

[0013] Optionally, obtaining the synthesis result includes: Each time step is treated as an affine pair, and the binary operators of the affine pairs are scanned using a defined block. Intra-block prefix scanning is performed within blocks of the target block length, and inter-block composition is performed on the block-level pairs using the binary operators to obtain the composition result. ; ; in, In hidden state, For the state transition after synthesis, This is the initial state. The input contribution after synthesis It is a binary operator. For the first Each time step is processed by the discretized state transition matrix. For the first Each time step is processed by the discretized input matrix. Let be the input vector at the s-th time step.

[0014] Optionally, obtaining the linear transformation fusion result includes: The collaborative representation and the synthesis result are stacked in multiple layers, and forward and reverse calculations and inversion alignment are performed. Further linear transformation fusion is then applied to obtain the linear transformation fusion result, denoted as... ,remember : ; ; in, The linear transformation weight matrix is ​​for bidirectional fusion. For bias, For bidirectional fusion output, This represents the sequence time reversal operator. This indicates concatenation along the channel dimension. The fused representation after time step t, This is the final output after bidirectional fusion. The hidden representation obtained by forward scanning, The hidden representation obtained by reverse scanning, The bias for linear fusion. To put all The sequence representation matrix obtained by stacking in the time dimension.

[0015] Optionally, obtaining the cross-entropy includes: ; in, Average cross-entropy loss for modeling masked languages, For the set of mask locations, For sequence position index, In a given representation Below, the model assigns conditional probabilities to each symbol in the vocabulary. Let be a discrete symbolic random variable at position t. This is the actual label for that location. Let t be the context representation of the position.

[0016] The beneficial effects of this invention are as follows: This invention utilizes a block-prefix scan Mamba recursion, achieving near-linear time / memory complexity. Growth, adapting to complete mRNA sequences of thousands of nt, utilizing convolutional pre-enhancement to extract local patterns, SSM recursion and bidirectional linear fusion to capture long-range dependencies, achieving stable context alignment even without position coding and gating constraints, and sharing forward and reverse pathway parameters (except for the temporal direction), combined with static diagonal SSM, reducing trainable parameters and improving cross-species and cross-task transfer capabilities, further unifying symbols and module interfaces ( With explicit training hyperparameters (AdamW, MLM masking strategies), it is easy to reproduce and implement in downstream tasks (such as stability / expression level / immunogenicity prediction). Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a framework diagram of an mRNA codon sequence language modeling method based on a bidirectional Mamba architecture according to an embodiment of the present invention; Figure 2 A visualization of t-SNE for model codon embeddings in an embodiment of the present invention; Figure 3 These are the test results for downstream tasks in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] like Figure 1 As shown, this embodiment discloses a method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture, including: Sequence tokenization and vocabulary: Dividing the coding region into codon sequences according to triplets. Constructing a vocabulary It contains 64 standard codons and special symbols. <cls> , <eos> , <pad> , <mask> , <unk>Therefore .

[0022] Embedding and Normalization: Batch Size is , length is Channel maintenance (Preferred) ). Table lookup yields embedded information: ;in, For normalization, As a numerically stable constant, no explicit position coding is added throughout the entire process.

[0023] Convolutional feature extraction: 1D convolution and pointwise convolution can be separated at the initial depth of each SSM layer to extract local patterns. ; Where B is the batch size and L is the sequence length. For location index, Here, d represents the channel index, and d represents the number of channels. The convolution kernel for channel c, The input features are used to enter this convolutional block, and k is the kernel length; Use the Sigmoid activation function; , These are the pointwise convolution weights and biases, respectively; for Applying a Sigmoid nonlinear function , , This is the two-dimensional feature matrix output by the convolutional feature extraction module. Let it be its row vector at time step t.

[0024] SSM state space unit: for each channel set up Dimensional state (preferred) ): ; in, Let be the hidden state of channel c at time step t. For the input vector, ; Here is the state transition matrix. This is the hidden state from the previous time step. Let c be the input matrix. For output, To read the matrix, For direct matrixing.

[0025] ; in, For operations based on element exponents, the step size is... Diagonalization discretization, This represents element-wise multiplication. Let these represent the continuous-time diagonal parameters and the continuous-time input coupling vector, respectively, initialized to... To ensure Stacking channels Apply residuals and layer normalization: ; in, For collaborative characterization, For embedding features, For element-wise nonlinearity, The output results are stacked for each channel.

[0026] Block-based scan synthesis: Each time step is treated as an affine pair, and a binary operator is used to scan the affine pair using a defined block. Intra-block prefix scans are performed within blocks of the target block length, and inter-block synthesis is performed on the block-level pairs using the binary operator to obtain the synthesis result. This achieves near-linear time and memory complexity consistent with training / inference. ; ; in, In hidden state, For the state transition after synthesis, This is the initial state. The input contribution after synthesis It is a binary operator. For the first Each time step is processed by the discretized state transition matrix. For the first Each time step is processed by the discretized input matrix. Let be the input vector at the s-th time step.

[0027] Bidirectional branching and linear fusion: The collaborative representation and the synthesis result are stacked in multiple layers, and forward and reverse calculations and inversion alignment are performed. Further linear transformation fusion is then applied to obtain the linear transformation fusion result. (Note: The original text contains some formatting errors and inconsistencies. A more accurate translation would require the full context.) ,remember Specifically, with Layer stacking (preferred) ) respectively in the positive direction and reverse The above calculation is performed, where L is the sequence length.

[0028] ; ; in, For bias, For bidirectional fusion output, This represents the sequence time reversal operator. This indicates concatenation along the channel dimension. The fused representation after time step t, This is the final output after bidirectional fusion. The hidden representation obtained by forward scanning, The hidden representation obtained by reverse scanning, The bias for linear fusion. To put all The sequence representation matrix obtained by stacking in the time dimension.

[0029] Output head and MLM loss: ; in, For output layer weights, For bias, In a given representation Below, the model assigns conditional probabilities to each symbol in the vocabulary, where j is the index of a word in the vocabulary. For the softmax function, Represents the probability distribution of the vocabulary, mask set (Each position is) The probability of selection is preferred. ; in the selected position Set as <mask>, Random replacement Calculate cross-entropy (keeping the original values): ; in, Average cross-entropy loss for modeling masked languages, For the set of mask locations, For sequence position index, Let be a discrete symbolic random variable at position t. This is the actual label for that location. Let t be the context representation of the position.

[0030] Training setup (preferred): Optimizer AdamW ( weightdecay ), initial learning rate warmup Step, cosine annealing; gradient clipping ;Residual Dropout Mixed precision (FP16). Maximum length. .

[0031] Sequence representation extraction after model training: The optimized model can generate corresponding representation vectors for any input mRNA sequence, forming a two-level representation. With encoder parameters frozen and in evaluation mode, the target sequence is input and forward processed to obtain... This is a high-dimensional embedding representation of the entire mRNA sequence, which can be used as a feature input for downstream tasks.

[0032] Positional representation: for valid positions (ignoring) <pad>Optional to ignore <cls> , <eos>Take the final layer vector: ; in, B is the sample index, and B is the batch size. Where L is the position index, L is the sequence length, and d is the representation dimension. This is the bidirectional fusion representation of the b-th sample at position t.

[0033] Sequence-level representation: Can be further excluded <cls> , <eos>The aggregation method includes at least one of the following: —Mean pooling: ; —Initial symbol pooling: When set When the start symbol is: ; Where N is the number of encoder stack layers, Let be the token of the b-th sample at position t. This represents the vector representing the output sample b at position t of the Nth layer. for <cls>The representation of in the Nth layer.

[0034] Downstream task capability testing of the model: In the pre-trained encoder (bidirectional Mamba-2, Layer output Based on this, take the sequence representation (Default: take) <eos>Location; optional: average pooling) Append a single-layer linear head to each downstream dataset and perform full fine-tuning (unfreeze the encoder and train it with the linear head): Multiple categories: ; in, For the category head weight, , For classification head offset, ; The Pearson correlation coefficient is used. Let c be a category vector, and c be the category. For the i-th sample in the true category The predicted probability at point B is the number of samples in the current mini-batch.

[0035] Two-category classification: ; in The probability of being a positive class. It is Sigmoid. b is the classification weight, and b is the bias. Let be the true binary label of the i-th sample.

[0036] return: ; in For scalar regression prediction, b is the classification weight, and b is the bias. Let i be the true label of the i-th sample. The Pearson correlation coefficient is used. The weights for relevance regularization.

[0037] Fine-tuning training protocol (unified): AdamW; encoder learning rate Linear head learning rate Warmup: 10% steps + cosine annealing; batch size: 128 (adaptive to GPU memory and task sequence length); early stopping (patient 10 rounds); partition for training / validation / testing (80 / 10 / 10); use a fixed random seed {42,43,44} repeated three times, reporting mean ± standard deviation. Input length should not exceed the maximum sequence length for each task, and encoder superparticipation should be consistent during pre-training.

[0038] Eight downstream tasks and objectives: 1. mRNA stability (regression): indicator Pearson 2. Tc-Riboswitches (Regression): Pearson indicator 3. mRFP expression (regression): Pearson indicator 4. SARS-CoV-2 vaccine degradation (regression): Pearson index 5. E. coli protein expression (3-category): indicator Accuracy; 6. Fungal expression (regression): indicator Pearson's regression. 7. MLOS Influenza Vaccine (Regression): Pearson Indicator 8. SwissProt subcellular localization (multi-class, 10 categories): indicators Accuracy, Weighted-F1.

[0039] Results Presentation and Statistics: The mean and standard deviation of each task across three replicates on the test set are summarized in... Figure 3 .

[0040] The invention will be further described below with reference to the accompanying drawings: Figure 1 As shown, the mRNA language model of this invention is a Mamba-based bidirectional encoder used to model the codon tokens of mRNA. Its core consists of convolutional enhancement, SSM (Mamba), bidirectional scanning, and linear fusion. The main steps are as follows: Data and Tokenization: Before feeding the mRNA sequence into the pre-trained bidirectional Mamba model, the input mRNA sequence needs to be segmented. Specifically, the input mRNA sequence is segmented into 64 codons, with each codon treated as an independent unit. Figure 1 The diagram illustrates the processing steps for the input sequence. The input mRNA sequence is first segmented into 64 codon units, then masked to generate a masked sequence, which is then input into the bidirectional Mamba model for training. Through this segmentation method, bidirectional Mamba can efficiently process and learn codon patterns in mRNA sequences, thus providing strong support for subsequent biological analysis tasks. Details of this process are as follows: Pre-training Data Preparation: The pre-training data in this example comes from the European Nucleotide Archive (ENA), with a data cutoff date of April 2022. The data has undergone rigorous processing in the publicly available research CaLM, including removing low-quality sequences, filtering outlier codons, and clustering redundant sequences, ultimately yielding 8,771,938 high-quality mRNA coding sequences. This dataset covers multiple species, is highly representative, and can be used for subsequent unsupervised masked language model pre-training.

[0041] Pre-training dataset partitioning: This invention uses 8,771,938 mRNA coding sequences from the CaLM dataset for pre-training. Based on experimental requirements, this invention decides to use 1% of the dataset as the validation set. That is, from the 8,771,938 sequences in the dataset, 1% (i.e., 87,719 sequences) is randomly selected as the validation set, and the remaining data (i.e., 8,684,219 sequences) is used as the training set.

[0042] Segmentation and Vocabulary: Codon tokens are split into triplets; the vocabulary contains 64 standard codons and... <cls> , <eos> , <pad> , <mask> , <unk>common .

[0043] Masking: To perform self-supervised learning, 15% of the cipher tokens in each batch are randomly selected for masking. The selection method for masked tokens is as follows: 80% of the masked tokens are replaced with predefined ones. <mask>Of the tokens, 10% are randomly replaced with other codons, and 10% retain their original values. Input sequence formatting: To ensure consistent input length, all sequences are padded to the length of the longest sequence in the batch, ensuring data consistency during training.

[0044] Input tensor shape and embedding: Let the batch size be... The sequence length is Channel dimension is ( ), token index The embedded information was obtained by looking up the table: ; in, For position-by-position normalization ( ).

[0045] Convolutional Enhancement: To extract local patterns, a depthwise separable one-dimensional convolution (kernel length) is applied before each SSM layer. ,default (Step size 1, samepadding): right Each channel is calculated independently. ; then use Pointwise convolution to blend channels and apply Sigmoid: ; Where B is the batch size and L is the sequence length. For location index, Here, d represents the channel index, and d represents the number of channels. The convolution kernel for channel c, The input features are used to enter this convolutional block, and k is the kernel length; Use the Sigmoid activation function; , These are the pointwise convolution weights and biases, respectively; for Applying a Sigmoid nonlinear function , , This is the two-dimensional feature matrix output by the convolutional feature extraction module. Let it be its row vector at time step t.

[0046] SSM state space unit: for each channel set up Dimensional state (preferred) ): ; in, Let be the hidden state of channel c at time step t. For the input vector, ; Here is the state transition matrix. This is the hidden state from the previous time step. Let c be the input matrix. For output, To read the matrix, For direct matrixing.

[0047] ; in, For operations based on element exponents, the step size is... Diagonalization discretization, This represents element-wise multiplication. Let these represent the continuous-time diagonal parameters and the continuous-time input coupling vector, respectively, initialized to... To ensure Stacking channels Apply residuals and layer normalization: ; in, For collaborative characterization, For embedding features, For element-wise nonlinearity, The output results are stacked for each channel.

[0048] Block-based scan synthesis (near-linear complexity): Each time step is treated as an affine pair, and a binary operator is used to scan the affine pairs using a defined block. Intra-block prefix scans are performed within blocks of the target block length, and inter-block synthesis is performed on the block-level pairs using the defined binary operator to obtain the synthesis result, achieving near-linear time and memory complexity consistent with training / inference. ; ; in, In hidden state, For the state transition after synthesis, This is the initial state. The input contribution after synthesis It is a binary operator. For the first Each time step is processed by the discretized state transition matrix. For the first Each time step is processed by the discretized input matrix. Let be the input vector at the s-th time step.

[0049] Bidirectional branching and linear fusion: The collaborative representation and the synthesis result are stacked in multiple layers, and forward and reverse calculations and inversion alignment are performed. Further linear transformation fusion is then applied to obtain the linear transformation fusion result. (Note: The original text contains some formatting errors and inconsistencies. A more accurate translation would require the full context.) ,remember Specifically, with Layer stacking ( ) respectively in the positive direction and reverse The above calculation is performed, where L is the sequence length.

[0050] ; ; in, For bias, For bidirectional fusion output, This represents the sequence time reversal operator. This indicates concatenation along the channel dimension. The fused representation after time step t, This is the final output after bidirectional fusion. The hidden representation obtained by forward scanning, The hidden representation obtained by reverse scanning, The bias for linear fusion. To put all The sequence representation matrix obtained by stacking in the time dimension.

[0051] Output head and MLM loss: ; in, For output layer weights, For bias, In a given representation Below, the model assigns conditional probabilities to each symbol in the vocabulary, where j is the index of a word in the vocabulary. For the softmax function, Represents the probability distribution of the vocabulary, mask set (Each position is) The probability of selection is preferred. ; in the selected position Set as <mask>, Random replacement Calculate cross-entropy (keeping the original values): ; in, Average cross-entropy loss for modeling masked languages, For the set of mask locations, For sequence position index, Let be a discrete symbolic random variable at position t. This is the actual label for that location. Let t be the context representation of the position.

[0052] Training parameters and environment: Optimizer: AdamW ( (weightdecay0.01); lr The warmup process lasts 1500 steps. During the subsequent training process, a cosine decay strategy is used to adjust the learning rate until the training ends.

[0053] Batch size and length: , ; block Gradient accumulation is performed every 4 batches to effectively increase batch size.

[0054] Regularization: Dropout 0.1 (after residual); Gradient clipping 1.0; Mixed precision (FP16).

[0055] Early stopping: Stop when the validation set loss function is no longer optimized.

[0056] Environment: PyTorch 2.3+, CUDA 12+; 4 A100 80GB data parallel.

[0057] Sequence representation extraction after language modeling: This embodiment provides methods for obtaining token-level and sequence-level mRNA embeddings after the bidirectional Mamba encoder completes forward modeling. These methods can be directly used for tasks such as similarity retrieval, visualization clustering, downstream supervised / self-supervised fine-tuning, and restricted synonymous codon design. Unless otherwise stated, encoder parameters are frozen by default, and representations are extracted in evaluation mode (with Dropout disabled and mixed precision enabled optional). Figure 2 The t-SNE visualization of the model's codon embeddings was tested, demonstrating that the model can accurately distinguish between synonymous and non-synonymous codons.

[0058] Representing hierarchy and symbols: Let the input batch be The final bidirectional stacked output is (Default d=768), its first Layer output is denoted as (In this embodiment, N=8). Positional representation: The context representation of the t-th position of the b-th sample is as follows: From the selected layer (default is the final layer) Sequence-level representation: Aggregation of the b-th sample yields... Masking: Ignored during pooling. <pad>Position and optional omission <cls> , <eos>The location.

[0059] Location-level (token-level) indicates retrieval; context-based representation indicates: ; in, B is the sample index, and B is the batch size. Where L is the position index, L is the sequence length, and d is the representation dimension. This provides a bidirectional fusion representation of the b-th sample at position t. It serves as the feature basis for site-level scoring, interpretable analysis, and local replacement proposals.

[0060] Sequence-level representation acquisition: From the final layer output of the bidirectional Mamba encoder, each sequence is obtained as a fixed-dimensional vector (default d=768) for retrieval, clustering, or downstream tasks.

[0061] Let the set of valid indices for the b-th sequence be . Can be further excluded <cls> , <eos>There are two possible solutions: (A) Mean pooling: ; (B) CLS pooling (if enabled) (Start character): ; Where N is the number of encoder stack layers, Let be the token of the b-th sample at position t. This represents the vector representing the output sample b at position t of the Nth layer. for <cls>The representation of in the Nth layer.

[0062] Model representation capability test (downstream task) test ( Figure 3 ): In the pre-trained encoder (bidirectional Mamba, Layer output Based on this, take the sequence representation Average pooling; or take <cls>(Location) Append a single-layer linear head to each downstream dataset and perform full fine-tuning (i.e., unfreeze all encoder parameters and train with the linear head): ; in, For the category head weight, , For classification head offset, ; The Pearson correlation coefficient is used. Let c be a category vector, and c be the category. For the i-th sample in the true category The predicted probability at point B is the number of samples in the current mini-batch.

[0063] ; in The probability of being a positive class. It is Sigmoid. b is the classification weight, and b is the bias. Let be the true binary label of the i-th sample.

[0064] ; in, For scalar regression prediction, b is the classification weight, and b is the bias. Let i be the true label of the i-th sample. The Pearson correlation coefficient is used. The weights are for relevance regularization. Evaluation metrics are detailed in the Evaluation Metrics section below.

[0065] Evaluation Metrics: To comprehensively evaluate the performance of bidirectional Mamba, this invention uses the following common evaluation metrics: - Accuracy: ; The second equal sign represents the general form of single-label multi-class classification. This represents the number of samples.

[0066] The F1 score is derived from the definition of precision and recall in binary classification. ; Then F1 is their harmonic mean: ; ; For each category Calculated using a binary classification method: "positive for this category, negative for the rest". , For the first Number of real samples This represents the number of categories.

[0067] (Multi-label example-based version) Let the first... The set of true labels for each sample is The prediction set is ,but: ; Pearson correlation coefficient Let the measured value be... Predicted value : ; ; in, For sample covariance, and The standard deviation of the sample is 1. .

[0068] Training Protocol (Unified Settings): Optimizer and Scheduler: AdamW ( (weightdecay 0.01); encoder learning rate Linear head learning rate { , Warmup 10% steps, followed by cosine annealing; gradient clipping 1.0; residual dropout 0.1; mixed precision (FP16). Batch size and epochs: batchsize 128 (automatically adjusted based on GPU memory and downstream task sequence length), stop early based on validation set metrics (patiently 10 epochs). Partitioning and reproduction: {partition training / validation / test (70 / 15 / 15), five-fold cross-validation}; fixed random seed {42,43,44} repeated three times (five-fold cross-validation does not need to be repeated 3 times), report mean ± standard deviation. Length and input: input length is the maximum length of the downstream task sequence, encoder parameters. Maintain consistency.

[0069] Eight downstream tasks and objectives: (a) mRNA stability (regression): predicting continuous stability values; loss ; Pearson indicator (b) Tc-Riboswitches (regression): predicting expression levels; loss ; Pearson indicator (c) mRFP expression (regression): predicting protein yield; loss ; Pearson indicator (d) SARS-CoV-2 vaccine degradation (regression): predicting continuous indicators related to stability / translation efficiency; loss ; Pearson index. (e) E. coli protein expression (multiclass): predicted expression level (high, medium, low); 3-class expression values; loss ; Indicator Accuracy. (f) Fungal expression (regression): Same as above; Loss ; Pearson indicator (g) MLOS influenza vaccine (regression): predicting protein expression levels; loss ; Pearson indicator (h) SwissProt subcellular localization (multi-class): 10 classes; loss The indicator is Weighted-F1.

[0070] Test Result Report: The average model performance for each task on the test set with 3 random seeds is as follows: Figure 3 As shown; the mean and standard deviation are given inside, with the standard deviation in parentheses.

[0071] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.< / cls> < / cls> < / eos> < / cls> < / eos> < / cls> < / pad> < / mask> < / mask> < / unk> < / mask> < / pad> < / eos> < / cls> < / eos> < / cls> < / eos> < / cls> < / eos> < / cls> < / pad> < / mask> < / unk> < / mask> < / pad> < / eos> < / cls>

Claims

1. A method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture, characterized in that, include: Obtain the mRNA coding sequence, tokenize the mRNA coding sequence, and use a vocabulary lookup to obtain the embedding features; The embedded features are sequentially input into the convolutional feature extraction module and the state space module of the bidirectional Mamba model to generate a collaborative representation. At the same time, each time step is used as an affine pair to synthesize the block-level pairs between blocks and obtain the synthesis result. The collaborative representation and the synthesis result are input into the bidirectional branching and linear fusion module to obtain the linear transformation fusion result. The linear transformation fusion result is then projected onto the vocabulary dimension to obtain the cross-entropy, which is used to train the model parameters of the bidirectional Mamba model. Using the trained bidirectional Mamba model, the target dimension vector of each sequence is generated.

2. The method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture according to claim 1, characterized in that, Sequence tokenization of the mRNA coding sequence includes: The mRNA coding sequence is divided into codon sequences according to the triplet, and a vocabulary is constructed; the vocabulary includes: standard codons and special tokens.

3. The mRNA codon sequence language modeling method based on bidirectional Mamba architecture according to claim 2, characterized in that, Obtaining the embedded features includes: Based on the vocabulary, the embedding features corresponding to the codon sequence are obtained and normalized positionally to obtain the embedding features: ； in, For embedding features, To achieve position-by-position normalization, The embedded features are to be normalized.

4. The method for mRNA codon sequence language modeling based on bidirectional Mamba architecture according to claim 1, characterized in that, Generating the collaborative representation includes: The embedded features are input into the convolutional feature extraction module, which applies depth-separable one-dimensional convolution and pointwise convolution to the embedded features to generate intermediate features. The intermediate features are input into the state space module, which sets a hidden state of the target dimension for each channel to discretize the state space and process the intermediate features. The output results of each channel are stacked and normalized by residuals and layers to generate the collaborative representation.

5. The mRNA codon sequence language modeling method based on bidirectional Mamba architecture according to claim 4, characterized in that, Generating the intermediate features includes: ； in, As an intermediate feature, , These represent the pointwise convolution weights and biases, respectively. This is the two-dimensional feature matrix output by the convolutional feature extraction module. Let it be its row vector at time step t.

6. The method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture according to claim 4, characterized in that, Discretizing the state space includes: ； ； in, Let be the hidden state of channel c at time step t. Here is the state transition matrix. This is the hidden state from the previous time step. Let c be the input matrix. For the input vector, For output, To read the matrix, It is a through matrix.

7. The mRNA codon sequence language modeling method based on bidirectional Mamba architecture according to claim 4, characterized in that, Generating the collaborative representation includes: ； in, For collaborative characterization, For embedding features, The output results are stacked for each channel.

8. The method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture according to claim 1, characterized in that, Obtaining the synthesis result includes: Each time step is treated as an affine pair, and the binary operators of the affine pairs are scanned using a defined block. Intra-block prefix scanning is performed within blocks of the target block length, and inter-block composition is performed on the block-level pairs using the binary operators to obtain the composition result. ； ； in, In hidden state, For the state transition after synthesis, This is the initial state. The input contribution after synthesis It is a binary operator. For the first Each time step is processed by the discretized state transition matrix. For the first Each time step is processed by the discretized input matrix. Let be the input vector at the s-th time step.

9. The method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture according to claim 1, characterized in that, Obtaining the linear transformation fusion result includes: The collaborative representation and the synthesis result are stacked in multiple layers, and forward and reverse calculations and inversion alignment are performed. Further linear transformation fusion is then applied to obtain the linear transformation fusion result, denoted as... ,remember : ； ； in, The linear transformation weight matrix is ​​for bidirectional fusion. For bias, For bidirectional fusion output, This represents the sequence time reversal operator. This indicates concatenation along the channel dimension. The fused representation after time step t, This is the final output after bidirectional fusion. The hidden representation obtained by forward scanning, The hidden representation obtained by reverse scanning, The bias for linear fusion. To put all The sequence representation matrix obtained by stacking in the time dimension.

10. The method for mRNA codon sequence language modeling based on a bidirectional Mamba architecture according to claim 1, characterized in that, Obtaining the cross-entropy includes: ； in, Average cross-entropy loss for modeling masked languages, For the set of mask locations, For sequence position index, In a given representation Below, the model assigns conditional probabilities to each symbol in the vocabulary. Let be a discrete symbolic random variable at position t. This is the actual label for that location. Let t be the context representation of the position.

Images