A method for pre-training RNA sequence and application thereof

Abstract

The present disclosure relates to a method for pre-training RNA sequence and its application. The method comprises: generating readable data for pre-training language model based on RNA sequence; and inputting the readable data into the trained pre-training language model to extract available features of the RNA sequence, wherein the initial bias of the pre-training language model is determined based on a pre-constructed weight matrix, and the weight matrix is constructed based on the probability of forming covalent bond between bases. In some examples of the present application, the above method can be used to extract high-dimensional dense features of the input RNA sequence, and accurate prediction of RNA structure and function can be achieved based on the aforementioned high-dimensional dense features.

Description

Technical Field

[0001] This application relates to the field of bioinformatics, and more specifically, to a method for pre-training RNA sequences and its application. Background Technology

[0002] Nucleic acid molecules play a crucial role in the fundamental biological processes of cells. Among them, RNA (ribonucleic acid), a single-stranded nucleic acid, exists in all living cells and performs a variety of biological functions, including but not limited to the transmission of genetic information, protein synthesis, and cell signal transduction. The structure and function of RNA are inseparable, and elucidating the structure of RNA is of great significance for understanding its functional mechanisms, assisting RNA engineering, and drug development.

[0003] To better understand the structure and function of RNA, researchers have developed various methods to predict RNA structure. These methods include wet-experiment-based, thermodynamic modeling-based, and machine learning-based approaches. In particular, with the development of machine learning technology, methods based on large-scale data have shown great potential in RNA structure prediction. However, due to the high-dimensionality and complexity of RNA sequences, as well as limitations in data volume, existing methods still have many shortcomings, such as poor accuracy in predicting RNA tertiary structures and the inability to account for special secondary structures.

[0004] It is worth noting that pre-training methods in the field of natural language processing offer new insights into solving the RNA structure prediction problem. Pre-trained large language models have achieved great success in natural language processing, demonstrating their ability to capture high-dimensional, dense features and fine-tune them for specific tasks to achieve better results. Similarly, transferring these pre-training methods to the RNA domain holds promise for better feature extraction and aiding in the prediction of RNA structure or function.

[0005] In summary, how to transfer pre-training methods from the field of natural language processing to the field of RNA in order to better extract high-dimensional dense features of RNA and thus assist in the prediction of structure or function has important theoretical and practical significance. Summary of the Invention

[0006] This application is filed by the inventor based on the discovery of the following facts and problems:

[0007] Currently used RNA pre-trained language models follow the basic architecture of BERT. When calculating self-attention in each block, they only consider the information of the one-dimensional sequence bases themselves and ignore the structural information of complementary pairing between bases. However, RNA is different from natural language text. Its function depends to a large extent on the structure formed. Structural information plays a very important role in parsing the semantic features of RNA. Therefore, current pre-trained language models have the problem of insufficient feature extraction in the end.

[0008] This application aims to address at least one of the technical problems existing in the prior art. Therefore, one objective of this application is to provide a method capable of effectively extracting RNA sequence features.

[0009] Another objective of this application is to propose a method that can effectively predict RNA structure and function.

[0010] In a first aspect, this application proposes a method for pre-training RNA sequences. According to embodiments of this application, the method includes: generating readable data for a pre-trained language model based on the RNA sequence; and inputting the readable data into a trained pre-trained language model to extract usable features from the RNA sequence, wherein the initial bias of the pre-trained language model is determined based on a pre-constructed weight matrix, which is constructed based on the probability of covalent bond formation between bases. In some examples of this application, the above method can be used to fully extract high-dimensional dense features from the input RNA sequence, and based on these high-dimensional dense features, accurate prediction of RNA structure (including secondary and tertiary structures) and function can be achieved.

[0011] In a second aspect of this application, a method for predicting RNA sequences is proposed. According to embodiments of this application, the method includes: pre-training the RNA sequence according to the method described in the first aspect of this application to obtain usable features; and inputting the usable features into a prediction model to predict the structure or function of the RNA sequence. In some examples of this application, high-dimensional dense features of the RNA sequence to be tested are extracted based on the method described in the first aspect, and further combined with a specific prediction model to achieve the prediction of the structure or function of the RNA to be tested.

[0012] In a third aspect, this application proposes an apparatus for pre-training RNA sequences. According to embodiments of this application, the apparatus includes: a model-readable data acquisition unit for generating pre-trained language model-readable data based on the RNA sequence; and an RNA sequence usable feature acquisition unit for inputting the readable data into a trained pre-trained language model to extract usable features of the RNA sequence, wherein the initial bias of the pre-trained language model is determined based on a pre-constructed weight matrix, which is constructed based on the probability of covalent bond formation between bases. In some examples of this application, the apparatus can be used to efficiently, accurately, and sufficiently extract high-dimensional dense features of RNA sequences.

[0013] In a fourth aspect, this application proposes a system for predicting RNA sequences. According to embodiments of this application, the system includes: an RNA sequence pre-training device for pre-training the RNA sequence based on the device described in the third aspect of this application to obtain usable features; and a prediction device for inputting the usable features into a prediction model to predict the structure or function of the RNA sequence. In some examples of this application, the above system can be used to accurately predict the structure (secondary structure, tertiary structure, and special structures such as pseudo-knot structures) or function of an input RNA sequence.

[0014] In a fifth aspect, this application provides a computer program product. According to embodiments of this application, the computer program product includes computer instructions that, when some or all of the computer instructions are executed on a computer, cause the method described in the first or second aspect of this application to be performed.

[0015] In an eighth aspect of this application, a computing device is proposed. According to an embodiment of this application, the device includes: a processor and a memory; the memory for storing a computer program; and the processor for executing the computer program to implement the method as described in the first or second aspect of this application.

[0016] In a ninth aspect of this application, a computer-readable storage medium is provided. According to an embodiment of this application, the storage medium includes computer instructions that, when executed by a computer, cause the computer to perform the method described in the first or second aspect of this application.

[0017] In some examples of this application, the aforementioned computer program products, computing devices, and computer-readable storage media achieve highly efficient automation and improve modeling efficiency and accuracy through the automatic execution of computer instructions for pre-training RNA sequences or predicting RNA sequences. Secondly, the instruction-based nature of these methods ensures high consistency and reliability across different environments.

[0018] It should be noted that the features and technical effects described in this article for different aspects can be mutually referenced, and will not be elaborated further here.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0021] Figure 1 This is a schematic diagram of the optimized structure of the pre-trained language model (ERNIE-RNA) according to a specific embodiment of this application;

[0022] Figure 2 This is a schematic diagram of an apparatus for pre-training RNA sequences according to a specific embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a system for predicting RNA sequences according to a specific embodiment of this application;

[0024] Figure 4 This is a schematic diagram illustrating the zero-shot results of the ERNIE-RNA and RNA-FM models capturing RNA secondary structures on the bpRNA-1m test set according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram illustrating the evaluation results of the ERNIE-RNA model in capturing specific structural features according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram showing the performance comparison of the ERNIE-RNA model feature extraction according to an embodiment of this application. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] In this document, unless otherwise stated, the singular forms “a,” “an,” etc., include plural referents (more than one); “a group” or “a plurality” refers to two or more.

[0029] In this document, unless otherwise stated, the terms “comprising” or “including” are open-ended expressions, meaning they include the contents specified in this invention but do not exclude other aspects.

[0030] In this document, unless otherwise stated, the terms “first,” “second,” “third,” “fourth,” etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated; features defined with “first,” “second,” etc., may explicitly or implicitly include one or more of the stated features.

[0031] In this paper, unless otherwise stated, the term "pre-trained language model" refers to an artificial intelligence model trained on large-scale text data, designed to learn the statistical regularities and semantic information of natural language, thereby gaining the ability to understand and generate text. These models are typically trained on massive corpora using self-supervised learning, learning the structure and context of language without the need for manually labeled tags. Typical tasks for pre-trained language models include language modeling, text classification, named entity recognition, and sentiment analysis. They can also serve as the foundation for downstream natural language processing tasks, allowing for customized optimization for specific tasks through fine-tuning and other techniques.

[0032] In this paper, unless otherwise stated, the term "BERT (Bidirectional Encoder Representations from Transformers) model" refers to a pre-trained language model based on the Transformer architecture. It learns lexical representations using bidirectional contextual information through self-supervised learning on large-scale text corpora. The BERT model effectively captures semantic and contextual information between words through two tasks in the pre-training phase: Masked Language Model (MLM) and Next Sentence Prediction (NSP). After pre-training, the BERT model can be fine-tuned or otherwise used for various downstream natural language processing tasks, such as text classification, named entity recognition, and question answering systems, achieving excellent performance on multiple tasks.

[0033] In this paper, unless otherwise stated, the term "multi-head attention mechanism" is a mechanism used to enhance the attention of neural network models when processing sequential data. In this mechanism, attention weights are calculated simultaneously by multiple attention heads, each with its own weight matrix. This design allows the model to learn different representations in different attention spaces, thereby better capturing different features and relationships in the sequential data.

[0034] Specifically, the multi-head attention mechanism first maps the input sequence to multiple subspaces, and then calculates attention weights in each subspace. These attention weights come from corresponding attention heads, each of which can focus on specific relationships at different positions in the sequence. Finally, the outputs of these attention heads are combined to form the final multi-head attention representation, which is used for subsequent model computation.

[0035] Multi-head attention mechanisms offer advantages when processing sequential data because they allow the model to focus on different parts of the input sequence simultaneously, thus better capturing both global and local information within the sequence. This parallel computing design also helps improve the model's computational efficiency, and the complexity and representational power of the model can be balanced by appropriately adjusting the number and parameters of the attention heads.

[0036] In this paper, unless otherwise stated, the term "neural network model" is a computational model consisting of a large number of neurons and their interconnections, where neurons transmit and process information through connections with adjustable parameters. Such models typically consist of an input layer, several hidden layers, and an output layer. Neurons in each hidden and output layer transform weighted inputs into outputs through nonlinear activation functions. Neural network models learn the feature representations and internal structure of the input data, progressively adjusting connection weights and model parameters to learn and predict complex data patterns.

[0037] In this paper, unless otherwise stated, the term "masking prediction" is a self-supervised learning method commonly used in the pre-training of language models. In masking prediction, the model's goal is to predict masked (or blocked) portions of a text sequence based on other words in the context. This method is typically used to process text data where a certain percentage of words are randomly selected and replaced with special markers, such as "[MASK]". The model is then asked to infer these masked words based on other words in the context.

[0038] The advantage of masking prediction lies in its ability to allow models to learn semantic and grammatical knowledge of language in an unsupervised manner. By having the model predict masked words, it must understand the semantics of the sentence given the context and be able to infer the masked words. This method has been widely used in the pre-training of various natural language processing tasks, such as the Masked Language Model (MLM) task in the BERT model.

[0039] In this paper, unless otherwise stated, the term "pseudoknot" refers to a structure in which a portion of a single strand intersects with other regions of that strand. Currently, there are several difficulties in predicting pseudoknot structures: (1) The structure search space is extremely large. The number of permutations of pseudoknots that can form complex crossed double helices in RNA sequences is very large, leading to a combinatorial explosion problem in structure prediction; (2) Difficulty in modeling. Common binary tree recursive structures are difficult to describe pseudoknots, and many models based on thermodynamic prediction of RNA secondary structures cannot predict pseudoknot structures; (3) NP (Non-deterministic polynomial) complete problem. Finding the optimal pseudoknot structure under given sequence conditions is an NP-hard combinatorial optimization problem; (4) Lack of sufficient data. Most solved RNA structures do not contain pseudoknots, and there is very little data available for fine-tuning. However, the structure solution space grows geometrically with the definition of pseudoknots, making it difficult for models to learn the feature space of pseudoknots; (5) Low computational efficiency. There are some simple methods for predicting pseudooknot, but their time complexity is as high as O(N). 6 The computational complexity is too high.

[0040] In recent years, pre-trained language models have developed rapidly in the RNA field, such as RNABERT, RNA-FM, RNA-MSM, CodonBERT, UNI-RNA, and UTR-LM, which have achieved certain successes in RNA-related tasks. These models employ different tasks and techniques during the pre-training stage, such as structure alignment learning, k-mer subsequence encoding, and RNA secondary structure prediction, to improve the model's performance on RNA tasks. However, they also have many problems. For example, the high-dimensional dense vectors extracted by RNA-FM perform even worse than one-hot encoding on basepair and solvent-accessible surface area prediction tasks. In addition, RNA-FM's attention-map performs poorly in zero-shot prediction on the bpRNA-1m test set, capturing almost no RNA secondary structure features. Although UNI-RNA does not currently have a publicly available checkpoint for RNA secondary structure prediction, a web server is available for access. After random sampling tests, UNI-RNA predicted almost single strands for some RNA sequences that were different from the training distribution, and for some long and short RNA sequences, without any complementary base pairing, indicating that the model's generalization ability was very poor. Although the attention-map of RNA-MSM can capture some RNA secondary structure features during the pre-training stage, it faces the problem of excessive time complexity in homologous sequence search in practical applications, as well as the challenge of poor performance when there are no homologous sequences as input.

[0041] To address the aforementioned problems, the inventors of this application propose a method for pre-training RNA sequences. Specifically, the method includes:

[0042] 1) Data can be read by generating pre-trained language models based on RNA sequences;

[0043] In some examples of this application, the pre-trained model is selected from the BERT model. In some examples of this application, RNA sequence information is used as the readable data for the BERT model.

[0044] 2) The readable data is input into a pre-trained language model (such as the BERT model) to extract usable features from the RNA sequence. In some examples of this application, the usable features of the aforementioned RNA sequence include the encoding method of the RNA sequence, such as some high-dimensional vectors.

[0045] The initial bias of the pre-trained language model is determined based on a pre-constructed weight matrix, which is built based on the probability of covalent bond formation between bases. For example, the construction of the aforementioned weight matrix includes: iterating through each pair of bases i and j (i≠j) of the sequence; if the base pair corresponding to i and j is A and U, then the two-dimensional matrices (i,j) and (j,i) are set to 2; if the base pair corresponding to i and j is C and G, then the two-dimensional matrices (i,j) and (j,i) are set to 3; if the base pair corresponding to i and j is G and U, then the two-dimensional matrices (i,j) and (j,i) are set to α. In a specific example of this application, the value of α is 0.8, and the main diagonal of the weight matrix is ​​all set to 0. The advantage of this method lies in providing pre-trained language models with prior structural information about the given RNA. Based on the strength of different base pairings, a corresponding weight matrix is ​​calculated, which is then added to the structural attention matrix and the attention matrix calculated from the sequence. This results in a fully informative attention matrix, enabling the model to extract more information-rich general features. Figure 1 ).

[0046] In some examples of this application, the aforementioned BERT model employs a multi-head attention mechanism, wherein the initial bias of the multi-head attention mechanism is obtained by inputting the weight matrix into the neural network model.

[0047] In some examples of this application, the aforementioned weight matrix includes structural information of the RNA sequence. Validation has shown that incorporating structural information as a starting bias into the attention mechanism enhances the BERT model's modeling of RNA molecular structure information, thereby enabling the extraction of more comprehensive usable features.

[0048] In some examples of this application, the aforementioned trained BERT model was trained using the occlusion prediction method.

[0049] In some examples of this application, the aforementioned training further includes fine-tuning the BERT model. Fine-tuning methods include, but are not limited to, transfer learning.

[0050] The above method replaces the bias calculated in the multi-head self-attention function of the first block of the BERT model with the pair-wise position bias calculated from the one-dimensional sequence (RNA sequence), and continues to pass the complete attention matrix, which is added to the sequence attention matrix, to subsequent blocks. Starting from the second block of the BERT model, the bias of each block is replaced by the complete attention matrix passed from the previous block, and the last block outputs a feature of size L×L attention-map, thereby enhancing the modeling of RNA molecular structure information. The pre-trained model obtained based on this method can fully and accurately extract feature information of RNA sequences, and can accept single sequences as input, eliminating the step of searching for homologous sequences, and the feature extraction time is extremely short. Experimental verification shows that the pre-trained language model obtained based on the above method can effectively capture the secondary structure features of RNA, and also has a certain ability to capture special structures (such as pseudoknots).

[0051] In another aspect of this application, a method for predicting RNA sequences is proposed. This method includes: pre-training the RNA sequence based on the method of any of the above embodiments to obtain usable features; and inputting the usable features into a prediction model to predict the structure or function of the RNA sequence. In some examples of this application, this method enables accurate prediction of the structure or function of the RNA to be tested.

[0052] In some examples of this application, the prediction model includes a convolutional residual neural network and a fully connected network. Based on features obtained from a pre-trained language model, the convolutional residual neural network and the fully connected network are further utilized to achieve accurate prediction of the structure or function of the RNA to be tested.

[0053] In some examples of this application, the structure includes at least one of RNA secondary and tertiary structures. Predictive modeling of RNA structure reveals state-of-the-art (SOTA) performance across various downstream tasks, including RNA secondary structure prediction, significantly addressing the problem of insufficient RNA structural information modeling during pre-training leading to poor extraction of high-dimensional dense vectors.

[0054] In some examples of this application, the functionality includes at least one of the average ribosome load of the 5'UTR sequence and RNA-binding proteins.

[0055] In another aspect of this application, an apparatus for pre-training RNA sequences is proposed. (Reference) Figure 2The device includes a model-readable data acquisition unit S100 and an RNA sequence available feature acquisition unit S200, and the model-readable data acquisition unit S100 and the RNA sequence available feature acquisition unit S200 are connected together.

[0056] S100: Used to generate readable data for pre-trained language models based on RNA sequences;

[0057] S200: Used to input the readable data into a trained pre-trained language model to extract usable features of the RNA sequence, wherein the initial bias of the pre-trained language model is determined based on a pre-constructed weight matrix, which is constructed based on the probability of covalent bond binding between bases.

[0058] In some examples of this application, the device can be used to efficiently and accurately extract high-dimensional dense features of RNA sequences, which will help promote research and application in the field of RNA sequence analysis.

[0059] In another aspect of this application, a system for predicting RNA sequences is proposed. (Reference) Figure 3 The system includes an RNA sequence pre-training device S01 and a prediction device S02. The RNA sequence pre-training device S01 and the prediction device S02 are connected together.

[0060] S01: For pre-training the RNA sequence based on the apparatus described in the third aspect of this application in order to obtain usable features;

[0061] S02: Used to input the available features into the prediction model in order to predict the structure or function of the RNA sequence.

[0062] In some examples of this application, the above system can be used to accurately predict the structure (secondary structure, tertiary structure, and special structures such as pseudo-knot structures) or function of an input RNA sequence.

[0063] In another aspect of this application, a computer program product is provided, the computer program product comprising computer instructions that, when some or all of the computer instructions are executed on a computer, cause the methods of pre-training RNA sequences and / or predicting RNA sequences as described in this application to be performed.

[0064] In another aspect of this application, a computing device is provided, the device comprising: a processor and a memory; the memory for storing a computer program; and the processor for executing the computer program to implement the methods of pre-training RNA sequences and / or predicting RNA sequences as described in this application.

[0065] In another aspect, this application provides a computer-readable storage medium comprising computer instructions that, when executed by a computer, cause the computer to implement the methods of pre-training RNA sequences and / or predicting RNA sequences as described in this application.

[0066] The various embodiments of the computer program products, computing devices, and computer-readable storage media described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0067] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0068] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks) to enable the efficient creation of predictive models.

[0069] Embodiments of the present invention will now be described in more detail, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0070] It should be noted that the pre-trained language model obtained based on the method of this application is the same as the ERNIE-RNA model. The Pair-wise position bias is a weight matrix constructed based on the probability of covalent bond formation between bases in the RNA sequence, which is used to replace the bias matrix in the first block of the BERT model for calculating multi-head self attention.

[0071] Example 1: Evaluation of the secondary structure characteristics of ERNIE-RNA capture RNA

[0072] The purpose of this embodiment is to verify whether the attention-map extracted by the pre-trained model can capture the structural information of RNA. Since UNI-RNA does not have publicly available pre-training checkpoints, and considering that the core architectures of UNI-RNA and RNA-FM are identical, differing only in their training corpora, this embodiment only compares and analyzes RNA-FM with publicly available model parameters. The specific experimental procedure is as follows:

[0073] The test sample is the bpRNA-1m test set, which contains 102,318 sequences from 2,588 RNA families, making it one of the most comprehensive and largest RNA structure datasets currently available. Almost all RNA secondary structure prediction models are tested on this dataset to assess their predictive performance and generalization capabilities. The bpRNA-1m dataset was processed using the CD-HIT tool to remove redundant sequences with an 80% sequence similarity threshold, further filtering out sequences shorter than 500 characters. The preprocessed data left 13,419 sequences, which were randomly divided into three sets: 10,814 sequences for the training set (TR0), with a median sequence length of 104, an average length of 130, a maximum length of 498, and a minimum length of 9; 1,300 sequences for the validation set (VL0), with a median sequence length of 104, an average length of 129, a maximum length of 497, and a minimum length of 9; and 1,305 sequences for the test set (TS0), with a median sequence length of 107, an average length of 134, a maximum length of 499, and a minimum length of 9. For fair comparison, the dataset used in this embodiment is consistent with other works in this field.

[0074] The bpRNA-1m test set contains 1305 RNA sequences. Each sequence has an L×L sparse matrix representing the RNA secondary structure as a structural label. The sequences in the test set are input into ERNIE-RNA sequentially. Each sequence generates an L×L×156 attention-map feature (a total of 12 blocks plus a constructed pair-wise position bias; each block has 12 heads, and each head has an attention-map, for a total of 156). The size of the attention-map in each head is L×L, which represents the amount of attention the model believes should be focused on between any two bases. It is assumed that the 156 attention-maps can capture a certain amount of RNA secondary structure. Post-processing is then performed to make the prediction matrix more consistent with the actual biological situation. After post-processing, the attention-map becomes a sparse matrix of 0s and 1s. Then, these 156 prediction matrices are matched with the structural labels corresponding to the input sequences. The matching metric used is f1-score, which ranges from 0 to 1. The closer the value is to 1, the more accurate the model's prediction. Finally, the average of the f1-score values ​​corresponding to these 1305 sequences is taken to obtain the quantitative value of the ability of each attention-map to capture RNA secondary structures.

[0075] The results are as follows Figure 4As shown, it can be seen that no attention map corresponding to any head in RNA-FM can capture RNA secondary structure features. However, some heads in the ERNIE-RNA model obtained in this application can effectively capture RNA secondary structure features, and the F1-score is as high as 0.55. Table 1 shows some of the best-performing end-to-end models in the field of RNA secondary structure prediction. These models were fine-tuned based on the bpRNA-1m training set, and the optimal checkpoint was selected through a validation set and validated on the test set. It can be found that although ERNIE-RNA has not seen any data with a similar distribution in the bpRNA-1m dataset, its performance on the test set is better than most fine-tuned models, indicating that ERNIE-RNA effectively captures RNA secondary structure features during pre-training.

[0076] Table 1

[0077]

[0078]

[0079] Example 2: Evaluation of Specific Structural Features of ERNIE-RNA Capture

[0080] This embodiment used two typical RNA sequences containing pseudooknot structures as input into the ERNIE-RNA model, resulting in 156 features of size L×L. The results are as follows: Figure 5 As shown, some heads' attention maps captured the structural features of pseudooknot, further demonstrating that ERNIE-RNA captures RNA structural information during the pre-training phase.

[0081] Example 3: Evaluation of ERNIE-RNA Feature Extraction Time

[0082] Currently, the RNA-MSM pre-trained model has demonstrated its ability to capture RNA structural information during the pre-training stage on a small dataset. However, the input construction process requires searching for homologous sequences in the input sequence using the RNAcmap tool, a step that is extremely time-consuming. For a 60-base RNA sequence, extracting its features takes approximately 9 hours, which is unacceptable. Furthermore, the search for homologous sequences requires an initial secondary structure prediction tool, which also limits the learning ability of the pre-trained model to some extent.

[0083] In this embodiment, an RNA sequence of the same length (60 bases) is input into the ERNIE-RNA model, and feature extraction can be completed in seconds. Furthermore, the ERNIE-RNA model obtained in this application does not need to search for homologous sequences when extracting RNA sequence features; a single sequence can be used as input.

[0084] Example 4: ERNIE-RNA feature extraction performance is superior to similar models.

[0085] This embodiment verifies the superiority of ERNIE-RNA extraction features compared to RNA-FM and One-hot methods through clustering experiments. The specific steps are as follows:

[0086] Using 244 small RNA classes from Caleb Weinreb as experimental data, 10 replicate experiments were conducted to compare the feature extraction performance of ERNIE-RNA, RNA-FM, and One-hot models. Details of each experiment are as follows: Considering that the time complexity of attention-map feature dimensionality reduction is exponentially positively correlated with sequence length, small RNA classes with sequence lengths exceeding 200 were first filtered out. From the remaining classes, 10 RNA classes were randomly selected, and 1000 sequences were randomly extracted from each class (classes with fewer than 1000 sequences were all extracted), and labeled...

[0087] First, class labels are assigned to each sequence. Then, RNA-FM is used to extract the clstokenembedding of the RNA sequence, and ERNIE-RNA is used to extract the corresponding clstokenembedding and attention-map features. Simultaneously, one-hot features of the sequences are constructed. These features are then dimensionality-reduced to 2D using TSNE for visualization, and the class labels are annotated on the graph. To quantify the differences in information extracted from different features, the k-means algorithm is used to cluster the dimensionality-reduced features. Finally, three metrics are used to quantify the difference between the clustering prediction metrics and the true class: mutual information metric, RAND metric, and FM metric. For each metric, the closer it is to 1, the more accurate the clustering prediction, indirectly reflecting the semantic richness of the extracted features. Figure 6 As shown, the features extracted by RNA-FM performed the worst in matching after dimensionality reduction and clustering, even worse than one-hot features. The two features of ERNIE-RNA were the best in all three metrics.

[0088] Example 5: Application of the ERNIE-RNA model in downstream prediction tasks

[0089] This embodiment adopts a pre-training + fine-tuning paradigm, designing different downstream network frameworks after ERNIE-RNA for different types of downstream tasks. This is to verify the performance of features extracted based on the ERNIE-RNA model on tasks such as RNA secondary structure prediction, UTR-MRL prediction, and RNA-protein binding prediction.

[0090] First, we focus on the RNA secondary structure prediction task. RNA secondary structures are primarily formed through hydrogen bonds between bases, with the most common pairing types being AU and GC base pairings. There are also some atypical base pairings, such as GU base pairing. RNA single strands fold through these pairings to form stable structures. Predicting RNA secondary structures can help identify disease-related RNA structural changes, providing clues for the diagnosis and treatment of related diseases. It can also assist in designing better RNA components for constructing artificially synthesized biological systems. The dataset used in this example is the widely accepted bpRNA-1m dataset. Currently, the state-of-the-art (SOTA) method for deciphering RNA secondary structures is UNI-RNA, which involves connecting a residual convolutional neural network to a pre-trained language model of UNI-RNA and fine-tuning it on labeled data. After fine-tuning, this method achieves a macroaverage f1-score of 0.821 on the bpRNA-1m test set, but ERNIE-RNA achieves an f1-score of 0.873 after fine-tuning (Table 2), making it the best method currently available.

[0091] Table 2

[0092]

[0093] Next is the UTR-MRL prediction task. UTR refers to the 5' non-coding region of mRNA. Optimizing the 5'UTR can optimize MRL, which is the average number of ribosome attachments. Doing this task well is beneficial for optimizing and screening 5'UTR structural regions with high expression levels. This task aims to predict the corresponding MRL value given the 5'UTR of a given RNA sequence. Currently, the state-of-the-art method is UNI-RNA. Although ERNIE-RNA (86M) is more than 4 times smaller than UNI-RNA (400M), it still outperforms UNI-RNA after fine-tuning the same downstream network. Even when the downstream network framework is replaced with a very simple two-layer MLP, the results are almost the same as UNI-RNA, which fully demonstrates the effectiveness of ERNIE-RNA in extracting features (Table 3).

[0094] Table 3

[0095]

[0096] Finally, there's the RNA-protein binding prediction task. This task aims to predict the interaction between RNA molecules and proteins, i.e., the probability of RNA binding to specific proteins. Successfully performing this task is beneficial for a better understanding of biological processes such as gene expression, regulatory mechanisms, and signal transduction within cells. Currently, the state-of-the-art (SOTA) method is the Prismnet method proposed by Zhang Qiangfeng's research group. This method uses wet-scale experiments to measure icSHAPE data of RNA, then combines this with one-hot features input into a convolutional residual network, and fine-tunes the overall performance on the labeled data. Due to current data limitations, this task requires training a model for each protein. Prismnet achieves an average AUPRC of 0.833 for 17 proteins. After fine-tuning, ERNIE-RNA achieves an average AUPRC of 0.865 on this task, and even after integrating a simple MLP network, the performance reaches 0.853 (Table 4).

[0097] Table 4

[0098]

[0099]

[0100] In summary, this demonstrates that features extracted based on ERNIE-RNA can achieve state-of-the-art (SOTA) results after fine-tuning on downstream task data.

[0101] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0102] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for pre-training RNA sequences, characterized in that, include: Pre-trained language models generated from RNA sequences can read data; and The readable data is input into a pre-trained language model to extract usable features from the RNA sequence. The initial bias of the pre-trained language model is determined based on a pre-constructed weight matrix, which is constructed based on the probability of covalent bond formation between bases. The pre-trained language model is selected from the BERT model, which employs a multi-head attention mechanism. The initial bias of the multi-head attention mechanism is obtained by inputting the weight matrix into the neural network model.

2. The method according to claim 1, characterized in that, The trained BERT model was trained using the occlusion prediction method.

3. The method according to claim 1 or 2, characterized in that, The training further includes fine-tuning the BERT model.

4. The method according to claim 3, characterized in that, The fine-tuning includes transfer learning.

5. A method for predicting RNA sequences, characterized in that, include: The method according to any one of claims 1 to 4, wherein the RNA sequence is pre-trained to obtain usable features; The available features are input into the prediction model to predict the structure or function of the RNA sequence.

6. The method according to claim 5, characterized in that, The prediction model includes: a convolutional residual neural network and a fully connected network.

7. The method according to claim 5 or 6, characterized in that, The structure includes at least one of the RNA secondary and tertiary structures.

8. The method according to claim 5 or 6, characterized in that, The function includes the average ribosome load of the 5'UTR sequence and at least one of RNA-binding proteins.

9. An apparatus for pre-training RNA sequences, characterized in that, include: The model-readable data acquisition unit is used to generate pre-trained language model-readable data based on RNA sequences; and The RNA sequence usable feature acquisition unit is used to input the readable data into a trained pre-trained language model in order to extract usable features from the RNA sequence. The initial bias of the pre-trained language model is determined based on a pre-constructed weight matrix, which is constructed based on the probability of covalent bond formation between bases. The pre-trained language model is selected from the BERT model, which employs a multi-head attention mechanism. The initial bias of the multi-head attention mechanism is obtained by inputting the weight matrix into the neural network model.

10. A system for predicting RNA sequences, characterized in that, include: An RNA sequence pre-training device for pre-training the RNA sequence based on the device of claim 9 in order to obtain usable features; A prediction device is used to input the available features into a prediction model in order to predict the structure or function of the RNA sequence.

11. A computing device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The storage medium includes computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images