Predicting mRNA properties using large-scale language transformer models

Abstract

A method, computer system, and apparatus for predicting mRNA characteristics, comprising a computer program encoded on a computer storage medium. The system obtains data representing the codon sequence of an mRNA molecule, generates an input token vector by numerically encoding the codon sequence, and generates an embedding feature vector by processing the input token vector using an embedding machine learning model having a first set of model parameters.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 516,226 filed on 28 July 2023, U.S. Provisional Patent Application No. 63 / 648,338 filed on 16 May 2024, and European Patent Application No. 24305758.5 ​​filed on 16 May 2024, all of which disclosures are incorporated herein by reference in their entirety.

[0002] This specification generally relates to predicting the properties of mRNA molecules using machine learning models, such as large-scale language transformer models. [Background technology]

[0003] mRNA, or messenger RNA, is a type of RNA molecule that plays a crucial role in gene expression and protein synthesis. The primary function of mRNA is to carry genetic instructions from DNA to ribosomes, where proteins are synthesized. mRNA is typically single-stranded and can be hundreds to thousands of nucleotides long. A full-length mRNA sequence includes the 5' untranslated region (UTR), the coding sequence (CDS), and the 3'UTR. The 5'UTR is the non-coding sequence at the beginning of the mRNA molecule. The 3'UTR is the non-coding sequence at the end of the mRNA molecule. The CDS consists of a codon sequence, where each codon consists of three nucleotides that specify a particular amino acid or start or stop signal during protein synthesis. The codon sequence determines the order in which amino acids are combined during translation. While the 5' and 3'UTR are not translated, they can play important roles in mRNA stability, localization, and translational regulation.

[0004] A machine learning model is a computational model that learns patterns and relationships in data and uses that knowledge to represent the data in different spaces and make predictions or decisions about new data. A neural network is a machine learning model that uses one or more nonlinear units of layers to predict an output for an incoming input. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from an incoming input according to the current values ​​of its respective set of parameters. [Overview of the project] [Problems that the invention aims to solve]

[0005] This disclosure describes methods, computer systems, and apparatus for predicting the properties of mRNA molecules, including computer programs encoded on a computer storage medium. [Means for solving the problem]

[0006] In one embodiment, the disclosure provides a predictive method for predicting one or more properties of an mRNA molecule. The method may be carried out by a system comprising one or more computers. Generally, the system generates token representations by numerically encoding the codon sequences of the mRNA sequence, generates embedding features of the mRNA sequence using an embedding machine learning model (such as a large-scale language model) using unsupervised learning, and further uses supervised learning to predict mRNA properties for downstream tasks. By pre-training the large-scale language model, the system enables the model to generate high-performance embedding representations that capture meaningful representations, codon interactions, and sequence-level patterns essential for understanding and predicting various mRNA properties in downstream tasks. Downstream tasks may include, for example, (1) predicting mRNA expression, (2) analyzing mRNA stability, and (3) predicting mRNA degradation. A two-step process, including pre-training and downstream task-based fine-tuning, makes it possible to generate high-quality predictions of mRNA properties based on limited labeled data.

[0007] To execute the prediction method, the system obtains data representing the codon sequences of mRNA molecules, generates an input token vector by numerically encoding the codon sequences, and generates an embedded feature vector by processing the input token vector using an embedded machine learning model having a first set of model parameters. The first set of model parameters has been updated using a first training process of a first machine learning model that includes the embedded machine learning model. The first training process is executed based on a dataset specifying known codon sequences of mRNA molecules, and the first machine learning model is configured to execute one or more pre-training tasks. The system uses a property prediction machine learning model to process the embedded feature vector to generate an output predicting one or more properties of the mRNA molecule. The property prediction machine learning model has a second set of model parameters, and the second set of model parameters has been updated using a second training process of a second machine learning model that includes the property prediction machine learning model based on a plurality of training examples. Each of the respective training examples includes (i) a respective training input specifying the representation of a respective mRNA molecule and (ii) a respective label specifying one or more properties of a respective mRNA molecule.

[0008] In some embodiments of the prediction method, the pre-training task includes a masked language model (MLM) learning task for predicting masked codons within known mRNA molecules. In these cases, the loss function may include an MLM loss function defined as:

Number

[0009] In some cases, the pre-training task includes a homologous sequence prediction (HSP) task to predict whether two input mRNA sequences belong to the same homologous class of organisms. In these cases, the loss function is:

number

[0010] In some cases, the loss function combines MLM loss and HSP loss.

[0011] In some embodiments of the prediction method, one or more properties of an mRNA molecule include the expression level of the mRNA molecule in a particular type of cell or tissue. For example, the mRNA molecule may be a component of a vaccine and encoded to express one or more antigen proteins of a target pathogen, and the predicted properties of the mRNA molecule may characterize the expression level of the antigen proteins of the target pathogen in a particular type of cell or tissue.

[0012] In some embodiments of the prediction method, one or more properties of the mRNA molecule include stability under one or more environmental conditions.

[0013] In some embodiments of the prediction method, one or more properties of the mRNA molecule include a switching factor for the mRNA molecule in a particular type of cell or tissue.

[0014] In some embodiments of the prediction method, one or more properties of an mRNA molecule include the degradation rate of the mRNA molecule under one or more environmental conditions. For example, the mRNA molecule may be a component of a SARS-CoV-2 vaccine, and a property prediction machine learning model may predict the degradation rate of the mRNA molecule under physiological conditions.

[0015] In some embodiments of the prediction method, generating an input token vector involves mapping each codon in a codon sequence to its respective numerical value and concatenating the numerical values ​​to generate the token vector.

[0016] In some embodiments of the prediction method, the first machine learning model includes a Large-Scale Language Model (LLM).

[0017] In some embodiments of the prediction method, the first machine learning model includes a bidirectional transformer.

[0018] In some embodiments of the prediction method, the characteristic prediction machine learning model may include one or more of the following: neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, or ridge regression models.

[0019] In some embodiments of the prediction method, the characteristic prediction machine learning model includes a convolutional neural network (CNN).

[0020] In another aspect, the disclosure provides another predictive method for predicting one or more properties of an mRNA molecule. The method may be carried out by a system comprising one or more computers. The system acquires data representing an mRNA molecule, which comprises (i) a 5' untranslated region (UTR), (ii) a coding sequence (CDS), and (iii) a 3'UTR. The system generates a first input token vector by numerically encoding the nucleotide sequence of the 5'UTR of the mRNA molecule, a second input token vector by numerically encoding the codon sequence of the CDS of the mRNA molecule, and a third input token vector by numerically encoding the nucleotide sequence of the 3'UTR of the mRNA molecule.

[0021] This system generates a first embedding feature vector by processing a first input token vector using a first embedding machine learning model, a second embedding feature vector by processing a second input token vector using a second embedding machine learning model, and a third embedding feature vector by processing a third input token vector using a third embedding machine learning model. The first, second, and third embedding machine learning models are trained on a training mRNA sequence set using a first training process. In some cases, the first, second, and third embedding machine learning models are trained separately in the first training process. In some cases, the first, second, and third embedding machine learning models are trained simultaneously in the first training process.

[0022] This system generates a combined embedding representation by combining a first embedding feature vector, a second embedding feature vector, and a third embedding feature vector. The system processes the combined embedding representation using a feature prediction machine learning model to generate an output that predicts one or more features of an mRNA molecule. The feature prediction machine learning model is trained on a labeled training example set using a second training process.

[0023] In some embodiments of the prediction method, to generate a first input token vector, the system generates the first input token vector by mapping each nucleotide in the 5'UTR nucleotide sequence to its respective numerical value and concatenating the numerical values. To generate a second input token vector, the system generates the second input token vector by mapping each codon in the CDS codon sequence to its respective numerical value and concatenating the numerical values. To generate a third input token vector, the system generates the third input token vector by mapping each nucleotide in the 3'UTR nucleotide sequence to its respective numerical value and concatenating the numerical values.

[0024] In some embodiments of the prediction method, in order to generate a combined embedding representation, the system performs a first pooling operation on a first embedding feature vector to generate a first embedding representation, a second pooling operation on a second embedding feature vector to generate a second embedding representation, a third pooling operation on a third embedding feature vector to generate a third embedding representation, and then concatenates the first, second, and third embedding representations to generate a combined embedding representation.

[0025] In some embodiments of the prediction method, the first, second, and third pooling operations are each average pooling operations.

[0026] In some embodiments of the prediction method, the first training process includes starting parameter values ​​for a first machine learning model, including first, second, and third embedded machine learning models, and training the first machine learning model by minimizing a pre-training loss function that includes one or more pre-training losses defined for one or more pre-training tasks. In some cases, the one or more pre-training tasks include a masked language model (MLM) training task for predicting one or more masked codons or nucleotides in a known mRNA molecule. In some cases, the one or more pre-training losses are

number

number

[0027] In some embodiments of the prediction method, the second training process includes starting the values of the parameters of the second machine learning model including the characteristic prediction machine learning model, and minimizing a downstream loss function including one or more prediction losses defined for one or more characteristic prediction tasks to train the second machine learning model.

[0028] In some embodiments of the prediction method, the pre-training loss function or the downstream loss function further includes a contrast loss aimed at maximizing the similarity between the embedding representations of different regions within the same mRNA sequence while minimizing the similarity between the embedding representations of different regions from different mRNA sequences.

[0029] In some cases, the contrast loss includes a first contrast loss aimed at maximizing the similarity between the embedding representations of the 5'UTR and CDS within the same mRNA sequence while minimizing the similarity between the embedding representations of the 5'UTR and CDS from two different mRNA sequences.

[0030] In some cases, the first contrast loss is

Number

[0031] In some cases, the control loss includes a second control loss aimed at maximizing the similarity between 3'UTR and CDS embedding representations within the same mRNA sequence while minimizing the similarity between 3'UTR and CDS embedding representations from two different mRNA sequences.

[0032] In some cases, the control loss is calculated as a composite control loss, which is a combination of the first control loss and the second control loss. For example, the composite control loss may be calculated as the average of the first and second control losses.

[0033] In some embodiments of the prediction method, one or more properties of the mRNA molecule include the expression level of the mRNA molecule in a particular type of cell or tissue.

[0034] In some cases, mRNA molecules are components of vaccines, encoded to express one or more antigenic proteins of a target pathogen, and the predicted properties of the mRNA molecule characterize the expression levels of the target pathogen's antigenic proteins in a particular type of cell or tissue.

[0035] In some embodiments of the prediction method, one or more properties of the mRNA molecule include stability under one or more environmental conditions.

[0036] In some embodiments of the prediction method, one or more properties of the mRNA molecule include a switching factor for the mRNA molecule in a particular type of cell or tissue.

[0037] In some embodiments of the prediction method, one or more properties of the mRNA molecule include the degradation rate of the mRNA molecule under one or more environmental conditions.

[0038] In some cases, mRNA molecules are components of the SARS-CoV-2 vaccine, and characteristic predictive machine learning models are configured to predict the degradation rate of mRNA molecules under physiological conditions.

[0039] In some embodiments of the prediction method, each of the first, second, and third embedded machine learning models includes its own large language model (LLM).

[0040] In some embodiments of the prediction method, each of the first, second, and third embedded machine learning models includes its respective bidirectional transformer.

[0041] In some embodiments of the prediction method, the characteristic prediction machine learning model includes one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

[0042] In some embodiments of the prediction method, the characteristic prediction machine learning model includes a convolutional neural network (CNN).

[0043] In another embodiment, the disclosure provides a design method for determining the optimal codon sequence of mRNA for performing a particular task. The design method may be carried out by a system comprising one or more computers. The system predicts the properties of each candidate mRNA molecule using one of the prediction methods described above and selects an mRNA molecule from the set of candidate mRNA molecules based on the predicted properties. In some cases, the design method further includes physically generating the selected mRNA molecule.

[0044] In some embodiments of this design method, downstream tasks include one or more of the following: maximizing mRNA expression levels in a particular type of cell or tissue, or maximizing mRNA stability in a particular environment.

[0045] In another aspect, the present disclosure provides mRNA molecules selected using the present design method.

[0046] In another aspect, the disclosure provides a reinforcement learning (RL) method for training an RL model to determine the optimal codon sequence of mRNA for performing a particular task. The RL method may be performed by a system comprising one or more computers. The system maintains data representing a set of candidate sequences of mRNA molecules and uses a reinforcement learning (RL) model to process one or more of the candidate sequences to generate one or more new sequences. The RL model is trained using a reward signal containing mRNA molecular properties predicted using one of the prediction methods described above. The system selects the optimal sequence from the new sequences. In some cases, the RL method further includes physically generating an mRNA molecule having the optimal sequence selected from the new sequences.

[0047] In another aspect, the disclosure provides an mRNA molecule having an optimal sequence selected from novel sequences generated using the RL method.

[0048] In another aspect, the disclosure provides a training method for training a predictive model for predicting the properties of mRNA molecules. The training method may be performed by a system comprising one or more computers. The predictive model includes (i) an embedding machine learning model configured to generate embedding feature vectors for model inputs representing the codon sequences of mRNA molecules, and (ii) a property prediction machine learning model configured to process the embedding feature vectors to produce outputs specifying one or more properties of mRNA molecules. The system acquires a first dataset containing a set of sequence representations of mRNA molecules and uses the first dataset to train a first machine learning model, including the embedding machine learning model, on one or more pre-training tasks. The system acquires a second dataset containing multiple training examples. Each of each training example includes (i) a respective training input specifying a representation of each mRNA molecule, and (ii) a respective label characterizing one or more properties of each mRNA molecule. The system performs supervised training of a second machine learning model, including the property prediction machine learning model, on the second dataset.

[0049] In some embodiments of this training method, the pre-training task includes a masked language model (MLM) learning task to predict masked codons within known mRNA molecules. In these cases, the loss function is:

number

[0050] In some embodiments of this training method, the pre-training task includes a homologous sequence prediction (HSP) task to predict whether two input mRNA sequences belong to the same homologous class of organisms. In these cases, the loss function is:

[0051]

number

[0052] In some embodiments of this training method, the loss function combines MLM loss and HSP loss.

[0053] In some embodiments of this training method, one or more properties of the mRNA molecule include one or more of the following: the expression level of the mRNA molecule in a particular type of cell or tissue, stability under one or more environmental conditions, or a switching factor for the mRNA molecule in a particular type of cell or tissue.

[0054] In some embodiments of this training method, generating an input token vector involves mapping each codon in a codon sequence to its corresponding numerical value and concatenating the numerical values ​​to generate the token vector.

[0055] In some embodiments of this training method, the first machine learning model includes a large-scale language model (LLM). For example, the first machine learning model may include a bidirectional transformer.

[0056] In some embodiments of this training method, the feature prediction machine learning model includes one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

[0057] In another aspect, the disclosure provides a training method for training a predictive model for predicting the properties of an mRNA molecule. The training method may be carried out by a system comprising one or more computers. The mRNA molecule comprises (i) a 5' untranslated region (UTR), (ii) a coding sequence (CDS), and (iii) a 3'UTR. The predictive model includes: (i) a first embedding machine learning model configured to process a first input token vector representing the nucleotide sequence of the 5'UTR of an mRNA molecule to generate a first embedding feature vector; (ii) a second embedding machine learning model configured to process a second input token vector representing the codon sequence of an mRNA molecule to generate a second embedding feature vector; (iii) a third embedding machine learning model configured to process a third input token vector representing the nucleotide sequence of the 3'UTR of an mRNA molecule to generate a third embedding feature vector; and (iv) a property predictive machine learning model configured to process a combined embedding representation generated by combining the first, second, and third embedding feature vectors to generate an output specifying one or more properties of an mRNA molecule.

[0058] The system obtains a first dataset containing a set of sequence representations of mRNA molecules. The system performs a first training process to train the first, second, and third embedded machine learning models on one or more pre-training tasks using the first dataset. The system obtains a second dataset containing multiple training examples. Each of the training examples includes (i) a training input specifying a representation of each mRNA molecule, and (ii) a label characterizing one or more properties of each mRNA molecule. The system performs a second training process to train the property prediction machine learning models on one or more property prediction tasks using the second dataset. In some cases, the first, second, and third embedded machine learning models are trained separately in the first training process. In some cases, the first, second, and third embedded machine learning models are trained simultaneously in the first training process.

[0059] In some embodiments of this training method, the first training process includes starting parameter values ​​for a first machine learning model, which includes first, second, and third embedded machine learning models, and training the first machine learning model by minimizing a pre-training loss function that includes one or more pre-training losses defined for one or more pre-training tasks.

[0060] In some embodiments of this training method, one or more pre-training tasks include a masked language model (MLM) learning task for predicting masked codons within known mRNA molecules.

[0061] In some embodiments of this training method, the pre-training loss function is:

number

[0062] In some embodiments of this training method, one or more pre-training tasks include homologous sequence prediction (HSP) tasks for predicting whether two input mRNA sequences belong to the same homologous class of organisms.

[0063] In some cases, the pre-training loss function is

number

[0064] In some embodiments of this training method, the pre-training loss function combines the MLM loss and the HSP loss.

[0065] In some embodiments of this training method, the second training process includes starting parameter values ​​for a second machine learning model, which includes a feature prediction machine learning model, and training the second machine learning model by minimizing a downstream loss function that includes one or more prediction losses defined for one or more feature prediction tasks.

[0066] In some cases, the pre-training loss function or downstream loss function further includes a control loss aimed at maximizing the similarity between embedding representations of different regions within the same mRNA sequence while minimizing the similarity between embedding representations of different regions from different mRNA sequences.

[0067] In some cases, the control loss includes a first control loss aimed at maximizing the similarity between 5'UTR and CDS embedding representations within the same mRNA sequence while minimizing the similarity between 5'UTR and CDS embedding representations from two different mRNA sequences.

[0068] In some cases, the first counter loss is

number

[0069] In some cases, the control loss includes a second control loss aimed at maximizing the similarity between 3'UTR and CDS embedding representations within the same mRNA sequence while minimizing the similarity between 3'UTR and CDS embedding representations from two different mRNA sequences.

[0070] In some cases, the control loss is calculated as a composite control loss, which is a combination of the first control loss and the second control loss.

[0071] In some cases, the composite control loss is calculated as the average of the first control loss and the second control loss.

[0072] In some cases, the control loss is included in the pre-training loss function.

[0073] In some cases, the counter loss is included in the downstream loss function.

[0074] In some embodiments of this training method, one or more properties of the mRNA molecule include one or more of the following: the expression level of the mRNA molecule in a particular type of cell or tissue, stability under one or more environmental conditions, or a switching factor for the mRNA molecule in a particular type of cell or tissue.

[0075] In some embodiments of this training method, the first input token vector is generated by mapping each nucleotide in the 5'UTR nucleotide sequence to its respective numerical value and concatenating the numerical values; the second input token vector is generated by mapping each codon in the CDS codon sequence to its respective numerical value and concatenating the numerical values; and the third input token vector is generated by mapping each nucleotide in the 3'UTR nucleotide sequence to its respective numerical value and concatenating the numerical values.

[0076] In some embodiments of this training method, each of the first, second, and third embedded machine learning models includes its own large-scale language model (LLM).

[0077] In some embodiments of this training method, each of the first, second, and third embedded machine learning models includes its own bidirectional transformer.

[0078] In some embodiments of this training method, the feature prediction machine learning model includes one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

[0079] In some cases, the second machine learning model further includes the first, second, and third embedded machine learning models, and starting the parameter values ​​of the second machine learning model includes starting the parameters of the first, second, and third embedded machine learning models to the values ​​obtained through the first training process, and the second training process includes (i) inserting one or more layers into each of the first, second, and third embedded machine learning models, and (2) updating the parameter values ​​of the inserted layers while fixing the parameter values ​​of the other layers of the first, second, and third embedded machine learning models.

[0080] The disclosure also provides a system comprising one or more computers and one or more storage devices that, when executed by one or more computers, store instructions causing one or more computers to perform the prediction method, training method, design method, or RL method described above.

[0081] The disclosure also provides one or more computer storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the prediction method, training method, design method, or RL method described above.

[0082] The subject matter described herein may be implemented in specific embodiments to achieve one or more advantages.

[0083] mRNA research has become increasingly important in various fields, including molecular biology, genetics, and therapeutics. Understanding and predicting mRNA properties such as expression levels, stability, structure, and switching factors in specific tissues or cells provides insights into gene expression, protein synthesis, and the development of mRNA-based therapies such as mRNA vaccines and gene therapies.

[0084] Existing techniques for predicting mRNA properties typically do not take into account codon position and order information, as well as long-range interactions between codons. These limitations result in reduced predictive efficiency and accuracy.

[0085] This disclosure presents techniques utilizing deep learning and transformer models to computationally predict mRNA properties based on mRNA codon sequences. Compared to models that process RNA base sequence representations, mRNA codon sequence representations provide essential functional insights into mRNA molecules, including the genetic code, translation process, start and stop signals, reading frame, and resulting amino acid sequence. By processing codon sequence representations using language models, meaningful embedding features are provided, enabling a deeper understanding of the genetic information and evolutionary relationships associated with mRNA molecules.

[0086] In particular, a specific embodiment of the technique described uses a pre-training process based on a large number of known mRNA sequences to train a language model to perform two tasks: (i) a masked language model (MLM) learning task and (ii) a homologous sequence prediction (HSP) task.

[0087] During MLM learning task training, the language model is optimized to predict masked codons within known mRNA molecules by utilizing the remaining codons as context and considering the interactions between masked and unmasked codons. As a result, MLM can acquire knowledge about codon representation, codon-codon interactions, and the relationships between codons and mRNA sequences.

[0088] The HSP task utilizes the observation that mRNAs from organisms sharing a common evolutionary origin exhibit greater similarity in their codon sequences. Therefore, training for the HSP task provides additional information and patterns regarding the sequence-level representation of mRNA.

[0089] By pre-training the language model to perform both MLM and HSP, this pre-training process enables the model to generate high-performance embedding representations that capture meaningful expressions, codon interactions, and sequence-level patterns essential for understanding and predicting various mRNA properties in downstream tasks. This two-stage process, including pre-training and downstream task-based fine-tuning, makes it possible to generate high-quality predictions of mRNA properties based on limited labeled data.

[0090] Based on predicted mRNA characteristics, the described system or another related system can determine the optimal mRNA molecule for a particular application. For example, such a system can generate an output indicating whether a particular mRNA is suitable for a particular application, or an output specifying the optimal mRNA molecule for a particular application. The system can send the output to a manufacturing device that can operate to carry out instructions for producing mRNA. Overall, by training a high-performance predictive model based on limited experimental data and using the trained model to predict mRNA characteristics, the described technique can significantly improve the effectiveness and efficiency of mRNA engineering.

[0091] In addition, in some embodiments of the described technique, the system uses two additional language models to generate embedding representations of the nucleotide sequences of the 3' and 5' UTRs, and combines the generated embedding representations for the CDS and the 3' and 5' UTRs as a composite embedding representation during pre-training. This approach allows the model to capture the interactions and dependencies between coding and non-coding regions. Since the 5' and 3' UTRs can play important roles in mRNA stability, localization, and translational control, combining their embedding representations with those generated from the codon sequences of the CDS allows the model to provide more accurate predictions of mRNA properties.

[0092] Furthermore, in some cases, the system incorporates control loss into the pre-training of embedding models or downstream training of characteristic prediction models, thereby improving the model's ability to distinguish between similar and dissimilar mRNA sequences. Control loss works by maximizing the distance between embedding representations of different regions from different mRNA sequences while minimizing the distance between embedding representations generated for different regions within the same mRNA sequence. Control loss helps the model better capture intrinsic characteristics and interactions within individual mRNA sequences, improving its ability to make accurate predictions about mRNA characteristics.

[0093] The features described above enable the model to learn and generalize more effectively from data, resulting in higher accuracy and robustness when predicting mRNA stability, expression levels, and other properties. This improved capability is particularly useful in applications requiring mRNA design, such as the development of mRNA vaccines and gene therapies, where understanding specific features of mRNA sequences is crucial for efficacy and safety. In addition, improved model performance allows for more efficient use of computational resources, including reduced training time and costs. For example, enhanced model learning capabilities enable high performance with fewer adjustments during training. As a result, the overall computational load is reduced, leading to lower energy consumption and shorter training periods.

[0094] Details of one or more embodiments of the subject matter described herein are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter should be apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0095] [Figure 1] This example workflow illustrates an application scenario using a characteristic prediction system to predict the properties of mRNA sequences. [Figure 2A] An example of an mRNA characteristic prediction system is shown. [Figure 2B] This illustrates an exemplary process for generating an embedded representation of the entire mRNA sequence. [Figure 3A] This flowchart illustrates an exemplary process for predicting the properties of an mRNA sequence. [Figure 3B] This flowchart shows another exemplary process 300B for predicting mRNA characteristics. [Figure 4A] This flowchart illustrates an exemplary process for training a predictive model to predict mRNA properties. [Figure 4B]This flowchart illustrates an exemplary process for training a predictive model to predict mRNA characteristic scales. [Figure 5] This is a block diagram of an exemplary computer system. [Figure 6] This paper compares the performance of different mRNA characteristic prediction models for various prediction tasks. [Figure 7] This section presents another performance comparison of different mRNA characteristic prediction models for various prediction tasks. [Modes for carrying out the invention]

[0096] Similar reference numbers and designations in various drawings refer to the same elements.

[0097] Predicting the characteristics of mRNA sequences is a crucial aspect of developing mRNA-based therapies. For example, predicting mRNA stability, expression levels, and potential immunogenicity is important for designing effective mRNA formulations, such as mRNA vaccines.

[0098] The expression level of mRNA vaccines directly affects their potency and effectiveness. Higher expression levels of the antigen protein encoded within the mRNA result in a stronger immune response. This also means that lower doses of the vaccine may be effective, potentially leading to simpler and more cost-effective manufacturing. Furthermore, higher expression levels help overcome immune tolerance, i.e., the condition where the immune system becomes less responsive to low levels of antigen over time. Additionally, it helps maintain the immune response for a longer period, providing more sustained protection. The mRNA sequence is also important for its stability and function.

[0099] Conventional methods for identifying the optimal mRNA sequence involve selecting codons based on their frequency in the target organism. However, relying solely on the most frequent codons has limitations, and the best sequence may not always be obtained.

[0100] Next-generation sequencing techniques have enabled researchers to generate thousands of mRNA sequences and measure their corresponding protein expression levels. However, the redundancy of the genetic code (64 codons for 21 amino acids) creates a vast search space that grows exponentially with sequence length. This makes it impossible to experimentally test all possible mRNA candidates.

[0101] Furthermore, in addition to the coding sequence (CDS), the full-length mRNA sequence also includes a 5' untranslated region (UTR) at the beginning of the mRNA molecule and a 3' UTR at the end. Although the 5' and 3' UTRs are not translated, they have been shown to play important roles in mRNA stability, localization, and translational regulation. The effects of the 5' and 3' UTRs, as well as their interactions with the CDS, have not been considered by conventional techniques.

[0102] Therefore, sophisticated computational models are needed to guide the search for the optimal sequence.

[0103] Figure 1 shows an exemplary workflow in an application scenario using the characteristic prediction system 200, which predicts the properties of mRNA sequences. In particular, Figure 1 shows the selection or design of mRNA sequences for a specific clinical application using the characteristic prediction system 200.

[0104] mRNA library 102 defines a candidate mRNA set in which each mRNA may be represented by its respective codon sequence. In these embodiments, the codon sequence represents the coding sequence (CDS) of the candidate mRNA. In some other embodiments, the representation of the candidate mRNA may include additional sequence information from the 3' and 5' untranslated regions (UTR). When only the CDS is considered, the influence of the UTR on mRNA properties is not considered. For brevity, in this specification, the term “mRNA codon sequence” refers to the codon sequence of the CDS of an mRNA molecule, and the term “mRNA sequence” may refer to either the entire mRNA sequence (including each UTR and CDS) or only the CDS codon sequence, depending on the specific context. mRNA library 102 may contain any appropriate number of mRNAs, e.g., 100, 1,000, 10,000, or 1,000,000 mRNAs.

[0105] In some embodiments, the characteristic prediction system 200 is configured to process mRNA codon sequences (e.g., from mRNA library 102) to generate mRNA characteristic scales 104 that characterize the predicted properties of the corresponding mRNA. In some other embodiments, the characteristic prediction system 200 is configured to process full-length mRNA sequences (e.g., from mRNA library 102) to generate mRNA characteristic scales 104 that characterize the predicted properties of the corresponding mRNA. The characteristic scales 104 may include, for example, the expression level of one or more proteins (e.g., antigen proteins of a target pathogen) of the mRNA in a particular type of cell or tissue, stability under one or more environmental conditions, switching factors of the mRNA molecule in a particular type of cell or tissue, or the degradation rate of the mRNA molecule under one or more environmental conditions.

[0106] The characteristic prediction system 200 can screen the mRNA library 102 to identify mRNAs that possess desirable characteristic scales. More specifically, the characteristic prediction system 200 can predict each characteristic scale 104 for each mRNA in the mRNA library 102. Based at least partially on the predicted characteristic scales 104, the characteristic prediction system 200 can designate an appropriate subset of mRNAs in the mRNA library 102 as "target" mRNA 106.

[0107] The characteristic prediction system 200 can select an appropriate subset of mRNAs in the mRNA library that should be designated as target mRNAs in any variety of ways. For example, the characteristic prediction system 200 can designate any mRNA having a characteristic scale 104 that satisfies a predefined threshold as a target mRNA. Alternatively, the characteristic prediction system 200 can designate a predefined number of mRNAs having the highest characteristic scale 104 as target mRNAs.

[0108] In some cases, the characteristic prediction system 200 can be used in conjunction with the reinforcement learning system 114 to determine the optimal codon sequence of mRNA. In some cases, the characteristic prediction system 200 can be used in conjunction with the reinforcement learning system 114 to determine the optimal entire sequence of mRNA. The reinforcement learning system 114 uses a reinforcement learning (RL) model to process one or more candidate sequences and generate one or more new sequences. The RL model can be trained using a reward signal that includes mRNA molecular characteristics predicted using the prediction system 200. The optimal sequence can be selected from the new sequences.

[0109] The generated or selected target mRNA can be manufactured using any suitable technique 108 and used in any variety of applications. For example, the manufactured mRNA can be applied to a patient 112 as a treatment 110, such as an mRNA vaccine for infectious diseases or cancer, or a gene transfer agent for treating hereditary disorders, to achieve a therapeutic effect on the patient.

[0110] Figure 2A shows an example of a characteristic prediction system 200. System 200 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below may be implemented.

[0111] Generally, the characteristic prediction system 200 processes input data 210 specifying the codon sequence of mRNA and predicts characteristic measures 240 in some embodiments. In some other embodiments, the input data 210 specifies the entire mRNA sequence. The characteristic prediction system 200 includes a sequence tokenizer 225 that generates an input token vector by numerically encoding the codon sequence, an embedding machine learning model 220 that processes the input token vector to generate an embedding feature vector 225, and a predictive machine learning model 230 that processes the embedding feature vector 225 to generate an output that predicts one or more characteristics 240 of the mRNA molecule. In some cases, the sequence tokenizer 225 generates the input token vector by numerically encoding the entire mRNA sequence. The predicted characteristics may include one or more of the following: the expression level of the mRNA molecule in a particular type of cell or tissue, stability under one or more environmental conditions, and / or switching factors of the mRNA molecule in a particular type of cell or tissue.

[0112] In some embodiments, the input mRNA sequence 210 is represented by the CDS codon sequence of the mRNA. The sequence tokenizer 225 can map each codon in the list of codons (e.g., a list of all 64 codons) to its corresponding numerical value. After mapping each codon in the input codon sequence 210 to its corresponding numerical value, the tokenizer 225 can generate a token vector of the input codon sequence 210 by concatenating the mapped numerical values.

[0113] In some other embodiments, the input mRNA sequence 210 is the whole mRNA sequence including the 5' and 3' UTRs and CDS. The tokenization process in these cases will be described in more detail with reference to Figure 2B.

[0114] The embedded machine learning model 220 is configured to process an input token vector to generate an embedded feature vector 225. The embedded feature vector 225 is a numerical representation that captures essential information required for one or more tasks. In particular, the embedded feature vector 225 may be a high-dimensional vector of real numbers that captures features of mRNA codon sequences. In some cases, as will be further illustrated with reference to Figure 2B, the embedded machine learning model 220 may include multiple embedding models (e.g., 220a, 220b, and 220c, as shown in Figure 2B), and the embedded representation 225 is a combined embedding representation computed by combining the embedding representations generated by the multiple embedding models.

[0115] In some embodiments, the embedded machine learning model 220 is a neural network. The embedded neural network can employ any suitable architecture. In particular, the embedded neural network 220 may include at least a portion (e.g., the embedding portion) of a state-of-the-art large-scale language model (LLM).

[0116] In some embodiments, the embedded neural network 220 may include a bidirectional transformer, such as a Bidirectional Encoder Representation by Transformers (BERT) model. An example of a BERT embodiment is described in “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Devlin et al., arXiv:1810.04805, 2018.

[0117] In an exemplary example of a particular embodiment, the embedded neural network 220 includes a stack of bidirectional transformer encoders (e.g., 12 layers). Each transformer layer processes its input with a self-attention headset (e.g., a set of 12 self-attention heads) and outputs a representation for each position. At each layer, the multi-head self-attention mechanism captures contextual information of the input sequence by considering all other codons in the sequence. A key advantage of the self-attention mechanism is the connections learned between all pairs of positions in the input sequence using parallel computation, which allows embodiments to model not only short-range interactions but also long-range interactions that affect translation efficiency and stability. In some embodiments, after the self-attention layers, the embedded neural network 220 further includes one or more feedforward layers that apply nonlinear transformations to the hidden representations output from the self-attention layers. Residual connections may be employed between the multi-head attention layers and the feedforward layers, respectively. After processing the input array with a stack of transformer encoders, the embedded neural network 220 generates a codon representation aligned with the final context, which can then be used to generate a classification layer for generating probability distributions across the vocabulary during pre-training.

[0118] The predictive machine learning model 230 is configured to process an input containing an embedded feature vector 225 and generate an output 240 that predicts the mRNA characteristic scale 240. The predictive machine learning model 230 can be any suitable model and can be implemented using one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

[0119] In some embodiments, system 200 or another system includes a self-supervised learning engine 250 configured to update model parameters 222 of embedded machine learning models using self-supervised learning based on an mRNA codon sequence representation set 255. In some cases, the self-supervised learning engine 250 is configured to update model parameters 222 of multiple embedded machine learning models 220 using self-supervised learning based on a full-length mRNA sequence representation set 255.

[0120] The goal of self-supervised learning is to learn meaningful embedding representations of mRNA codon sequences without requiring the use of labeled data. Specifically, the self-supervised learning engine 250 learns embedding representations using unlabeled mRNA codon sequence data, i.e., data that specifies or represents each of a set of mRNA codon sequences without mRNA characteristic labels. In other words, the self-supervised learning engine 250 can leverage a large number of known mRNA codon sequences to learn embedding representations without the need to acquire a large amount of experimental benchmark data on known mRNA characteristics. Generally, dataset 225 contains a large number of mRNA codon sequences, e.g., hundreds of thousands, millions, tens of millions, or hundreds of millions of mRNA sequences.

[0121] In some cases, the goal of self-supervised learning is to learn meaningful embedding representations of full-length mRNA sequences without requiring the use of labeled data. Specifically, the self-supervised learning engine 250 learns embedding representations using unlabeled full-length mRNA sequence data, i.e., data that specifies or represents each of a set of mRNA sequences without mRNA characteristic labels.

[0122] The training dataset 225 may include mRNA codon sequences from multiple organisms, including, for example, mRNA sequences of mammalian, bacterial, yeast, and viral origin. In one particular example, the reference mRNA sequence is obtained from the NCBI dataset, which includes a mammalian reference sequence, an E. coli (E. coli) reference sequence, and a Homo sapiens virus whole nucleotide sequence. In some cases, the reference sequence is preprocessed and filtered to remove invalid and duplicate sequences. For example, preprocessing and filtering may include specifying that mRNA sequences have a sequence length that is a multiple of 3, that nucleotide T is replaced with U, that begin with a start codon ("AUG"), end with a stop codon ("UAA", "UAG", "UGA"), and / or contain only nucleotides from the set {A,U,G,C,N}.

[0123] In some cases, the mammalian sequence can be divided into three groups: marsupials, monotremes, and placental mammals. Placental mammals, which include the majority of extant mammals, can be further classified into nine subgroups: primates, carnivores, rodents, artiodactyls, bats, odd-toed ungulates, insectivores, rabbits, and further placental mammals.

[0124] A codon has three adjacent nucleotides, and each position has five options {A, U, G, C, N} after pretreatment. Therefore, the number of possible combinations is 5. 3 (125) There are. In addition, in some cases, five special tokens, namely the Classification Token [CLS], Separation Token [SEP], Unknown Word Token [UNK], Padding Token [PAD], and Masking Token [MASK], are added to the vocabulary. In total, the vocabulary of embedding model 220 has 130 tokens.

[0125] The self-supervised learning engine 250 starts with parameter values ​​222 of a first machine learning model, including an embedded model 220, and updates the parameter values ​​by minimizing a loss function defined for one or more pre-training tasks. In some cases, the pre-training tasks may include a masked language model (MLM) learning task for predicting masked codons in a known mRNA molecule and a homologous sequence prediction (HSP) task for predicting whether two input mRNA sequences belong to the same homologous class of organisms.

[0126] For an MLM learning task, the MLM loss function is:

number

[0127] For the HSP task, the HSP loss function is:

number

[0128] In exemplary embodiments, 13 categories of mRNA sequences are used for the HSP task, including Escherichia coli (E. coli), Homo sapiens virus, two groups of mammals (marsupials and monotremes), and nine subgroups of placental mammals (primates, carnivores, rodents, artiodactyls, bats, odd-toed ungulates, insectivores, rabbits, and further placental mammals). Two sequences belonging to the same category are considered homologous, and otherwise are not.

[0129] In some cases, to calculate the HSP loss, the embedding model 220 takes codon sequence pairs as input, concatenates them using a delimiter token ([SEP]), and adds a classification token ([CLS]) to the beginning of the combined sequence. The output embedding representation is used to calculate a binary classification to predict whether the sequence pairs are homologous sequences. The classification loss is calculated based on ground truth labels.

[0130] System 200 or another system may further include a supervised learning engine 260 configured to update model parameters 232 of a predictive model 230 based on a labeled dataset 265. The labeled dataset 265 includes multiple labeled training examples. Each training example includes (i) a training input specifying the representation of each mRNA, and (ii) a label specifying a characteristic measure of each mRNA. mRNA labels can be obtained based on experimental measurements of the corresponding mRNA characteristics.

[0131] Generally, a labeled dataset 265 contains a much smaller number of training sequences compared to an unlabeled dataset 255. In an illustrative example, an unlabeled dataset 255 might contain millions, tens of millions, or hundreds of millions of mRNA sequences, while a labeled dataset 265 might contain thousands, tens of thousands, or hundreds of thousands of labeled training sequences. In a specific example, a labeled dataset 265 could include data from publicly available mRNA datasets that provide diverse biophysical properties associated with these molecules.

[0132] For example, labeled dataset 265 may include data from the mRFP expression dataset, which provides protein production levels and whole coding sequences for several gene mutations in Escherichia coli (E. coli). This data characterizes whole-gene codon randomization that can help discover patterns and features correlated with translation efficiency, and assist in designing synthetic genes optimized for enhanced protein expression. Further information on the mRFP expression dataset is described in “Revealing determinants of translation efficiency via whole-gene codon randomization and machine learning,” Nieuwkoop et al., Nucleic Acids Research, 51(5):2363-2376, 012023.

[0133] Labeled dataset 265 can include data from fungal expression datasets containing protein-coding and tRNA genes derived from a wide range of fungal genomes. This data enables the discovery of common patterns of adaptive evolution across the fungal kingdom and provides insights into the evolutionary mechanisms and functional constraints that shape the evolution of fungal genes. Further information on fungal expression datasets is described in “Kingdom-Wide Analysis of Fungal Protein-Coding and tRNA Genes Reveals Conserved Patterns of Adaptive Evolution,” Wint et al., Molecular Biology and Evolution, 39(2), 01 2022. msab372.

[0134] Labeled dataset 265 can include data from heterologous proteins and corresponding experimental data under the same expression conditions in *E. coli* proteins. This data includes protein sequences, expression levels, and related features, providing insights into factors influencing expression efficiency. This data is useful for optimizing protein production and understanding protein expression mechanisms. Further information on heterologous protein data is described in “Mpepe, a predictive approach to improve protein expression in e. coli based on deep learning,” *Computational and Structural Biotechnology Journal*, 20:1142-1153, 2022.

[0135] Labelled dataset 265 can include data from the Tc-riboswitch dataset, which consists of a set of tetracycline (Tc) riboswitch dimer sequences upstream of the GFP protein. A riboswitch is a segment of mRNA that binds to a small molecule and causes an alteration in the production of the protein it encodes. Here, only the 5'UTR region is modified, while the GFP coding sequence remains the same. The measured variable in this dataset is the riboswitch switching factor, which generally refers to the differential effect of the riboswitch in the presence or absence of its Tc. This dataset is suitable for evaluating the structural and confirmation dynamics of the 5'UTR region of mRNA constructs. Further information on the Tc-riboswitch dataset is described in “Tuning the performance of synthetic riboswitches using machine learning,” Ann-Christin Groher, et al., ACS Synthetic Biology, 8(1):34-44, 2019.

[0136] Labeled dataset 265 may include data from CoV2 vaccine degradation datasets that encompass mRNA sequence sets adjusted for structural features, stability, and translational efficiency, providing insights for designing more effective RNA-based therapies. Further information on the CoV2 vaccine degradation datasets is described in “Combinatorial optimization of mrna structure, stability, and translation for rna-based therapeutics,” Kathrin Leppek, et al., Nature Communications, 13(1):1536, 2022.

[0137] The supervised learning engine 260 is configured to perform supervised learning of a second machine learning model, including the predictive machine learning model 230, on a labeled dataset 265. In other words, the supervised learning engine 260 is configured to update the parameters of the second machine learning model (including the model parameters 232 of the predictive machine learning model 230) based on the labeled dataset 265.

[0138] The supervised learning engine 260 can update the parameters of a second machine learning model (including the model parameters 232 of the embedded machine learning model 230) by minimizing the prediction error between the predicted mRNA characteristic scale and the characteristic scale specified by the label. The supervised learning engine 260 can update the model parameters using any appropriate machine learning technique.

[0139] Based on the predicted mRNA characteristic scale 240, the system 200 can select an mRNA sequence with desirable characteristics from candidate mRNA sequences for a specific application. For example, the system 200 can generate an output indicating whether a particular mRNA is suitable for a particular application, or an output specifying the optimal mRNA sequence for a particular application. The system can send the output to a manufacturing apparatus that can operate to carry out instructions for producing mRNA.

[0140] Figure 2B shows an exemplary process for generating an embedded representation of the entire mRNA sequence 210, including (i) 5'UTR 210a, (ii) CDS 210b, and (iii) 3'UTR 210c. The system then numerically encodes the nucleotide sequence 212a of the 5'UTR into a first input token vector, the codon sequence 212b of the CDS into a second input token vector, and the nucleotide sequence 212c of the 3'UTR into a third input token vector. To generate the token vectors for the 3' and 5'UTR, the system can generate the input token vectors by mapping each nucleotide of the corresponding nucleotide sequence (212a or 212c) to its respective numerical value and concatenating the numerical values. To generate the token vector for the CDS, the system can generate the second input token vector by mapping each codon of the CDS codon sequence 212b to its respective numerical value and concatenating the numerical values.

[0141] To extract meaningful features from these encoded sequences, the system uses three separate embedding machine learning models 220a, 220b, and 220c, each configured to process one of the input token vectors. Specifically, the first embedding model 220a processes the 5'UTR input vector to generate the first embedding feature vector 225a, the second embedding model 220b processes the CDS input vector to generate the second embedding feature vector 225b, and the third embedding model 220c processes the 3'UTR input vector to generate the third embedding feature vector 225c.

[0142] Next, the system combines the first, second, and third feature vectors (225a-225c) into the bound embedding representation 225 as a representation of the entire mRNA molecule. For example, the system can perform a pooling operation on these embedding feature vectors (225a-225c) and concatenate the embedding representations into the bound embedding representation 225. The bound embedding representation is then processed by a feature prediction machine learning model.

[0143] As further explained with reference to Figure 4B, the embedded models 210a–210b can be trained with a training mRNA sequence set, for example, an unlabeled mRNA sequence set.

[0144] Further explanation of the use and training of machine learning models for predicting mRNA molecular properties is provided below with reference to Figures 3A to 4B. Specifically, Figures 3A and 4A illustrate the process for using and training machine learning models to predict mRNA molecular properties based on the CDS codon sequence of the mRNA molecule.

[0145] Figure 3A is a flowchart illustrating an exemplary process 300A for predicting mRNA properties. For convenience, process 300A is described as being performed by a system of one or more computers located at one or more locations. For example, a property prediction system appropriately programmed in accordance with this disclosure, e.g., property prediction system 200 in Figure 2A, may perform process 300A.

[0146] In step 310a, the system obtains an mRNA sequence, i.e., a token sequence representing the codon sequence of the mRNA. The mRNA sequence may be a candidate mRNA sequence intended for a specific application.

[0147] In 320a, the system generates an input token vector by numerically encoding a token sequence. For example, the system can generate a vector by concatenating numerical values ​​assigned to different codons to form an input token vector.

[0148] In 330a, the system generates an embedded feature vector by processing an input token vector using an embedded machine learning model. The embedded machine learning model has a first set of model parameters. The first set of model parameters is updated using a first training process of a first machine learning model, which includes the embedded machine learning model, and the first training process is performed on a dataset specifying known codon sequences of mRNA molecules, and the first machine learning model is configured to perform one or more pre-training tasks.

[0149] In 340a, the system uses a predictive machine learning model to process an input containing embedded feature vectors and generate an output that predicts the characteristic scale of the input mRNA. The characteristic predictive machine learning model has a second model parameter set, which is updated using a second training process of the second machine learning model containing the characteristic predictive machine learning model based on multiple training examples. Each of the training examples includes (i) a training input that specifies a representation of each mRNA molecule, and (ii) a label that specifies one or more characteristics of each mRNA molecule.

[0150] In some cases, process 300A may be used to select mRNA molecules from a set of candidate mRNA molecules to perform downstream tasks (e.g., tasks to maximize mRNA expression levels in a particular type of cell or tissue, or tasks to maximize mRNA stability in a particular environment). In particular, the system can use the above process to predict the properties of each candidate mRNA molecule and select mRNA molecules from the set of candidate mRNA molecules based on the predicted properties.

[0151] In some cases, Process 300A can be used to determine the optimal codon sequence for an mRNA molecule to perform a specific task. The system maintains data representing a set of candidate mRNA molecule sequences and can process one or more of these candidate sequences using a reinforcement learning (RL) model to generate one or more new sequences. The RL model is trained using a reward signal containing mRNA molecule properties predicted using Process 300A. The system can then select the optimal sequence from the new sequences.

[0152] To train the RL model, the system can use the RL model to process an input sequence representing an mRNA molecule, generate one or more sets of actions that modify the input sequence, and determine a new sequence based on the input sequence and the set of actions. The system can calculate one or more reward values ​​indicating how well a particular task is performed by the mRNA molecule represented by the new sequence, and the reward values ​​can be calculated using one or more mRNA molecule properties predicted using process 300A. The system can tune one or more parameters of the RL model based on at least the reward values.

[0153] Figure 4A is a flowchart illustrating an exemplary process 400A for training a predictive model to predict mRNA characteristic measures. For convenience, process 400A is described as being performed by a system of one or more computers located at one or more locations. For example, a characteristic prediction system appropriately programmed in accordance with this disclosure, e.g., the characteristic prediction system 200 in Figure 2A, may perform process 400A.

[0154] Generally, a predictive model includes (i) an embedding machine learning model configured to generate embedding feature vectors for model inputs representing the codon sequences of mRNA, and (ii) a predictive machine learning model configured to process the embedding feature vectors to generate outputs that specify characteristics of mRNA.

[0155] In step 410a, the system obtains a first dataset containing a set of mRNA sequence representations. For example, a sequence representation could be a token vector that numerically encodes the codon sequences of a known mRNA sequence.

[0156] In 420a, the system performs self-supervised learning of a first machine learning model, including an embedded machine learning model, using the first dataset. To perform self-supervised learning, the system starts with the parameter values ​​of the first machine learning model and updates the parameter values ​​of the first machine learning model by minimizing a loss function defined for one or more pre-trained tasks.

[0157] In some cases, the pre-training task includes a Masked Language Model (MLM) training task to predict masked codons within known mRNA molecules. In these cases, the loss function is:

number

[0158] In some cases, the pre-training task includes a homologous sequence prediction (HSP) task to predict whether two input mRNA sequences belong to the same homologous class of organisms. In these cases, the loss function is:

number

[0159] In some cases, the loss function combines MLM loss and HSP loss.

[0160] In 430a, the system acquires a second dataset containing multiple training examples. Each training example includes (i) a training input specifying the representation of each mRNA, and (ii) a label characterizing one or more properties of each mRNA molecule.

[0161] In 440a, the system performs supervised learning of a second machine learning model, including a predictive machine learning model, based on a second dataset. The second machine learning model may include one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

[0162] Referring to Figures 3B and 4B, the following description illustrates the process of using and training a machine learning model to predict mRNA molecular properties from the whole mRNA sequence, including the 5'UTR, CDS, and 3'UTR.

[0163] Figure 3B is a flowchart showing another exemplary process 300B for predicting mRNA properties. For convenience, process 300B is described as being performed by a system of one or more computers located at one or more locations. For example, a property prediction system appropriately programmed in accordance with this disclosure, e.g., property prediction system 200 in Figure 2A, may perform process 300B.

[0164] In 310a, the system acquires data representing an mRNA molecule. The mRNA molecule includes (i) the 5' untranslated region (UTR), (ii) the coding sequence (CDS), and (iii) the 3' UTR. The mRNA sequence may be the sequence of a candidate mRNA intended for a specific application.

[0165] In 320b, this system generates a first input token vector by numerically encoding the nucleotide sequence of the 5'UTR of the mRNA molecule, a second input token vector by numerically encoding the codon sequence of the CDS of the mRNA molecule, and a third input token vector by numerically encoding the nucleotide sequence of the 3'UTR of the mRNA molecule. For example, this system can generate the first input token vector by mapping each nucleotide of the 5'UTR nucleotide sequence to its respective numerical value and concatenating the numbers. This system can generate the second input token vector by mapping each codon of the CDS codon sequence to its respective numerical value and concatenating the numbers. This system can generate the third input token vector by mapping each nucleotide of the 3'UTR nucleotide sequence to its respective numerical value and concatenating the numbers.

[0166] In 330b, the system generates a first embedding feature vector by processing a first input token vector using a first embedding machine learning model, a second embedding feature vector by processing a second input token vector using a second embedding machine learning model, and a third embedding feature vector by processing a third input token vector using a third embedding machine learning model. Before using the first, second, and third embedding machine learning models to generate the embedding feature vectors, the system or another system may train the models on a training mRNA sequence set using a first training process. The first training process is described in more detail with reference to Figure 4B. In some cases, each of the first, second, and third embedding machine learning models includes its respective Large Language Model (LLM). For example, each of the first, second, and third embedding machine learning models includes its respective bidirectional transformer.

[0167] In 340b, the system generates a combined embedding representation by combining a first embedding feature vector, a second embedding feature vector, and a third embedding feature vector. In some embodiments, to generate a combined embedding representation, the system performs a first pooling operation on the first embedding feature vector to generate a first embedding representation, a second pooling operation on the second embedding feature vector to generate a second embedding representation, a third pooling operation on the third embedding feature vector to generate a third embedding representation, and then concatenates the first, second, and third embedding representations to generate a combined embedding representation.

[0168] In 350b, the system processes the bound embedding representation using a feature prediction machine learning model to generate an output that predicts one or more features of an mRNA molecule. Before generating an output using the feature prediction machine learning model, the system or another system may train the feature prediction machine learning model on a labeled training example set using a second training process. The second training process is described in more detail with reference to Figure 4B.

[0169] In some cases, process 300B may be used to select mRNA molecules from a set of candidate mRNA molecules to perform downstream tasks (e.g., tasks to maximize mRNA expression levels in a particular type of cell or tissue, or tasks to maximize mRNA stability in a particular environment). In particular, the system can use the above process to predict the properties of each candidate mRNA molecule and select mRNA molecules from the set of candidate mRNA molecules based on the predicted properties.

[0170] In some cases, Process 300B can be used to determine the optimal sequence of an mRNA molecule for performing a specific task. This system maintains data representing a set of candidate mRNA molecule sequences and can use a reinforcement learning (RL) model to process one or more of these candidate sequences to generate one or more new sequences. The RL model is trained using a reward signal containing mRNA molecule properties predicted using Process 300B. The system can then select the optimal sequence from the new sequences.

[0171] To train the RL model, the system can use the RL model to process an input sequence representing an mRNA molecule, generate one or more sets of actions that modify the input sequence, and determine a new sequence based on the input sequence and the set of actions. The system can calculate one or more reward values ​​indicating how well a particular task is performed by the mRNA molecule represented by the new sequence, and the reward values ​​can be calculated using one or more mRNA molecule properties predicted using process 300A. The system can tune one or more parameters of the RL model based on at least the reward values.

[0172] Figure 4B is a flowchart illustrating an exemplary process 400B for training a predictive model to predict mRNA characteristic measures. For convenience, process 400B is described as being performed by a system of one or more computers located at one or more locations. For example, a characteristic prediction system appropriately programmed in accordance with this disclosure, e.g., the characteristic prediction system 200 in Figure 2A, may perform process 400A.

[0173] Generally, an mRNA molecule includes (i) a 5' untranslated region (UTR), (ii) a coding sequence (CDS), and (iii) a 3'UTR. The prediction model includes: (i) a first embedding machine learning model configured to process a first input token vector representing the nucleotide sequence of the 5'UTR of an mRNA molecule to generate a first embedding feature vector; (ii) a second embedding machine learning model configured to process a second input token vector representing the codon sequence of an mRNA molecule to generate a second embedding feature vector; (iii) a third embedding machine learning model configured to process a third input token vector representing the nucleotide sequence of the 3'UTR of an mRNA molecule to generate a third embedding feature vector; and (iv) a feature prediction machine learning model configured to process a combined embedding representation generated by combining the first, second, and third embedding feature vectors to generate an output specifying one or more features of an mRNA molecule.

[0174] In 410b, the system obtains a first dataset containing a set of mRNA sequence representations. For example, each mRNA sequence representation may include a first token vector that numerically encodes the nucleotide sequence of the 5'UTR of a known mRNA molecule, a second token vector that numerically encodes the CDS codon sequence of the known mRNA sequence, and a third token vector that numerically encodes the nucleotide sequence of the 3'UTR of the known mRNA molecule.

[0175] In 420b, the system performs self-supervised learning to train first, second, and third embedded machine learning models on one or more pre-training tasks using a first dataset. To perform self-supervised learning, the system may train the first machine learning model by starting the parameter values ​​of the first machine learning model, which includes the first, second, and third embedded machine learning models, and then minimizing one or more pre-training loss functions, which include one or more pre-training losses defined for one or more pre-training tasks.

[0176] In some cases, the first, second, and third embedded machine learning models are trained simultaneously. That is, the system can update the parameters of these models by minimizing a combined loss function that combines at least the pre-trained losses calculated for each of the first, second, and third embedded machine learning models.

[0177] In some other cases, to improve computational efficiency, the first, second, and third embedded machine learning models are trained separately. That is, the system updates the parameters of each particular embedded model by minimizing the loss function of that particular model.

[0178] In some cases, the pre-training task includes training a Masked Language Model (MLM) to predict masked tokens within a known mRNA molecule. In these cases, the loss function is:

number

[0179] In some cases, the pre-training task includes a homologous sequence prediction (HSP) task to predict whether two input mRNA sequences belong to the same homologous class of organisms. In these cases, the loss function is:

number

[0180] In some cases, the loss function combines MLM loss and HSP loss.

[0181] In 430b, the system acquires a second dataset containing multiple training examples. Each training example includes (i) a training input specifying the representation of each mRNA molecule, and (ii) a label characterizing one or more properties of each mRNA molecule.

[0182] In 440b, the system performs supervised learning to train a feature prediction machine learning model on one or more feature prediction tasks using a second dataset. The second machine learning model may include one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model. To train the feature prediction machine learning model, the system may train the second machine learning model by starting the parameter values ​​of the second machine learning model, which contains the feature prediction machine learning model, and minimizing a downstream loss function that includes one or more prediction losses defined for one or more feature prediction tasks.

[0183] In some embodiments, downstream training of a feature prediction machine learning model is performed such that the parameters of the embedding model are fixed to the values ​​obtained by self-supervised pre-training. In other words, the parameters of the embedding model are not updated during downstream supervised training. This technique leverages the embedding process learned during pre-training without further modification.

[0184] In some other embodiments, the parameter values ​​of the embedding model are updated during downstream supervised training. In these cases, the parameters of the embedding model may be further fine-tuned for each downstream task, potentially leading to improved performance at estimation. However, updating all parameters of both the embedding model and the predictive model during downstream training can require significant computing resources, especially for large models. For example, in some cases, a composite model integrating an embedding model and a characteristic predictive model may contain hundreds of millions of parameters. This can require a considerable amount of computing resources, longer training times, and increased energy consumption.

[0185] To address these challenges, the system can employ techniques such as Low-Rank Adaptation (LoRA), which allows for the selective fine-tuning of smaller subsets of parameters within the model, thereby improving training efficiency while maintaining robust model performance and reducing the risk of overfitting. Specifically, with LoRA, the system inserts a trainable rank decomposition matrix into each transformer layer of the embedding model while keeping the parameters of the originally pre-trained embedding model fixed during downstream training.

[0186] In one example of implementing LoRA, this system has a size of d in ×r and d out Two fully connected layers with ×r are injected, d in and d out d are the size of the hidden layers and the size of the intermediate layers of the pre-trained embedding model, and r is the LoRA rank, a hyperparameter that affects the number of weights. In an exemplary example, d in and d out The values ​​for all 12 layers are 764 and 3072, respectively. Using 32 ranks in LoRA, the system obtains a total of (768 + 3072) × 12 × 32 = 1,474,560 adjusted parameters, which may be less than 2% of the total parameters in the pre-trained embedding model.

[0187] In some embodiments, the system can further incorporate a control loss into the pre-training of the embedding model or the downstream training of the feature prediction model. The control loss aims to maximize the similarity between embedding representations of different regions within the same mRNA sequence while minimizing the similarity between embedding representations of different regions from different mRNA sequences, and can be incorporated into the pre-training loss function or the downstream loss function.

[0188] ISI, control loss L C The first control loss L aims to maximize the similarity between 5'UTR and CDS embedding expressions within the same mRNA sequence while minimizing the similarity between 5'UTR and CDS embedding expressions from two different mRNA sequences. C1 Includes. In some cases, the counter loss L C The first control loss L aims to maximize the similarity between 3'UTR and CDS embedding expressions within the same mRNA sequence while minimizing the similarity between 3'UTR and CDS embedding expressions from two different mRNA sequences. C2 Includes.

[0189] As an exemplary example, the first and / or second control loss may be calculated as follows:

[0190]

number

[0191] In some cases, the control loss may be calculated as a composite control loss, which is a combination of the first control loss and the second control loss. For example, the composite control loss is the average of the first control loss and the second control loss, i.e., LC =(L C1 +L C2 It can be calculated as ) / 2.

[0192] In some cases, this system can incorporate the counter loss into the downstream training of the characteristic prediction model by including the counter loss in the downstream loss function. For example, the downstream loss function is L downsstream =L prediction +αL C It can be calculated as follows, where α is the adjustment coefficient.

[0193] In some other cases, the system can incorporate the control loss into the pre-training of the embedding models by including the control loss in the pre-training loss function. For example, if the first, second, and third embedding machine learning models are trained simultaneously, the system can incorporate the control loss into the combined loss function. In another example, if the first, second, and third embedding machine learning models are trained separately, the system can incorporate the control loss into the loss function of the second embedding machine learning model that generates the embedding representation for CDS.

[0194] Figure 5 is a block diagram of an exemplary computer system 500 that may be used to perform the operations described above. The system 500 includes a processor 510, memory 520, storage devices 530, and input / output devices 540. Each of the components 510, 520, 530, and 540 may be interconnected using, for example, a system bus 550. The processor 510 can process instructions to be executed within the system 500. In one embodiment, the processor 510 is a single-threaded processor. In another embodiment, the processor 510 is a multi-threaded processor. The processor 510 can process instructions stored in memory 520 or storage devices 530.

[0195] The memory 520 stores information within the system 500. In one embodiment, the memory 520 is a computer-readable medium. In one embodiment, the memory 520 is a volatile memory unit. In another embodiment, the memory 520 is a non-volatile memory unit.

[0196] The storage device 530 can provide large-capacity storage to the system 500. In one embodiment, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large-capacity storage device.

[0197] The input / output device 540 provides input / output operations to the system 500. In one embodiment, the input / output device 540 may include one or more network interface devices, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as a 502.11 card. In another embodiment, the input / output device may include a driver device configured to receive data and transmit output data to other input / output devices, such as a keyboard, printer, and display device 560. However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, and set-top box television client devices.

[0198] An exemplary processing system is shown in Figure 5, but embodiments of the subject matter and functional operation described herein may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, or a combination thereof, including the structures disclosed herein and their structural equivalents.

[0199] Figure 6 shows a performance comparison of different mRNA characteristic prediction models for various prediction tasks. Performance is measured by the root mean square error (RMSE) of the prediction results compared to benchmark data.

[0200] These models include the "CodonBERT" model, which includes a BERT model for encoding mRNA codon sequences and a downstream prediction model. Embedding models for embedding the 3' and 5' UTRs are not included in this model. The "RNABERT" model includes a BERT model for encoding mRNA nucleotide sequences. The "TextCNN-Nuc" model uses a convolutional neural network (CNN) to process mRNA nucleotide sequences. The "TextCNN-codon" model uses a CNN to process mRNA codon sequences. As shown in Figure 6, CodonBERT outperforms the other models in protein expression tasks, which demonstrates the effectiveness of the disclosed method.

[0201] Figure 7 shows another performance comparison of different mRNA characteristic prediction models for various prediction tasks. Performance is measured by Spearman's correlation coefficient or the area under the receiver manipulation characteristic curve (AUROC).

[0202] These models include a first BERT model for encoding the 5'UTR, a second BERT model for encoding mRNA codon sequences, a third BERT model for encoding the 3'UTR, and the “mRNA-LM” model, which includes downstream predictive models. The “RNA-FM” model is described in Chen, et al., “Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function”, arXiv:2204.00300, 2022. The “Saluki” model is described in Agarwal et al., “The genetic and biochemical determinants of mRNA degradation rates in mammals”, Genome Biology”, 23(1), 245, 2022. As shown in Figure 7, mRNA-LM outperforms the other models in predicting mRNA half-life, translation rate, transcript expression, and protein expression. This comparison demonstrates the effectiveness of the disclosed methods.

[0203] This disclosure uses the term “configured” in relation to systems and computer program components. In the case of a system of one or more computers, being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed on the system to cause the system to perform the operation or action during operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs, when executed by a data processing device, include instructions that cause that device to perform the operation or action. The subject matter and functional operation embodiments described in this disclosure may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described in this disclosure may be implemented as one or more modules of computer program instructions, i.e., computer program instructions, which are executed by one or more computer programs, i.e., data processing devices, or encoded in tangible non-temporary storage media for controlling their operation. Computer storage media may be machine-readable storage devices, machine-readable storage boards, random or serial access memory devices, or one or more combinations thereof. Alternatively or additionally, program instructions may be encoded with mechanically generated electrical, optical, or electromagnetic signals, such as those generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0204] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for data processing, including, for example, programmable processors, computers, or multiple processors or computers. A device may be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include such circuits. Optionally, in addition to hardware, a device may include code that forms the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0205] Computer programs, also called or written as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computer environment. A program may or may not correspond to a file in a file system. A program may be stored in part of a file that holds other programs or data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple collaborative files, for example, in a file that stores one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0206] In this disclosure, the term “database” is used broadly to refer to any collection of data, that is, data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored in one or more storage devices. Thus, for example, an index database may contain multiple collections of data, each of which may have a different organization and access.

[0207] Similarly, in this disclosure, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. One or more computers may be dedicated to a particular engine, or multiple engines may be installed and operate on one or more of the same computers.

[0208] The processes and logic flows described herein may be executed by one or more programmable computers running one or more computer programs that perform functions by acting on input data and producing outputs. The processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0209] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a central processing unit that executes or carries out instructions, and one or more memory devices that store instructions and data. The central processing unit and memory may be augmented by or incorporated into dedicated logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operablely coupled to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into another device, to name a few, such as a mobile phone, personal digital assistant (PDA), mobile voice or video player, game console, Global Positioning System (GPS) receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0210] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks.

[0211] To enable user interaction, embodiments of the subject matter described herein may be implemented using a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a pointing device, such as a keyboard and mouse or trackball, to which the user can provide input to the computer. Other types of devices may be used to enable user interaction, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and the input from the user may be received in any form, including acoustic, voice, or haptic input. In addition, the computer may interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving response messages from the user.

[0212] The data processing unit that implements the machine learning model may also include, for example, a dedicated hardware accelerator unit to handle the general, computationally intensive parts of machine learning training or production, i.e., inference, workload.

[0213] Machine learning models can be implemented and deployed using machine learning frameworks, such as PyTorch, Scikit-learn, Keras, or TensorFlow.

[0214] Multiple embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component as an application server, or a client computer equipped with a frontend component, such as a graphical user interface, web browser, or application that allows a user to interact with embodiments of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0215] A computing system may include a client and a server. The client and server are generally located remotely from each other and typically interact via a communication network. The relationship between the client and the server is established by computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving input from the user. Data generated on the user device, such as the results of the interaction with the user, may be received from the device on the server side.

[0216] This disclosure includes details of many specific embodiments, but these should not be construed as limiting the scope of any invention or claims, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this disclosure in relation to separate embodiments may be implemented in combination within a single embodiment. Conversely, various features described in relation to a single embodiment may be implemented separately or in any suitable partial combination in multiple embodiments. Furthermore, features may be described as operating in a particular combination, and may be described in the claims from the outset as such, but one or more features from a claimed combination may, in some cases, be excluded from that combination, and the claimed combination may refer to a partial combination or a variation of a partial combination.

[0217] Similarly, although operations are shown in the drawings and enumerated in the claims in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown, or that all of the operations shown must be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems should generally be understood to be able to be incorporated together in one software product or packaged into multiple software products.

[0218] We have described specific embodiments of the subject matter. Other embodiments are also included in the scope of the following claims. For example, the desired results can be achieved by performing the higher steps listed in the claims in a different order. As an example, the processes shown in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to obtain the desired results. Multitasking and parallel processing may be advantageous in some cases.

Claims

1. A computer-based method for predicting the properties of an mRNA molecule, wherein the method is To obtain data representing the codon sequence of the mRNA molecule, The input token vector is generated by numerically encoding the aforementioned codon sequence, The method involves generating an embedding feature vector by processing the input token vector using an embedding machine learning model having a first set of model parameters, wherein the first set of model parameters is updated using a first training process of a first machine learning model including the embedding machine learning model, the first training process is performed on a dataset specifying known codon sequences of mRNA molecules, and the first machine learning model is configured to perform one or more pre-training tasks. The method involves processing the embedding feature vector using a feature prediction machine learning model to generate an output that predicts one or more characteristics of the mRNA molecule, wherein the feature prediction machine learning model has a second set of model parameters, the second set of model parameters is updated using a second training process of a second machine learning model including the feature prediction machine learning model, based on a plurality of training examples, each of which training examples includes (i) a training input specifying a representation of each mRNA molecule, and (ii) a label specifying one or more characteristics of each mRNA molecule. A computer implementation method, including

2. The generation of the aforementioned input token vector is Mapping each codon in the aforementioned codon sequence to its respective numerical value, The token vector is generated by concatenating the aforementioned numerical values. The method according to claim 1, including the method described in claim 1.

3. The first training process described above is The parameters of the first machine learning model described above are to be started, The values ​​of the parameters of the first machine learning model are updated by minimizing the loss function defined for one or more pre-training tasks. The method according to claim 1 or 2, including the method described in claim 1 or 2.

4. The method according to claim 3, wherein the one or more pre-training tasks include a masked language model (MLM) training task for predicting masked codons in known mRNA molecules.

5. The aforementioned loss function is, [Math 1] Includes an MLM loss function defined as, In the formula, X represents the array batch, and p(x) ｉ │x Ｍ ) is the unmasked part x of the input array x Ｍ Given, token x ｉ The method according to claim 4, representing the probability of the first machine learning model predicting that the object is located at a specific masked position i.

6. The method according to any one of claims 3 to 5, wherein the one or more pre-training tasks include a homologous sequence prediction (HSP) task for predicting whether two input mRNA sequences belong to the same homologous class of organisms.

7. The aforementioned loss function is, [Math 2] Includes an HSP loss function defined as, In the formula, y ｎ This represents the ground truth label indicating whether the two input token sequences represent mRNA codon sequences belonging to the same homologous class, and p ｎ The method according to claim 6, wherein the two input token sequences represent the predicted probability that each of them represents an mRNA codon sequence belonging to the same homologous class.

8. The method according to any one of claims 3 to 7, wherein the loss function combines MLM loss and HSP loss.

9. The method according to any one of claims 1 to 8, wherein one or more of the mRNA molecule's properties include the expression level of the mRNA molecule in a particular type of cell or tissue.

10. The method according to claim 9, wherein the mRNA molecule is a component of a vaccine and is encoded to express one or more antigen proteins of a target pathogen, and the predicted properties of the mRNA molecule characterize the expression level of the antigen proteins of the target pathogen in the particular type of cell or tissue.

11. The method according to any one of claims 1 to 10, wherein the one or more properties of the mRNA molecule include stability under one or more environmental conditions.

12. The method according to any one of claims 1 to 11, wherein one or more of the mRNA molecule's properties include a switching factor for the mRNA molecule in a specific type of cell or tissue.

13. The method according to any one of claims 1 to 12, wherein one or more of the characteristics of the mRNA molecule includes the degradation rate of the mRNA molecule under one or more environmental conditions.

14. The method according to claim 13, wherein the mRNA molecule is a component of a SARS-CoV-2 vaccine, and the characteristic prediction machine learning model is configured to predict the degradation rate of the mRNA molecule under physiological conditions.

15. The method according to any one of claims 1 to 14, wherein the first machine learning model includes a large-scale language model (LLM).

16. The method according to any one of claims 1 to 15, wherein the first machine learning model includes a bidirectional transformer.

17. The method according to any one of claims 1 to 16, wherein the characteristic prediction machine learning model includes one or more of the following: a neural network, a K nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

18. The method according to any one of claims 1 to 17, wherein the characteristic prediction machine learning model includes a convolutional neural network (CNN).

19. A method for selecting an mRNA molecule from a set of candidate mRNA molecules to perform a downstream task, wherein the method is Predicting the characteristics of each of the candidate mRNA molecules using the method according to any one of claims 1 to 18, Based on the predicted characteristics, select the mRNA molecule from the candidate mRNA molecule set. Methods that include...

20. To physically generate the selected mRNA molecule. The method according to claim 19, further comprising:

21. The mRNA molecule selected using the method described in claim 19.

22. The aforementioned downstream task is To maximize the expression level of the mRNA in a specific type of cell or tissue, or to maximize the stability of the mRNA in a specific environment. The method according to claim 19, comprising one or more of the above.

23. A method for determining the optimal codon sequence of an mRNA molecule for performing a specific task, wherein the method is Maintaining data representing candidate sequence sets for the aforementioned mRNA molecule, Using a reinforcement learning (RL) model, one or more of the candidate sequences are processed to generate one or more new sequences, wherein the RL model is trained using a reward signal that includes mRNA molecular properties predicted using the method according to any one of claims 1 to 18. Selecting the optimal sequence from the aforementioned new sequences and Methods that include...

24. A method for training a reinforcement learning (RL) model to determine the optimal codon sequence of an mRNA molecule for performing a specific task, wherein the method is Using the aforementioned RL model, the system processes an input sequence representing an mRNA molecule to generate one or more sets of actions that modify the input sequence. Determining a new sequence based on the aforementioned input sequence and the aforementioned action set, Calculating one or more reward values ​​indicating how well the particular task is performed by the mRNA molecule represented by the new sequence, wherein the reward values ​​are calculated using one or more mRNA molecule characteristics predicted using the method of any one of claims 1 to 18. Adjusting at least one parameter of the RL model based on the reward value: Methods that include...

25. A computer implementation method for training a predictive model for predicting the properties of an mRNA molecule, wherein the predictive model includes (i) an embedding machine learning model configured to generate embedding feature vectors for model inputs representing the codon sequence of the mRNA molecule, and (ii) a property prediction machine learning model configured to process the embedding feature vectors to generate an output specifying one or more properties of the mRNA molecule, and the method Obtaining a first dataset containing a sequence representation set of mRNA molecules, Training a first machine learning model, including the embedded machine learning model, on one or more pre-training tasks using the first dataset, Obtaining a second dataset containing multiple training examples, wherein each training example includes (i) a training input specifying a representation of each mRNA molecule, and (ii) a label characterizing one or more properties of each mRNA molecule. Performing supervised learning of a second machine learning model, including the characteristic prediction machine learning model, on the second dataset: A computer implementation method, including

26. Training the first machine learning model described above is The parameters of the first machine learning model described above are to be started, The values ​​of the parameters of the first machine learning model are updated by minimizing the loss function defined for one or more pre-training tasks. The method according to claim 25, including the method described in claim 25.

27. The method according to claim 26, wherein the one or more pre-training tasks include a masked language model (MLM) training task for predicting masked codons in known mRNA molecules.

28. The aforementioned loss function is, [Math 3] Includes an MLM loss function defined as, In the formula, X represents an array batch, and p(xi│xM) is the unmasked portion x of the input array x. Ｍ Given, token x ｉ The method according to claim 26, representing the probability of the first machine learning model predicting that the object is located at a specific masked position i.

29. The method according to any one of claims 26 to 28, wherein the one or more pre-training tasks include a homologous sequence prediction (HSP) task for predicting whether two input mRNA sequences belong to the same homologous class of organisms.

30. The aforementioned loss function is, [Math 4] Includes an HSP loss function defined as, where y ｎ represents a ground truth label indicating whether two input token arrays represent mRNA codon arrays belonging to the same homology class, and p ｎ represents the predicted probability that the two input token arrays represent mRNA codon arrays belonging to the same homology class, the method according to claim 29.

31. The method according to any one of claims 26 to 30, wherein the loss function combines MLM loss and HSP loss.

32. The one or more of the above characteristics of the mRNA molecule are The expression level of the mRNA molecule in a specific type of cell or tissue, Stability under one or more environmental conditions, or Switching factors for the mRNA molecule in specific types of cells or tissues The method according to any one of claims 25 to 31, comprising one or more of the above.

33. The generation of the aforementioned input token vector is Mapping each codon in the aforementioned codon sequence to its respective numerical value, The token vector is generated by concatenating the aforementioned numerical values. The method according to any one of claims 25 to 32, including the method described in that claim.

34. The method according to any one of claims 25 to 33, wherein the first machine learning model includes a large-scale language model (LLM).

35. The method according to claim 34, wherein the first machine learning includes a bidirectional transformer.

36. The method according to any one of claims 25 to 35, wherein the characteristic prediction machine learning model includes one or more of the following: a neural network, a K nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

37. One or more computers, When executed by the one or more computers, the one or more storage devices store instructions that cause the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 36. A system that includes this.

38. One or more computer-readable storage media that, when executed by one or more computers, stores instructions causing one or more computers to perform the operation of each of the methods described in any one of claims 1 to 36.

Images