A method for optimizing mRNA stability based on reinforcement learning, electronic device and storage medium

Abstract

A method for optimizing mRNA stability based on reinforcement learning, an electronic device and a storage medium belong to the technical field of biological sequence data processing. In order to solve the problems of poor generalization, initial value dependence, and single optimization target of the existing methods, the present invention constructs a training data set, an intelligent agent and an environment, wherein the intelligent agent is composed of an actor network and a critic network, and the actor network and the critic network share the parameters of the input layer, hidden layer 1, and hidden layer 2; the environment accepts the output action of the actor network and makes corresponding modifications to the state variables, outputs a reward function at the same time, collects training trajectory data, uses a proximal strategy optimization algorithm to train the learning process of the reinforcement learning intelligent agent in the environment, updates the intelligent agent network parameters, and calculates the MFE of the optimized sequence until the MFE of the optimized sequence is lower than the template MFE or the MFE of the optimized sequence decreases and tends to stabilize, and the intelligent agent training is completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of biological sequence data processing, and specifically relates to an mRNA stability optimization method based on reinforcement learning, an electronic device and a storage medium. Background Art

[0002] As a powerful weapon against viruses, mRNA vaccines can be quickly designed and optimized based on sequencing sequences, and can enter animal experiments and clinical trial screening. At the same time, mRNA will not be integrated into the host genome, which is highly safe. However, the instability of mRNA single strands often causes hydrolysis, which greatly hinders the production, preparation, transportation and distribution of vaccines. Intact mRNA molecules are crucial to the effectiveness of vaccines, and the instability of mRNA in vivo often leads to insufficient protein expression. Slight degradation reactions in any area of ​​mRNA will severely slow down or even terminate the translation process of mRNA, ultimately leading to incomplete expression of the target antigen. Therefore, how to improve the stability of mRNA is particularly important for vaccine preparation.

[0003] Traditional mRNA algorithms are often based on dynamic programming methods or heuristic methods. Dynamic programming methods are generally based on traditional thermodynamic assumptions, and optimize the thermodynamic stability of sequences from the bottom up by gradually calculating the optimal substructure. However, the computational efficiency and complexity of these methods are relatively high. When the coding protein sequence changes, the sequence needs to be calculated from scratch. And the optimization capability is limited to thermodynamic stability. Another type of optimization algorithm based on heuristic methods can not only optimize thermodynamic stability, but also achieve multi-objective optimization. However, this type of method often has problems such as unstable optimization results and initial value dependence. And when the coding sequence changes, the sequence still needs to be re-searched.

[0004] In recent years, artificial intelligence methods, as a "data-driven" method, have achieved certain results in biomedical fields such as protein and genetic engineering, and have gradually developed into a new and efficient strategy for biological sequence design, but it also requires more in-depth research. Summary of the invention

[0005] The problem to be solved by the present invention is to solve the problems of poor generalization, initial value dependence, single optimization target, etc. of the existing methods, and propose an mRNA stability optimization method, electronic device and storage medium based on reinforcement learning.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A method for optimizing mRNA stability based on reinforcement learning comprises the following steps:

[0008] S1. Extract multiple similar amino acid sequences from the protein library or use the wild-type antigen amino acid sequence to be optimized to perform amino acid substitution and deletion mutation to obtain multiple similar amino acid sequences, construct a training data set, and obtain the optimized synonymous mRNA sequence of each amino acid sequence according to the maximum nucleotide pairing method for the multiple similar amino acid sequences in the training data set, and calculate the minimum free energy MFE of the mRNA sequence for subsequent reward setting;

[0009] S2. Construct an agent and an environment, wherein the agent is composed of an actor network and a critic network. The actor network is composed of an input layer-hidden layer 1-hidden layer 2-actor output layer, and the critic network is composed of an input layer-hidden layer 1-hidden layer 2-critic output layer; the actor network and the critic network share the parameters of the input layer, hidden layer 1, and hidden layer 2; the environment accepts the output action of the actor network and makes corresponding modifications to the state variables, and outputs a reward function at the same time;

[0010] S3. Collect training trajectory data, perform multiple sequence comparison on the training data set obtained in step S1, determine the size of the state space in the environment constructed in step S2 by the length of the longest amino acid sequence or the amino acid sequence of the wild-type antigen to be optimized, concatenate the one-hot encoding of the amino acid sequence in the training data set with the one-hot encoding of the mRNA sequence as a state variable, input it into the intelligent agent actor network to obtain the corresponding action output, modify the state variable using the action output and give the corresponding reward from the environment, collect the combination of {state-action-reward}, and obtain the training trajectory data set;

[0011] S4. Based on the training trajectory data obtained in step S3, a proximal policy optimization algorithm is used to train the learning process of the reinforcement learning agent in the environment, update the agent network parameters, and calculate the MFE of the optimization sequence;

[0012] S5. Repeat steps S3 and S4 until the MFE of the optimized sequence is lower than the template MFE or the MFE of the optimized sequence decreases and tends to stabilize, completing the agent training.

[0013] Furthermore, the specific implementation method of step S1 includes the following steps:

[0014] S1.1. Rules for setting nucleotide pairing function:

[0015] The paired nucleotides are one of adenine A-uracil U pairing, guanine G-cytosine C pairing, and guanine G-uracil U pairing, and there must be at least three nucleotides between the paired nucleotides. The expression is:

[0016]

[0017] Among them, δ <n i ,n j > is the nucleotide pairing function between the nucleotide at the i-th position and the nucleotide at the j-th position, n i is the nucleotide at position i, n j is the nucleotide at the jth position; it takes 1 when the nucleotide matches, otherwise it takes 0;

[0018] Each nucleotide pairs with at most one nucleotide, expressed as:

[0019]

[0020] S1.2. Based on the nucleotide pairing function of step S1, set a recursive function ψ of the number of nucleotide pairings, The maximum number of nucleotide matches between the nucleotide at the i-th position and the nucleotide at the j-th position in the RNA subsequence is used as an approximate criterion for calculating the minimum free energy. The expression is:

[0021]

[0022] Among them, N i+1 |n i Represents the selection of n i After n i+1 The feasible set, N j-1 ∩n j Represents the selection of n j After n j-1 The feasible set of , k is an integer between i and j;

[0023] S1.3. Solve the dynamic programming equation F(x) based on the recursive function of the number of nucleotide pairings in step S1.2, and the expression is:

[0024]

[0025] st{n 3i-2 , n 3i-1 ,n 3i}∈codon_set(p i ), i = 1, 2, ..., L

[0026] Where P is the amino acid sequence constraint, p i is the ith amino acid in the amino acid sequence, codon_set(p i ) is an optional set of codons under given acid constraints;

[0027] The optimized synonymous mRNA sequence is obtained through the dynamic programming equation, and then the minimum free energy value of the mRNA is calculated as the template MFE corresponding to the amino acid sequence data.

[0028] Furthermore, the activation function of the hidden layer 1 and the hidden layer 2 in step S2 is ReLU; the input of the actor network and the critic network is the state variable s t The output of the actor network is the selection of amino acid codons, which is used to modify the mRNA sequence; the output of the critic network is the correction of the reward function, which is used to calculate the back-propagation gradient of the actor network. The input dimension of the actor network and the critic network is set to 32L, and the output dimension is set to 6 and 1 respectively. The corresponding network parameters are randomly initialized using the orthogonal initialization method, where L is the longest amino acid sequence length.

[0029] Furthermore, the specific implementation method of step S3 includes the following steps:

[0030] S3.1. The mRNA sequence optimization process is established as a Markov decision process. In different rounds i, the training data set P = {p1, p2, ... p k …,p M The amino acid sequence p in k For training, k = i mod M;

[0031] S3.2. After performing multiple sequence comparisons on the training data set obtained in step S1, the length L of the longest amino acid sequence or the amino acid sequence of the wild-type antigen to be optimized is determined to be 32L in the state space in the environment constructed in step S2;

[0032] S3.3. Concatenate the one-hot encoding of the amino acid sequence in the training dataset with the one-hot encoding of the mRNA sequence as the state variable s t , where the amino acid sequence is kept constant in a single round as a constraint, and the mRNA is set to an empty sequence three times the length of the protein sequence at time t0; the actor network is used to output the probability distribution p of the optimal action under a given state. θ (a t |s t ), where θ is the parameter of the actor network, and the action a at time t t The expression is:

[0033]

[0034] a t The meaning is that according to the encoding of the t-th amino acid, the corresponding codon is selected to fill in the corresponding position of the mRNA;

[0035] S3.4. Characteristic encoding is performed on the mRNA sequence formed after filling according to the nucleotides, and it is spliced ​​with the characteristic encoding of the amino acid to form the state s at time t t+1 , calculate MFE at the termination time to calculate the reward function r N , at the non-terminal moment, the reward function r t Get trajectory data for 0 {s t ,a t+1 ,r t+1 ,s t+1 |t=0,…,N-1}.

[0036] Furthermore, the calculation method of the reward function in step S3.4 is:

[0037] At the end time, the minimum free energy MFE (rna seq ) size and template MFEJ template Do the difference and get the reward function, which is expressed as:

[0038] r N =J template -MFE(rna seq );

[0039] Or add the codon adaptive exponential weighting and complete the multi-objective optimization, and the expression is:

[0040]

[0041] where ω(c) is a function of the adaptability of codon c, defined as the frequency of codon c divided by the frequency of the most common synonymous codon, codon(rna seq ,i) is the mRNA sequence rna seq The i-th codon.

[0042] Furthermore, the specific implementation method of step S4 is to train the intelligent agent actor network based on the training trajectory data obtained in step S3 using the proximal strategy optimization algorithm, and t ,a t+1 ,s t+1 ,r t+1 Store it in the buffer area, and calculate the loss function based on the data in the buffer area after each interaction trajectory. The expression is:

[0043]

[0044] Among them, θ old is the actor network parameter when sampling trajectory data, θ is the actor network parameter to be optimized, It is an importance sampling indicator, which is used to evaluate whether the network used for sampling is similar to the current network, and to control the optimization target based on the degree of similarity. Buffer is the buffer zone.

[0045] Set clip as the clipping function, the expression is:

[0046]

[0047] Where ∈ is the cropping factor, set ∈ = 0.2, and update the actor parameters θ and the critic network parameters φ by maximizing the loss function using the ADAM algorithm;

[0048] Setting the advantage function The expression is:

[0049]

[0050] The input of the actor network and the critic network is the observation state s at time t. t , the output of the actor network is action a t , the output of the critic network is the value function V φ (s t ), used to estimate the advantage function To replace the actual reward value r t .

[0051] Further, step S5 repeats step S3 and step S4 until the reward function output of the optimized sequence at the termination time is greater than 0, at which point the MFE of the optimized sequence is lower than the template MFE, or the variance of the MFE is less than 5 kcal / mol within 20 training rounds, at which point the MFE of the optimized sequence decreases and tends to stabilize, completing the agent training.

[0052] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for optimizing mRNA stability based on reinforcement learning when executing the computer program.

[0053] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the mRNA stability optimization method based on reinforcement learning.

[0054] Beneficial effects of the present invention:

[0055] The mRNA stability optimization method based on reinforcement learning described in the present invention applies the reinforcement learning algorithm to mRNA optimization, provides users with a new black box tool for optimizing similar mRNA sequences, has good generalization in offline testing, and also has optimization capabilities under variable-length tasks.

[0056] The mRNA stability optimization method based on reinforcement learning described in the present invention has a relatively high operating efficiency in offline testing and can optimize a hundred times the number of protein constraint sequences of traditional methods in the same amount of time.

[0057] In the mRNA stability optimization method based on reinforcement learning described in the present invention, the setting of the reward function is highly free, and multi-objective optimization capability can be achieved by adjusting the setting of the reward function. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a method for optimizing mRNA stability based on reinforcement learning according to the present invention;

[0059] Figure 2 It is an optimization flow chart within the trajectory of the present invention;

[0060] Figure 3 A schematic diagram of the optimization process of a single round in the trajectory of the present invention;

[0061] Figure 4 A schematic diagram of solving an mRNA sequence that is still stable at room temperature for the reinforcement learning agent of the present invention;

[0062] Figure 5 This is the MFE distribution diagram of the offline optimization sequence of Example 1 of the present invention;

[0063] Figure 6 This is a rendering of the offline dual-objective optimization experiment of Example 2 of the present invention;

[0064] Figure 7 This is the optimization effect under variable-length mutation data of Example 3 of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0066] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0067] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 7 The detailed instructions are as follows: Specific implementation method one:

[0069] A method for optimizing mRNA stability based on reinforcement learning comprises the following steps:

[0070] S1. Extract multiple similar amino acid sequences from the protein library or use the wild-type antigen amino acid sequence to be optimized to perform amino acid substitution and deletion mutation to obtain multiple similar amino acid sequences, construct a training data set, and obtain the optimized synonymous mRNA sequence of each amino acid sequence according to the maximum nucleotide pairing method for the multiple similar amino acid sequences in the training data set, and calculate the minimum free energy MFE of the mRNA sequence for subsequent reward setting;

[0071] Furthermore, the specific implementation method of step S1 includes the following steps:

[0072] S1.1. Rules for setting nucleotide pairing function:

[0073] The paired nucleotides are one of adenine A-uracil U pairing, guanine G-cytosine C pairing, and guanine G-uracil U pairing, and there must be at least three nucleotides between the paired nucleotides. The expression is:

[0074]

[0075] Among them, δ <n i ,n j > is the nucleotide pairing function between the nucleotide at the i-th position and the nucleotide at the j-th position, n i is the nucleotide at position i, n j is the nucleotide at the jth position; it takes 1 when the nucleotide matches, otherwise it takes 0;

[0076] Each nucleotide pairs with at most one nucleotide, expressed as:

[0077]

[0078] S1.2. Based on the nucleotide pairing function of step S1, set a recursive function ψ of the number of nucleotide pairings, The maximum number of nucleotide matches between the nucleotide at the i-th position and the nucleotide at the j-th position in the RNA subsequence is used as an approximate criterion for calculating the minimum free energy. The expression is:

[0079]

[0080] Among them, N i+1 |n i Represents the selection of n i After n i+1 The feasible set, N j-1 ∩n j Represents the selection of n j After n j-1 The feasible set of , k is an integer between i and j;

[0081] S1.3. Solve the dynamic programming equation F(x) based on the recursive function of the number of nucleotide pairings in step S1.2, and the expression is:

[0082]

[0083] St{n 3i-2 , n 3i-1 ,n 3i}∈codon_set(p i ), i=1,_2,...,L

[0084] Where P is the amino acid sequence constraint, p i is the ith amino acid in the amino acid sequence, codon_set(p i ) is an optional set of codons under given acid constraints;

[0085] The optimized synonymous mRNA sequence is obtained through the dynamic programming equation, and then the minimum free energy value of the mRNA is calculated as the template MFE corresponding to the amino acid sequence data;

[0086] Furthermore, the template MFE corresponding to the amino acid sequence data is used for calculating the reward function in the subsequent agent training process.

[0087] Furthermore, the training data set is constructed by randomly extracting sequences from the user's own mutation library and then performing multiple sequence alignment. Alternatively, deletion and substitution mutations are performed based on an existing single wild-type sequence to obtain multiple similar data, and the corresponding vacant positions of the data deletion mutations are filled with empty characters. At this time, the multiple sets of sequences obtained do not need multiple sequence alignment and can be used directly; thus obtaining the training data set. The purpose of sequence alignment is to align the positions of amino acids between different sequences, which is conducive to enhancing the generalization ability of the network;

[0088] S2. Construct an agent and an environment, wherein the agent is composed of an actor network and a critic network. The actor network is composed of an input layer-hidden layer 1-hidden layer 2-actor output layer, and the critic network is composed of an input layer-hidden layer 1-hidden layer 2-critic output layer; the actor network and the critic network share the parameters of the input layer, hidden layer 1, and hidden layer 2; the environment accepts the output action of the actor network and makes corresponding modifications to the state variables, and outputs a reward function at the same time;

[0089] Furthermore, the activation function of the hidden layer 1 and the hidden layer 2 in step S2 is ReLU; the input of the actor network and the critic network is the state variable s t , the output of the actor network is the selection of amino acid codons, which is used to modify the mRNA sequence; the output of the critic network is the correction of the reward function, which is used to calculate the back-propagation gradient of the actor network. The input dimension of the actor network and the critic network is set to 32L, and the output dimension is set to 6 and 1 respectively. The corresponding network parameters are randomly initialized using the orthogonal initialization method, where L is the longest amino acid sequence length;

[0090] Furthermore, the longest sequence is used as the template chain, and its length is recorded as L. The one-hot encoding concatenation of the amino acid sequence and the mRNA sequence is taken as the state s t , since there are 20 kinds of amino acids and 4 kinds of bases, the size of the state space is set to 32L=20×L+3×4×L; the specific network structure is shown in Table 1:

[0091] Table 1 Network structure

[0092]

[0093]

[0094] S3. Collect training trajectory data, perform multiple sequence comparison on the training data set obtained in step S1, determine the size of the state space in the environment constructed in step S2 by the length of the longest amino acid sequence or the amino acid sequence of the wild-type antigen to be optimized, concatenate the one-hot encoding of the amino acid sequence in the training data set with the one-hot encoding of the mRNA sequence as a state variable, input it into the intelligent agent actor network to obtain the corresponding action output, modify the state variable using the action output and give the corresponding reward from the environment, collect the combination of {state-action-reward}, and obtain the training trajectory data set;

[0095] Furthermore, the specific implementation method of step S3 includes the following steps:

[0096] S3.1. The mRNA sequence optimization process is established as a Markov decision process. In different rounds i, the training data set P = {p1, p2, ... p k …,p M The amino acid sequence p in k For training, k = i mod M;

[0097] S3.2. After performing multiple sequence comparisons on the training data set obtained in step S1, the length L of the longest amino acid sequence or the amino acid sequence of the wild-type antigen to be optimized is determined to be 32L in the environment constructed in step S2;

[0098] S3.3. Concatenate the one-hot encoding of the amino acid sequence in the training dataset with the one-hot encoding of the mRNA sequence as the state variable s t , where the amino acid sequence is kept constant in a single round as a constraint, and the mRNA is set to an empty sequence three times the length of the protein sequence at time t0; the actor network is used to output the probability distribution p of the optimal action under a given state. θ (a t |s t ), where θ is the parameter of the actor network, and the action a at time t t The expression is:

[0099]

[0100] a t The meaning is that according to the encoding of the t-th amino acid, the corresponding codon is selected to fill in the corresponding position of the mRNA;

[0101] S3.4. Characteristic encoding is performed on the mRNA sequence formed after filling according to the nucleotides, and it is spliced ​​with the characteristic encoding of the amino acid to form the state s at time t t+1 , calculate MFE at the termination time to calculate the reward function r N, at the non-terminal moment, the reward function r t Get trajectory data for 0 {s t ,a t+1 ,r t+1 ,s t+1 |t=0,…,N-1};

[0102] Furthermore, the calculation method of the reward function in step S3.4 is:

[0103] At the end time, the minimum free energy MFE (rna seq ) size and template MFEJ template Do the difference and get the reward function, which is expressed as:

[0104] r N =J template -MFE(rna seq );

[0105] Or add the codon adaptive exponential weighting and complete the multi-objective optimization, and the expression is:

[0106]

[0107] where ω(c) is a function of the adaptability of codon c, defined as the frequency of codon c divided by the frequency of the most common synonymous codon, codon(rna seq ,i) is the mRNA sequence rna seq The i-th codon;

[0108] Furthermore, the adaptive index reflects the matching length of the synonymous codons in the coding region and the optimal codons of the target organism, and uses a number between 0 and 1 to measure the expression level of the exogenous gene in the host. The higher the value, the higher the expression of the exogenous gene.

[0109] Furthermore, in terms of implementation details, the environment selects the synonymous codon corresponding to the amino acid based on the action value output by the actor, and the setting of the mRNA modification reward function is achieved in the trajectory.

[0110] S4. Based on the training trajectory data obtained in step S3, a proximal policy optimization algorithm is used to train the learning process of the reinforcement learning agent in the environment, update the agent network parameters, and calculate the MFE of the optimization sequence;

[0111] Furthermore, the specific implementation method of step S4 is to train the intelligent agent actor network based on the training trajectory data obtained in step S3 using the proximal strategy optimization algorithm, and t ,a t+1 ,s t+1 ,rt+1 Store it in the buffer area, and calculate the loss function based on the data in the buffer area after each interaction trajectory. The expression is:

[0112]

[0113] Among them, θ old is the actor network parameter when sampling trajectory data, θ is the actor network parameter to be optimized, It is an importance sampling indicator, which is used to evaluate whether the network used for sampling is similar to the current network, and to control the optimization target based on the degree of similarity. Buffer is the buffer zone.

[0114] In each interaction round, the output of the actor network is masked to avoid illegal actions due to the different number of codons corresponding to amino acids. t ,a t+1 ,s t+1 ,r t+1 Store in the buffer area, and calculate the loss function based on the data in the buffer area after each interaction trajectory ends;

[0115] Set clip as the clipping function, the expression is:

[0116]

[0117] Where ∈ is the cropping factor, set ∈ = 0.2, and update the actor parameters θ and the critic network parameters φ by maximizing the loss function using the ADAM algorithm;

[0118] Setting the advantage function The expression is:

[0119]

[0120] The input of the actor network and the critic network is the observation state s at time t. t , the output of the actor network is action a t , the output of the critic network is the value function V φ (s t ), used to estimate the advantage function To replace the actual reward value r t ; Here γ is a fixed-size reward discount, simply set to 1, and φ is a parameter of the critic network.

[0121] Furthermore, according to the collected trajectory data, it is sequentially filled into the buffer for storage. The buffer is a queue container with a fixed size. When the data in the buffer exceeds the storage limit, the old trajectory data needs to be deleted and filled with new trajectory data. After each round, the network parameters are gradient-dropped according to the data in the buffer;

[0122] S5. Repeat steps S3 and S4 until the MFE of the optimized sequence is lower than the template MFE or the MFE of the optimized sequence decreases and tends to stabilize, completing the agent training.

[0123] Further, step S5 repeats step S3 and step S4 until the reward function output of the optimized sequence at the termination time is greater than 0, at which point the MFE of the optimized sequence is lower than the template MFE, or the variance of the MFE is less than 5 kcal / mol within 20 training rounds, at which point the MFE of the optimized sequence decreases and tends to stabilize, and the agent training is completed;

[0124] Furthermore, the optimized mRNA is constructed using the same process as the training process. At this time, there is no need to estimate the stability index after optimization. The final output at the termination time is the optimized mRNA. At this time, multiple intelligent agent models that have completed training can also be saved on disk. According to the test results of the test set, the intelligent agent with excellent performance can be selected as the model for offline optimization. Specific implementation method 2:

[0126] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for optimizing mRNA stability based on reinforcement learning when executing the computer program.

[0127] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Furthermore, the processor is used to implement the steps of the above-mentioned method for optimizing mRNA stability based on reinforcement learning when executing the computer program stored in the memory.

[0128] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0129] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Specific implementation method three:

[0131] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the mRNA stability optimization method based on reinforcement learning is implemented.

[0132] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned method for optimizing mRNA stability based on reinforcement learning can be implemented.

[0133] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0134] Example 1

[0135] In this embodiment, the LinearDesign method and the present optimization method (i.e., the reinforcement learning mRNA optimization algorithm) are used to optimize the mRNA under the same protein sequence constraint data set, and the stability distribution of the trained intelligent agent under offline optimization is obtained respectively.

[0136] An amino acid sequence with 50 amino acids was selected as the wild-type protein-encoding constraint. A substitution mutation was used on the sequence to generate 10 mutant sequences with 2, 3, and 4 mutant amino acids as mixed training set 1, and 10 mutant sequences with 1, 3, 5, 7, 9, and 11 mutant amino acids as mixed training set 2. The same method was used to generate 20 mutant sequences with 1, 3, 5, 7, 9, and 11 mutant amino acids as test sets. There was no overlap between the training set and the test set. In addition, in order to compare the ability improvement brought by using multiple sequences for training, it is also considered to train an agent using only the wild-type protein-encoding constraint. The minimum free energy is used as the evaluation indicator of stability. The operating results of the two methods are shown in the figure below. Figure 5 As shown in the figure, it can be seen that using only a single wild-type protein constraint has a certain optimization ability, but the generalization performance is relatively poor. As the training set increases, the generalization performance of the model gradually increases and has the same optimization ability as LinearDesign.

[0137] Example 2

[0138] This example is used to test the dual-objective optimization capability of the method of the present invention, and the optimization objective function is the combination of the codon adaptive index and the minimum free energy. The settings of the training set and the test set are consistent with the settings of the training set 1 used in Example 1. The optimization effects of the test set and the training set are shown in Figure 1. Figure 6As shown, it can be seen that the reinforcement learning method has multi-objective optimization capabilities, can coordinately optimize the minimum free energy index and the adaptive index index, and its optimization ability is also better than the LinearDesign method.

[0139] Example 3

[0140] This example is used to test the optimization ability of the method of the present invention under variable-length tasks, by optimizing the objective function and only considering the minimum free energy. An amino acid sequence with 50 amino acids is selected as the protein encoding constraint, and 1-10 deletion mutations are made to the sequence to obtain 10 amino acid sequence data of unequal lengths. For each sequence, 10 mutant sequences with one to three mutant amino acids are generated as training sets. The setting of the test set is consistent with that of the training set. Figure 7 It can be seen that the reinforcement learning method also has the ability to optimize variable-length sequences, and the effect is better than random generation, but there is still a certain gap from the LinearDesign method.

[0141] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0142] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for optimizing mRNA stability based on reinforcement learning, characterized in that: The steps include: S1. Extract multiple similar amino acid sequences from the protein library or use the wild-type antigen amino acid sequence to be optimized to perform amino acid substitution and deletion mutation to obtain multiple similar amino acid sequences, construct a training data set, and obtain the optimized synonymous mRNA sequence of each amino acid sequence according to the maximum nucleotide pairing method for the multiple similar amino acid sequences in the training data set, and calculate the minimum free energy MFE of the mRNA sequence for subsequent reward setting; The specific implementation method of step S1 includes the following steps: S1.

1. Rules for setting nucleotide pairing function: The paired nucleotides are one of adenine A-uracil U pairing, guanine G-cytosine C pairing, and guanine G-uracil U pairing, and there must be at least three nucleotides between the paired nucleotides. The expression is: Among them, δ <n i , n j > is the nucleotide pairing function between the nucleotide at the i-th position and the nucleotide at the j-th position, n i is the nucleotide at position i, n j is the nucleotide at the jth position; it takes 1 when the nucleotide matches, otherwise it takes 0; Each nucleotide pairs with at most one nucleotide, expressed as: S1.

2. Based on the nucleotide pairing function of step S1, set a recursive function ψ of the number of nucleotide pairings, The maximum number of nucleotide matches between the nucleotide at the i-th position and the nucleotide at the j-th position in the RNA subsequence is used as an approximate criterion for calculating the minimum free energy. The expression is: Among them, N i+1 |n i Represents the selection of n i After n i+1 The feasible set, N j-1 ∩n j Represents the selection of n j After n j-1 The feasible set of , k is an integer between i and j; S1.

3. Solve the dynamic programming equation F(x) based on the recursive function of the number of nucleotide pairings in step S1.2, and the expression is: s.t.{n 3i-2 ,n 3i-1 ,n 3i }∈codon_set(p i ),i=1,2,…,L Where P is the amino acid sequence constraint, p i is the ith amino acid in the amino acid sequence, codon_set(p i ) is an optional set of codons under given acid constraints; The optimized synonymous mRNA sequence is obtained through the dynamic programming equation, and then the minimum free energy value of the mRNA is calculated as the template MFE corresponding to the amino acid sequence data; S2. Construct an agent and an environment, wherein the agent is composed of an actor network and a critic network. The actor network is composed of an input layer-hidden layer 1-hidden layer 2-actor output layer, and the critic network is composed of an input layer-hidden layer 1-hidden layer 2-critic output layer; the actor network and the critic network share the parameters of the input layer, hidden layer 1, and hidden layer 2; the environment accepts the output action of the actor network and makes corresponding modifications to the state variables, and outputs a reward function at the same time; S3. Collect training trajectory data, perform multiple sequence comparison on the training data set obtained in step S1, determine the size of the state space in the environment constructed in step S2 by the length of the longest amino acid sequence or the amino acid sequence of the wild-type antigen to be optimized, concatenate the one-hot encoding of the amino acid sequence in the training data set with the one-hot encoding of the mRNA sequence as a state variable, input it into the intelligent agent actor network to obtain the corresponding action output, modify the state variable using the action output and give the corresponding reward from the environment, collect the combination of {state-action-reward}, and obtain the training trajectory data set; S4. Based on the training trajectory data obtained in step S3, a proximal policy optimization algorithm is used to train the learning process of the reinforcement learning agent in the environment, update the agent network parameters, and calculate the MFE of the optimization sequence; S5. Repeat steps S3 and S4 until the MFE of the optimized sequence is lower than the template MFE or the MFE of the optimized sequence decreases and tends to stabilize, completing the agent training.

2. The method for optimizing mRNA stability based on reinforcement learning according to claim 1, characterized in that: The activation function of hidden layer 1 and hidden layer 2 in step S2 is ReLU; the input of the actor network and the critic network is the state variable s t The output of the actor network is the selection of amino acid codons, which is used to modify the mRNA sequence; the output of the critic network is the correction of the reward function, which is used to calculate the back-propagation gradient of the actor network. The input dimension of the actor network and the critic network is set to 32L, and the output dimension is set to 6 and 1 respectively. The corresponding network parameters are randomly initialized using the orthogonal initialization method, where L is the longest amino acid sequence length.

3. The method for optimizing mRNA stability based on reinforcement learning according to claim 2, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. The mRNA sequence optimization process is established as a Markov decision process. In different rounds i, the training data set P = {p1, p2, ... p k ..., p M The amino acid sequence p in k For training, k = i mod M; S3.

2. After performing multiple sequence comparisons on the training data set obtained in step S1, the length L of the longest amino acid sequence or the amino acid sequence of the wild-type antigen to be optimized is determined to be 32L in the state space in the environment constructed in step S2; S3.

3. Concatenate the one-hot encoding of the amino acid sequence in the training dataset with the one-hot encoding of the mRNA sequence as the state variable s t , where the amino acid sequence is kept constant in a single round as a constraint, and the mRNA is set to an empty sequence three times the length of the protein sequence at time t0; the actor network is used to output the probability distribution p of the optimal action under a given state. θ (a t |s t ), where θ is the parameter of the actor network, and the action a at time t t The expression is: a t The meaning is that according to the encoding of the t-th amino acid, the corresponding codon is selected to fill in the corresponding position of the mRNA; S3.

4. Characteristic encoding is performed on the mRNA sequence formed after filling according to the nucleotides, and it is spliced ​​with the characteristic encoding of the amino acid to form the state s at time t t+1 , calculate MFE at the termination time to calculate the reward function r N , at the non-terminal moment, the reward function r t Get trajectory data for 0 {s t , a t+1 , r t+1 ,s t+1 |t=0,...,N-1}.

4. The method for optimizing mRNA stability based on reinforcement learning according to claim 3, characterized in that: The reward function of step S3.4 is calculated as: At the end time, the minimum free energy MFE (rna seq ) size and template MFEJ template Do the difference and get the reward function, which is expressed as: r N =J template -MFE(rna seq ); Or add the codon adaptive exponential weighting and complete the multi-objective optimization, and the expression is: where ω(c) is a function of the adaptability of codon c, defined as the frequency of codon c divided by the frequency of the most common synonymous codon, codon(rna seq , i) is the mRNA sequence rna seq The i-th codon.

5. The method for optimizing mRNA stability based on reinforcement learning according to claim 4, characterized in that: The specific implementation method of step S4 is to train the intelligent agent actor network based on the training trajectory data obtained in step S3 using the proximal strategy optimization algorithm. t , a t+1 ,s t+1 , r t+1 Store it in the buffer area, and calculate the loss function based on the data in the buffer area after each interaction trajectory. The expression is: Among them, θ old is the actor network parameter when sampling trajectory data, θ is the actor network parameter to be optimized, It is an importance sampling indicator, which is used to evaluate whether the network used for sampling is similar to the current network, and to control the optimization target based on the degree of similarity. Buffer is the buffer zone. Set clip as the clipping function, the expression is: Where ∈ is the cropping factor, set ∈ = 0.2, and update the actor parameters θ and the critic network parameters Φ by maximizing the loss function using the ADAM algorithm; Setting the advantage function The expression is: The input of the actor network and the critic network is the observation state s at time t. t , the output of the actor network is action a t , the output of the critic network is the value function V Φ (s t ), used to estimate the advantage function To replace the actual reward value r t , γ is a fixed reward discount.

6. The method for optimizing mRNA stability based on reinforcement learning according to claim 5, characterized in that: Step S5 repeats steps S3 and S4 until the reward function output of the optimized sequence at the termination time is greater than 0. At this time, the MFE of the optimized sequence is lower than the template MFE, or the variance of the MFE is less than 5 kcal / mol within 20 training rounds. At this time, the MFE of the optimized sequence decreases and stabilizes, and the agent training is completed.

7. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for optimizing mRNA stability based on reinforcement learning as described in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mRNA stability optimization method based on reinforcement learning according to any one of claims 1 to 6 is implemented.

Images