Training method and device of protein sequence generation model based on reinforcement learning

Abstract

The application provides a training method and device of a protein sequence generation model based on reinforcement learning, including an offline calculation stage and an online optimization stage. In the offline calculation stage, the method comprises: obtaining a pre-trained policy model, wherein the policy model is used to generate a protein sequence in reverse based on random noise; constructing a first data set based on the protein sequence generated by the policy model and the quantum chemical calculation result of the protein sequence, the first data set being used for training of a reward model; training an initial reward model based on the first data set to obtain a reward model; and in the online optimization stage, the method comprises: taking the score output by the reward model as a reward signal, optimizing the parameters of the policy model based on the reward signal to obtain a protein sequence generation model. The application improves the efficiency and accuracy of generating proteins.

Description

Technical Field

[0001] This application relates to a training method and device for a protein sequence generation model based on reinforcement learning. Background Technology

[0002] Protein design is a key technology in fields such as synthetic biology, drug development, and industrial biocatalysis. Its core objective is to obtain protein molecules with specific functions by modifying existing amino acid sequences or constructing amino acid sequences from scratch.

[0003] Traditional protein design methods typically rely on directed evolution techniques, requiring the construction of large-scale mutant libraries for high-throughput screening and multiple rounds of experimental iterations to obtain optimized protein variants. The rise of artificial intelligence has brought revolutionary breakthroughs to protein design methods. Deep learning-based sequence generation models (such as ProteinMPNN and ESM-InverseFold) can learn the sequence-structure-function mapping relationships of natural proteins, enabling rational design. Diffusion models are also being applied in protein design, improving the quality and diversity of generated sequences through the noise addition and denoising processes of discretized amino acid sequences.

[0004] However, existing artificial intelligence methods still face key challenges. For example, it is difficult and costly to obtain experimentally determined activity data, while physics-based computational chemistry evaluation methods (such as molecular dynamics simulations) are highly accurate but can take several hours for a single calculation, making them unsuitable for direct embedding into the training of artificial intelligence models. Summary of the Invention

[0005] Based on this, this application provides a training method and device for a protein sequence generation model based on reinforcement learning, so as to improve the efficiency and accuracy of protein sequence generation.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a method for training a protein sequence generation model, the method comprising: Obtain a pre-trained policy model, which is used to reverse generate protein sequences based on random noise; Based on the protein sequences generated by the policy model and the quantum chemical calculation results for the protein sequences, the first dataset was constructed and used to train the reward model. Based on the first dataset, the initial reward model is trained to obtain the reward model; And in the online optimization phase, the methods include: The score output by the reward model is used as a reward signal, and the parameters of the policy model are optimized based on the reward signal to obtain a protein sequence generation model.

[0007] Optionally, the score output by the reward model is used as a reward signal, and the parameters of the policy model are optimized based on the reward signal to obtain a protein sequence generation model, including: The reward model is used to score each protein sequence output by the strategy model to obtain a reward signal; Based on the policy gradient algorithm and reward signal, the parameters of the policy model are adjusted to maximize the expected reward of the protein sequence generated by the policy model. And when the preset conditions are met during the online optimization phase, the latest strategy model is used as the protein sequence generation model.

[0008] Optionally, the method further includes: Add a divergence constraint to the policy model, where the divergence constraint is used to balance maximizing the expected reward with the rationality of the protein sequence generated by the policy model.

[0009] Optionally, the preset conditions include any one of the following: the number of optimization rounds reaches a preset number of rounds, the expected reward reaches a preset threshold, and the expected reward is in a convergent state.

[0010] Optionally, based on the protein sequences generated by the strategy model and the quantum chemical calculation results for the protein sequences, a first dataset is constructed, including: Multiple candidate protein sequences were generated using a strategy model; According to the preset configuration method, multiple candidate protein sequences are paired to obtain multiple candidate protein sequence pairs; For each of the multiple candidate protein sequence pairs, quantum chemical calculations are performed to obtain the quantum chemical calculation results for each candidate protein sequence pair. Based on the quantum chemical calculation results of each pair of candidate protein sequences, a preference label is determined for each pair of candidate protein sequences. The preference label is used to indicate the chemical performance of the two protein sequences in each pair of candidate protein sequences. Each pair of candidate protein sequence pairs and its corresponding preference label are grouped together to form the first dataset.

[0011] Optionally, based on the first dataset, an initial reward model is trained to obtain a reward model, including: Each candidate protein sequence pair was scored using an initial reward model; The loss function of the initial reward model is trained and optimized based on the score and preference label of each candidate protein sequence pair. When the difference between the score and the preference label of the initial reward model for each candidate protein sequence pair is less than a preset threshold, training of the initial reward model is stopped, and the reward model is obtained.

[0012] Optionally, a pre-trained policy model is obtained, including: Construct an initial strategy model based on a diffusion model; The initial policy model is trained using natural protein sequences from a natural protein sequence database to obtain a pre-trained policy model.

[0013] Optionally, the initial policy model is trained based on natural protein sequences from a natural protein sequence database to obtain a pre-trained policy model, including: The natural protein sequence is subjected to forward noise processing based on a preset noise to obtain a noisy protein sequence. By using noisy protein sequences and natural protein sequences, the denoising network in the initial policy model is trained so that the pre-trained policy model has the ability to reverse generate protein sequences from random noise.

[0014] Secondly, another embodiment of this application provides a protein sequence generation method, the method comprising: Obtain random noise; Based on random noise, a protein sequence is generated using a protein sequence generation model, wherein the protein sequence generation model is a protein sequence generation model trained using any of the reinforcement learning-based protein sequence generation model training methods described in the first aspect above.

[0015] Thirdly, another embodiment of this application provides a protein sequence generation model training device based on reinforcement learning, the device comprising: The first acquisition module is used to acquire a pre-trained policy model, wherein the policy model is used to reverse generate protein sequences based on random noise; The building block is used to construct the first dataset based on the protein sequences generated by the policy model and the quantum chemical calculation results for the protein sequences. The first dataset is used to train the reward model. The training module is used to train the initial reward model based on the first dataset to obtain the reward model; The optimization module is used to take the score output by the reward model as a reward signal and optimize the parameters of the strategy model based on the reward signal to obtain a protein sequence generation model.

[0016] Fourthly, another embodiment of this application provides a protein sequence generation apparatus, the apparatus comprising: The second acquisition module is used to acquire random noise; The generation module is used to generate protein sequences based on random noise using a protein sequence generation model, wherein the protein sequence generation model is a protein sequence generation model trained using any of the reinforcement learning-based protein sequence generation model training methods described in the first aspect above.

[0017] Fifthly, another embodiment of this application provides a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform any of the methods described in the first and second aspects above.

[0018] In a sixth aspect, another embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs any of the methods described in the first and second aspects above.

[0019] In a seventh aspect, another embodiment of this application provides a computer program product, which includes computer program code that, when executed by a computing device, performs any of the methods described in the first and second aspects above.

[0020] The beneficial effects of this application are: This application provides a training method and apparatus for a protein sequence generation model based on reinforcement learning. The method involves acquiring a pre-trained policy model offline, constructing a first dataset based on the protein sequences generated by the policy model and the quantum chemical calculation results for those sequences, and then training an initial reward model using this dataset to obtain a reward model. In the online optimization phase, the score output by the reward model is used as a reward signal, and the parameters of the policy model are optimized based on this signal to obtain the protein sequence generation model. This application avoids the time cost of repeatedly using quantum computing resources during reinforcement learning and avoids dependence on high-cost experimental data, ensuring that model optimization is always anchored to the physicochemical mechanisms at the real molecular level. The segmented training method, combining offline and online training, ensures the scientific validity and consistency of the reward signal, improves the training accuracy of the generated proteins, and reduces the training cost of the protein sequence generation model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a protein sequence generation model training method based on reinforcement learning, provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the process of determining the protein sequence generation model in a reinforcement learning-based protein sequence generation model training method provided in this application embodiment; Figure 3 A schematic diagram illustrating the process of determining the first dataset in a protein sequence generation model training method based on reinforcement learning provided in this application embodiment; Figure 4 A schematic diagram of the quantum Gaussian boson sampling method in a protein sequence generation model training method based on reinforcement learning provided in an embodiment of this application; Figure 5 A flowchart illustrating the process of determining the reward model in a protein sequence generation model training method based on reinforcement learning, as provided in an embodiment of this application. Figure 6 A flowchart illustrating the process of determining a pre-trained policy model in a reinforcement learning-based protein sequence generation model training method provided in this application embodiment; Figure 7 A flowchart illustrating the process of determining the pre-trained policy model in another reinforcement learning-based protein sequence generation model training method provided in this application embodiment; Figure 8 A schematic flowchart of a protein sequence generation method provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of a protein sequence generation model training device based on reinforcement learning provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a protein sequence generation device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and operations S without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0024] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0025] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0026] Currently, protein design is a core technology in synthetic biology, drug development, and industrial biocatalysis. Its core objective is to obtain protein molecules with specific structures and functions by modifying amino acid sequences or designing them de novo. Existing methods typically train generative models based on large-scale natural protein sequence databases, enabling the models to learn co-evolutionary patterns and structural constraints between amino acids, thereby achieving intelligent generation of rational sequences. However, the feedback signals relied upon for training in existing technologies mostly come from experimental testing, where acquiring experimental data is difficult and time-consuming, making it hard to meet the data volume requirements for large-scale model training. Therefore, this application provides a reinforcement learning-based protein sequence generation model training method, including an offline computation stage and an online optimization stage. In the offline computation stage, a pre-trained policy model is acquired. Based on the protein sequences generated by the policy model and the quantum chemical calculation results for the protein sequences, a first dataset is constructed. This first dataset is used to train a reward model. Based on the first dataset, an initial reward model is trained to obtain a reward model. In the online optimization stage, the method includes: using the score output by the reward model as a reward signal, and optimizing the parameters of the policy model based on the reward signal to obtain a protein sequence generation model. This application trains the protein sequence generation model by combining an offline computation stage and an online optimization stage, thereby improving the training efficiency of the protein sequence generation model, reducing training complexity, and increasing the accuracy of generated protein sequences.

[0027] To clearly describe the method provided in the embodiments of this application, the method will be described below in conjunction with the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a reinforcement learning-based protein sequence generation model training method provided in an embodiment of this application. The reinforcement learning-based protein sequence generation model training method includes an offline computation stage and an online optimization stage. For example... Figure 1 As shown, the offline computation phase of this method includes: S101. Obtain the pre-trained policy model.

[0029] The policy model in this application is used to inversely generate protein sequences based on random noise. The policy model can also be referred to as a policy network. The policy model can be implemented using a denoising network built on a neural network. Random noise refers to unstructured input signals sampled in the latent space of the generative model. Random noise is a controllable perturbation source with a well-defined mathematical definition and functional purpose. For example, random noise can be a latent vector sampled from a standard normal distribution, a highly noisy sequence of terminal states, or an embedding vector injected with Gaussian noise. Optionally, random noise can be implemented in the form of a sequence. Furthermore, the protein sequence in this application refers to a protein composed of multiple amino acids arranged in a specific order, not a sequence composed of multiple proteins. The protein sequence can also be understood as the amino acid sequence of a protein.

[0030] In this embodiment, the pre-trained policy model has the ability to progressively generate reasonable protein sequences from random noise, and the policy model has learned the evolutionary rationality and physical feasibility of the protein sequences, which can provide high-quality initialization and search space constraints for subsequent optimization. For example, the input of the pre-trained policy model is random noise, and the output is a protein sequence or a pair of protein sequences (i.e., a pair of protein sequences).

[0031] Optionally, the neural network can be trained using multiple random noises and their corresponding protein sequences to obtain a pre-trained policy model. For details on how to obtain the pre-trained policy model, please refer to [link to relevant documentation]. Figure 6 and Figure 7 And related textual descriptions, which will not be elaborated here. In another alternative implementation, the pre-trained policy model can also be directly obtained, that is, any policy model that already has the ability to progressively reverse-engineer reasonable protein sequences from random noise can be used as the pre-trained policy model in this application.

[0032] S102. Based on the protein sequences generated by the policy model and the quantum chemical calculation results for the protein sequences, the first dataset is constructed and used to train the reward model.

[0033] The quantum chemical calculation results of protein sequences are physicochemical indices obtained from principle calculations of key regions of the protein sequence. These quantum chemical calculation results reflect the functional potential of the protein sequence at the molecular level. Optionally, the quantum chemical calculation results can be achieved using methods such as quantum variable quantum eigenvalue solver algorithms, quantum machine learning algorithms, and quantum Gaussian boson sampling. The quantum chemical calculation results (also known as quantum chemical evaluation results) can be chemical parameters such as reaction energy barriers, charge transfer efficiency, enzyme-catalyzed reaction pathways, acid dissociation constants, and frontier orbital gaps.

[0034] Optionally, methods such as quantum variable quantum eigenvalue solver algorithms, quantum machine learning algorithms, and quantum Gaussian boson sampling can be used to calculate the protein sequences generated by the strategy model in order to obtain the quantum chemical calculation results corresponding to the protein sequences.

[0035] Then, a first dataset is constructed based on multiple protein sequences and the quantum chemical calculation results for those sequences. Each set of data in the first dataset of this application includes a protein sequence pair and the corresponding quantum chemical calculation result. For example, a set of data includes a protein sequence pair (A, B) and the corresponding quantum chemical calculation result for the protein sequence pair (A, B), where A and B are both protein sequences.

[0036] S103. Based on the first dataset, train the initial reward model to obtain the reward model.

[0037] The reward model in this application is a parameterized neural network, and therefore can also be referred to as a reward network. After training, the input to the reward model is a protein sequence, and the output is a score, which reflects the overall quality of the protein sequence. For example, catalytic activity can be used to evaluate a protein sequence, and the overall quality score in this case represents the catalytic efficacy of the protein sequence.

[0038] See also Figure 1 ,like Figure 1 The reinforcement learning-based protein sequence generation model training method shown includes the following steps in the online optimization phase: S104. Use the score output by the reward model as a reward signal, and optimize the parameters of the strategy model based on the reward signal to obtain a protein sequence generation model.

[0039] The policy model in this application can denoise random noise and generate corresponding protein sequences from it. The reward model can score the protein sequences, and the score output by the reward model is used as a reward signal. The reward signal is then used as a supervision signal to optimize the parameters of the policy model, thereby obtaining a protein sequence generation model.

[0040] In this embodiment, a pre-trained policy model is obtained. Based on the protein sequences generated by the policy model and the quantum chemical calculation results for the protein sequences, a first dataset is constructed. Based on the first dataset, an initial reward model is trained to obtain a reward model. The score output by the reward model is used as a reward signal, and the parameters of the policy model are optimized based on the reward signal to obtain a protein sequence generation model. This application obtains a reward model by training the initial reward model through quantum chemical calculations in the offline computing phase. This avoids the time cost of repeatedly calling quantum computing resources during reinforcement learning and avoids dependence on high-cost experimental data, ensuring that model optimization is always anchored to the physicochemical mechanisms at the real molecular level. Through the segmented training method of offline and online training, the scientific validity and consistency of the reward signal are ensured, maintaining the accuracy advantage of quantum computing while meeting the efficiency requirements of reinforcement learning in evaluating large amounts of data. This improves the training accuracy of the protein sequence generation model and reduces the training cost of the protein sequence generation model.

[0041] Figure 2 This is a schematic diagram illustrating the process of obtaining a protein sequence generation model in the reinforcement learning-based protein sequence generation model training method provided in this application embodiment. Figure 2 As shown, the score output by the reward model is used as the reward signal, and the parameters of the policy model are optimized based on the reward signal to obtain a protein sequence generation model, including the following steps: S201. Use the reward model to score each protein sequence output by the strategy model to obtain the reward signal.

[0042] For example, a reward model can be used to score each protein sequence output by the strategy model, resulting in a comprehensive quality score for each protein sequence, which can then be used as a reward signal. Similarly, a reward model can be used to score each protein sequence output by the strategy model, resulting in a catalytic activity score for each protein sequence, which can then be used as a reward signal.

[0043] S202. Based on the policy gradient algorithm and reward signal, the parameters of the policy model are adjusted to maximize the expected reward of the protein sequence generated by the policy model.

[0044] For example, the policy gradient algorithm can be either Proximal Policy Optimization (PPO) or Monte Carlo Policy Gradient Algorithm (REINFORCE).

[0045] In this embodiment, the reward signal is used as the supervision signal of the policy model. The parameters of the policy model are adjusted based on the policy gradient algorithm to maximize the expected reward of the protein sequence generated by the policy model as the optimization objective.

[0046] S203. When the preset conditions are met during the online optimization phase, the latest strategy model is used as the protein sequence generation model.

[0047] In this embodiment, the preset condition can be at least one of the following: optimization round number condition, reward expectation condition, protein sequence pass rate, etc. The optimization round number condition is, for example, reaching a preset number of optimizations. The reward expectation condition is, for example, [missing information]. The protein sequence pass rate is, for example, a protein sequence pass rate exceeding a preset pass rate, such as 98%.

[0048] Optionally, during the online optimization phase, the latest strategy model can be used as the protein sequence generation model if any preset condition is met.

[0049] In this embodiment, a reward signal is obtained by providing a precise score to each protein sequence generated by the strategy model through a reward model. Then, the strategy model parameters are adjusted based on the reward signal using a strategy gradient algorithm to maximize the expected reward of the generated protein sequence. This enables targeted optimization of protein sequence generation in a data-driven manner, without relying on complex manually designed rules. It guides the strategy model to continuously generate protein sequences that better meet expectations. At the same time, the strategy gradient algorithm can adapt to the discrete space optimization problem of protein sequence generation, improving the quality of the generated sequence while having strong scalability.

[0050] Based on the above embodiments, this application provides a protein sequence generation model training method based on reinforcement learning, which further includes: Add divergence constraints to the strategy model.

[0051] Divergence constraints are used to balance maximizing the expected reward with the plausibility of the protein sequences generated by the policy model. For example, a divergence constraint is a constraint on the deviation between the parameters of the policy model and its reference parameters during parameter tuning. Accordingly, the expected reward includes both the reward signal and the divergence constraint.

[0052] For example, strategy model Generate the corresponding protein sequence based on random noise. Using a reward model protein sequence An evaluation (e.g., scoring) is performed to obtain the reward parameter of the protein sequence corresponding to the random noise. The reward parameters of the protein sequence. As a strategy model The supervisory signal. Based on the policy model. The reference strategy model corresponding to the strategy model Reward parameters The loss function for constructing the pre-policy model:

[0053] in, The loss function of the policy model, For the parameters of the strategy model, For strategy model The expected reward for generating the sequence. To reward the model for protein sequences The reward parameters, For the coefficients or penalty weights of the KL divergence term, To measure the pre-trained policy network With reference policy network The difference between them is the divergence constraint.

[0054] Optionally, the loss function of the policy model. Minimize as the objective, determine the parameters of the strategy model. According to the parameters of the strategy model The policy model was optimized based on the policy gradient algorithm to obtain a protein sequence generation model.

[0055] In this embodiment, a divergence constraint is added to the policy model to prevent large changes in parameters when the policy model is updated based on the reward signal. The divergence constraint can also provide stable guidance for model updates, accelerate model convergence, and improve the robustness of the training process.

[0056] Optionally, the preset conditions include any one of the following: the number of optimization rounds reaches a preset number of rounds, the expected reward reaches a preset threshold, and the expected reward is in a convergent state.

[0057] Optionally, if the number of optimization rounds in the online optimization phase reaches a preset number, the latest strategy model is used as the protein sequence generation model. The preset number of rounds is determined based on the training scale of the current protein sequence generation model and the complexity of the training data. If the training scale is large and the complexity of the training data is high, the preset number of rounds is larger; otherwise, the preset number of rounds is smaller.

[0058] Optionally, if the expected reward during the online optimization phase reaches a preset threshold, the latest strategy model is used as the protein sequence generation model. The expected reward is determined by the reward signal and divergence constraints, and the preset threshold is used to determine whether the expected reward meets the conditions, based on the aforementioned formula for calculating the expected reward.

[0059] Optionally, if the expected reward during the online optimization phase is in a convergent state, the latest policy model is used as the protein sequence generation model. For example, when the expected reward does not show a significant increase in multiple consecutive iterations, has minimal fluctuations, and remains stable within a specific range, it can be determined that the expected reward has reached a convergent state.

[0060] In this embodiment, any one of the following is used as a preset termination condition: the number of optimization rounds reaches a preset number of rounds, the expected reward reaches a preset threshold, or the expected reward is in a convergent state. This constructs a multi-dimensional and robust optimization termination judgment mechanism. By setting the number of rounds, the model is prevented from getting stuck in infinite iterations due to abnormal situations. The preset expected reward threshold quickly locks in the strategy model that meets the basic performance requirements, reducing unnecessary subsequent iterations. The convergence state of the expected reward is used to accurately capture the node where the model performance has reached a stable and optimal state, avoiding the waste of computing resources and the risk of overfitting caused by overtraining, and improving the practicality of the entire protein sequence generation model training process.

[0061] Figure 3 This is a schematic diagram illustrating the process of determining the first dataset in a reinforcement learning-based protein sequence generation model training method provided in an embodiment of this application. Figure 3 As shown, the first dataset is determined, including: S301. Use a strategy model to generate multiple candidate protein sequences.

[0062] In this embodiment, the candidate protein sequence is a novel amino acid sequence generated by the strategy model based on random noise.

[0063] Optionally, a strategy model is used to remove the noise sequences in the random noise based on the relationship between the protein sequences and noise sequences in multiple random noise sequences, thereby generating multiple candidate protein sequences.

[0064] S302. According to the preset configuration method, multiple candidate protein sequences are paired to obtain multiple candidate protein sequence pairs.

[0065] Optionally, the preset configuration method can be random sampling or cluster sampling. Random sampling is a method of randomly selecting pairs of sequences from all candidate sequences with uniform probability to form sequence pairs. Cluster sampling is a method of first grouping candidate sequences according to a certain similarity index, and then selecting sequences between different clusters or within the same cluster to form pairs.

[0066] Optionally, the number of sequence pairs to be generated is determined based on the number of multiple candidate protein sequences, thereby randomly sampling multiple candidate protein sequences to obtain multiple protein sequence pairs.

[0067] For example, the number of candidate protein sequences is 100, and the number of sequence pairs generated is 50, so that the multiple candidate protein sequences are randomly sampled to obtain 50 protein sequence pairs.

[0068] Optionally, multiple candidate protein sequences are converted into vector values ​​of fixed dimensions, and similarity is calculated based on the vector data of multiple candidate protein sequences to obtain multiple clusters corresponding to multiple candidate protein sequences. Then, multiple candidate protein sequences in multiple clusters are matched pairwise to obtain multiple protein sequence pairs.

[0069] S303. Perform quantum chemical calculations on each of the multiple candidate protein sequence pairs to obtain the quantum chemical calculation results for each candidate protein sequence pair.

[0070] Quantum chemical calculation methods can include quantum variable quantum eigenvalue solver algorithms, quantum machine learning algorithms, or quantum Gaussian Bose sampling. Quantum chemical evaluation results are physicochemical indices obtained from principle calculations of key regions of protein sequences, reflecting their functional potential at the molecular level. The corresponding quantum chemical evaluation results obtained using quantum variable quantum eigenvalue solver algorithms, quantum machine learning algorithms, and quantum Gaussian Bose sampling can include chemical parameters such as reaction energy barriers, charge transfer efficiency, enzyme-catalyzed reaction pathways, acid dissociation constants, and frontier orbital gaps.

[0071] Optionally, a quantum variational quantum eigenvalue solver algorithm is used to calculate the molecular physical quantities corresponding to each protein sequence pair, such as ground state energy, excited state energy, molecular orbital energy and layout, or electronic density distribution. A quantum machine learning algorithm is used to calculate the molecular physical quantities corresponding to each protein sequence pair, such as energy, electronic properties, and spectral properties. A quantum Gaussian boson sampling method is used to calculate the molecular physical quantities corresponding to each protein sequence pair, such as vibrational spectral characteristics and molecular vibrational coupling strength.

[0072] For example, Figure 4 This is a schematic diagram of the quantum Gaussian boson sampling method in a protein sequence generation model training method based on reinforcement learning provided in an embodiment of this application.

[0073] like Figure 4As shown, the quantum Gaussian boson sampling method includes: squeezed state preparation, a linear interferometer, and photon detection. Squeezed state preparation converts the incident laser into a multimode squeezed light field, providing photon input with quantum properties for subsequent interference. The linear interferometer includes a beam splitter and a phase shifter. After the squeezed photons enter the linear interferometer, they undergo multiple quantum interferences at the beam splitter, with the path controlled by the phase shifter. The entire process is equivalent to a unitary matrix transformation of the photon state, thus generating a complex quantum superposition state. Photon detection is used to capture the photon number distribution at the output. The photon detection records the number of photons at each output port; the combination of these photon numbers constitutes a sampling result, and the statistical distribution of a large number of sampling results is the output of Gaussian boson sampling.

[0074] S304. Based on the quantum chemical calculation results of each pair of candidate protein sequence pairs, determine the preference label corresponding to each pair of candidate protein sequence pairs.

[0075] Preference labels are used to indicate the relative chemical properties of the two protein sequences in each candidate protein sequence pair. Preference labels are binary preference labels. Representing candidate sequences Molecular physical quantities are greater than candidate sequences Molecular physical quantities, preference labels Representing candidate sequences Molecular physical quantities are less than candidate sequences Molecular physical quantities.

[0076] Optionally, based on the quantum chemical calculation results of each pair of candidate protein sequence pairs, a binary preference label corresponding to each pair of candidate protein sequence pairs is determined.

[0077] For example, the molecular physical quantity is catalytic activity, if the candidate sequence The reaction energy barrier is lower than that of the candidate sequence. Right now Then the candidate sequence If the catalytic activity is higher, it is marked as a preferred catalytic type. And preferred labels .

[0078] S305. Combine each pair of candidate protein sequence pairs and its corresponding preference label into a set of data to form the first dataset.

[0079] Optionally, based on protein sequence preference tags and each pair of candidate protein sequences This yields the protein sequence preference dataset, which is also known as the first dataset. As a result of quantum chemical evaluation, among which, For the i-th protein sequence pair, Let be the preference label for the i-th protein sequence pair. This represents the number of protein sequence pairs.

[0080] In this embodiment, a protein sequence preference dataset is constructed by calculating molecular physical quantities for each protein sequence pair, which avoids the scaling bias and model dependency problems common in absolute scoring and improves the generalization ability and robustness of the reward network.

[0081] Figure 5 This is a schematic diagram illustrating the process of determining the reward model in a reinforcement learning-based protein sequence generation model training method provided in an embodiment of this application. Figure 5 As shown, based on the first dataset, the initial reward model is trained to obtain the reward model, including: S501. Use the initial reward model to score each candidate protein sequence pair.

[0082] Optionally, an initial reward model is used to score each candidate protein sequence in each pair of candidate protein sequences to obtain a comprehensive quality score for each candidate protein sequence.

[0083] S502. Based on the score and preference label of each candidate protein sequence pair, the loss function of the initial reward model is trained and optimized.

[0084] Optionally, an initial reward model is used, based on a protein sequence preference dataset. Each pair of candidate protein sequences Generate reward parameters for each protein sequence pair. and Based on preference labels in the protein sequence preference dataset Each pair of candidate protein sequences The generation and corresponding reward parameters of protein sequence pairs. and The loss function for constructing the initial reward model. This is used to train and optimize the loss function of the initial reward model.

[0085]

[0086] in, The loss function for the initial reward model. These are the parameters for the initial reward model. For preference dataset Each candidate protein sequence pair was sampled. The average loss, For the preference dataset, For preference tags, The sigmoid function, also known as the logical function. To reward the network for candidate sequences Evaluation parameters, To reward the network for candidate sequences Evaluation parameters.

[0087] S503. When the difference between the score and the preference label of the initial reward model for each candidate protein sequence pair is less than a preset threshold, stop training the initial reward model and obtain the reward model.

[0088] In this embodiment, the preset threshold can be determined based on the scoring accuracy. If the scoring accuracy requirement is high, the preset threshold is small; if the scoring accuracy requirement is low, the preset threshold is large.

[0089] Optionally, when the difference between the initial reward model's score and the preference label for each candidate protein sequence pair is less than a preset threshold, i.e., the loss function... If the difference between the score and the preference label of each candidate protein sequence pair is less than a preset threshold, then training of the initial reward model is stopped, and the loss function at this point is obtained. parameters The corresponding reward model.

[0090] In this embodiment, a pairwise sequence comparison training method based on preference labels is adopted, which enables the reward model to learn the function of protein sequences more accurately, avoids the one-sidedness of single-sequence scoring, and improves the reliability and discriminativeness of the reward signal. Simultaneously, using a training termination condition where the difference between the score and the label is below a preset threshold ensures that the reward model's scoring accuracy meets the target while avoiding overtraining and reducing computational resource consumption. This application trains the initial reward model offline, transforming the time-consuming and costly quantum computing evaluation into reward signal generation for the reward model. This reduces the strong dependence on large-scale wet experimental data or time-consuming molecular dynamics simulations, providing sufficient, reliable, and physically meaningful reward signals for reinforcement learning training, enabling the model to conduct efficient and guided exploration in a broad sequence space.

[0091] Figure 6 This is a schematic diagram illustrating the process of determining the pre-trained policy model in a reinforcement learning-based protein sequence generation model training method provided in an embodiment of this application. Figure 6 As shown, obtaining the pre-trained policy model includes: S601. Construct an initial strategy model based on the diffusion model.

[0092] The diffusion model generates structured data from random noise through a stepwise denoising process, which can adapt to the distribution characteristics of protein sequences.

[0093] Optionally, an initial strategy model can be constructed based on a diffusion model to match the protein sequence generation requirements.

[0094] S602. Based on the natural protein sequences in the natural protein sequence database, train the initial policy model to obtain a pre-trained policy model.

[0095] Natural protein sequence databases are databases containing the amino acid sequences of a large number of proteins found in nature. These databases can be public or proprietary. A target category of natural protein sequences is a collection of proteins with common characteristics. For example, a target category could be natural protein sequences of the same functional class, the same structural class, or the same family. A natural protein sequence is a chain-like molecule composed of 20 standard amino acids arranged linearly in a specific order. The sequence of a protein determines its three-dimensional structure and function.

[0096] Optionally, the initial policy model is trained based on natural protein sequences in a natural protein sequence database to improve the accuracy of the protein sequences generated by the initial policy model, thereby obtaining a pre-trained policy model.

[0097] In the embodiments of this application, the diffusion model can adapt to the complex distribution characteristics of protein sequences and spatial structures. Pre-training based on natural protein sequences allows the model to fully learn the prior knowledge and distribution patterns of natural proteins in terms of sequence patterns, structural conservation, and functional association characteristics, thereby improving the strategy model's ability to generate reasonable protein sequences and reducing the training difficulty and convergence cost in the subsequent targeted optimization stage.

[0098] Figure 7 This is a schematic diagram illustrating the process of determining the pre-trained policy model in another reinforcement learning-based protein sequence generation model training method provided in this application embodiment. Figure 7 As shown, based on natural protein sequences in a natural protein sequence database, an initial policy model is trained to obtain a pre-trained policy model, including: S701. Perform forward noise processing on the natural protein sequence according to the preset noise to obtain the noise-added protein sequence.

[0099] Preset noise refers to the noisy version generated according to rules during the forward noise addition process.

[0100] Optionally, a discrete diffusion method is used, based on the natural protein sequence and a preset noise level, to replace the original amino acids with other types at each step with a certain probability, eventually obtaining a state that gradually approaches full noise, which is the noisy protein sequence.

[0101] Optionally, a discretized sequence diffusion model is used to perform forward noise processing on the natural protein sequence based on the Markov diffusion process and a preset noise level, resulting in a noisy protein sequence. Specifically, forward noise is performed based on the current sequence passing through the natural protein sequence. ,in For a moment The amino acid sequence state. Replace any sequence in the current sequence with the native protein sequence to obtain a new protein sequence. Perform noise addition sequentially in chronological order until a noisy protein sequence is obtained.

[0102] S702. Using noisy protein sequences and natural protein sequences, train the denoising network in the initial policy model so that the pre-trained policy model has the ability to reverse generate protein sequences from random noise.

[0103] The noise-generating sequence can be determined based on the natural protein sequence. During the diffusion process, the noise changes over time, and the association between the natural protein sequence and the predicted noise sequence is different at different noise levels.

[0104] Optionally, a discrete sequence diffusion model is used to learn the association between the natural protein sequence and the predicted noise sequence, determine the association between the natural protein sequence and the predicted noise sequence under different noise levels, and perform backward denoising on the noisy protein sequence based on the association to obtain the predicted noise sequence.

[0105] Optionally, the denoising network is parameter-tuned based on the predicted noise and the preset noise sequence to obtain a pre-trained policy network. Specifically, a loss function for the denoising network is constructed based on the predicted noise sequence and the preset noise sequence. The denoising network is then parameter-tuned with the goal of minimizing the loss function to obtain the trained denoising network.

[0106] Optionally, based on the predicted noise sequence and preset noise Construct the loss function:

[0107] in, Preset noise, Predict the noise sequence at time t. Let the loss function of the denoising network be... As the expected value, For the target category of natural protein sequences, For diffusion time step, It is a random noise vector.

[0108] Optionally, the parameters of the denoising network are determined with the goal of minimizing the loss function of the denoising network. The optimal value is determined based on the parameters of the policy network. The optimization values ​​are used to tune the parameters of the policy network to obtain the denoising network.

[0109] In this embodiment, the user uses the forward noise addition and reverse noise reduction process of the diffusion model to enable the policy model to accurately learn the structural patterns of natural protein sequences, and restore random noise to natural sequence features with biological rationality. This enables the pre-trained policy model to obtain the basic ability to generate reasonable protein sequences from random noise, which not only provides a stable and biologically prior starting point for subsequent targeted optimization, but also effectively improves the folding feasibility and functional potential of the generated sequences.

[0110] This application also provides a method for generating protein sequences. Figure 8 This is a schematic flowchart illustrating a protein sequence generation method provided in an embodiment of this application. Figure 8 As shown, the method includes: S801, Obtain random noise.

[0111] Random noise refers to unstructured input signals sampled in the latent space of a generative model. Random noise is a controllable perturbation source with a well-defined mathematical definition and functional purpose. It can be a latent vector sampled from a standard normal distribution, a highly noisy sequence of terminal states, or an embedding vector injected with Gaussian noise.

[0112] Optionally, a randomized sequence is sampled, which can be a random vector sampled from a standard normal distribution; or each position of the sequence can be an amino acid randomly selected from a uniform distribution or a specific sequence. The length of this noise sequence should match the length of the target protein sequence.

[0113] S802. Generate protein sequences using a protein sequence generation model based on random noise.

[0114] The protein sequence generation model is a protein sequence generation model trained using any of the reinforcement learning-based protein sequence generation model training methods mentioned above. A protein sequence refers to a novel amino acid sequence generated by the target model under given random noise conditions.

[0115] Optionally, based on the random noise, the association between the protein sequence and the noise is determined using a protein sequence generation model. Backward denoising is then performed according to the time sequence to determine the probability of 20 amino acids at each position. Through greedy sampling or random sampling, the output at each position is converted into specific amino acid letters, thereby generating the protein sequence.

[0116] In this embodiment, by introducing random noise as the starting point for generation and combining it with a trained protein sequence generation model for the target category, end-to-end directional design from initial perturbation without specific structure to target protein sequences with clear functional characteristics is achieved, thereby improving the production efficiency of novel candidate molecules and overcoming the technical limitations of traditional protein design that relies on human experience or limited mutation scanning.

[0117] Based on the same inventive concept, this application also provides a reinforcement learning-based protein sequence generation model training device corresponding to the reinforcement learning-based protein sequence generation model training method. Since the principle of the device in this application is similar to the reinforcement learning-based protein sequence generation model training method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0118] Figure 9 A schematic diagram of a protein sequence generation model training device based on reinforcement learning provided in this application embodiment is shown below. Figure 9 As shown, the device includes: a first acquisition module 901, a construction module 902, a training module 903, and an optimization module 904; wherein, the first acquisition module 901 is used to acquire a pre-trained policy model, wherein the policy model is used to reverse generate protein sequences based on random noise; Module 902 is used to build the first dataset based on the protein sequences generated by the policy model and the quantum chemical calculation results for the protein sequences. The first dataset is used to train the reward model. Training module 903 is used to train the initial reward model based on the first dataset to obtain the reward model; The optimization module 904 is used to use the score output by the reward model as a reward signal, and optimize the parameters of the strategy model based on the reward signal to obtain a protein sequence generation model.

[0119] In one possible implementation, the optimization module 904 is specifically used to: score each protein sequence output by the strategy model using a reward model to obtain a reward signal; Based on the policy gradient algorithm and reward signal, the parameters of the policy model are adjusted to maximize the expected reward of the protein sequence generated by the policy model. And when the preset conditions are met during the online optimization phase, the latest strategy model is used as the protein sequence generation model.

[0120] In one possible implementation, the optimization module 904 is further configured to: add a divergence constraint to the policy model, wherein the divergence constraint is used to balance the maximization of reward expectation with the rationality of the protein sequence generated by the policy model.

[0121] In one possible implementation, the preset conditions include any one of the following: the number of optimization rounds reaches a preset number of rounds, the expected reward reaches a preset threshold, and the expected reward is in a convergent state.

[0122] In one possible implementation, the construction module 902 is specifically used for: Multiple candidate protein sequences were generated using a strategy model; According to the preset configuration method, multiple candidate protein sequences are paired to obtain multiple candidate protein sequence pairs; For each of the multiple candidate protein sequence pairs, quantum chemical calculations are performed to obtain the quantum chemical calculation results for each candidate protein sequence pair. Based on the quantum chemical calculation results of each pair of candidate protein sequences, a preference label is determined for each pair of candidate protein sequences. The preference label is used to indicate the chemical performance of the two protein sequences in each pair of candidate protein sequences. Each pair of candidate protein sequence pairs and its corresponding preference label are grouped together to form the first dataset.

[0123] In one possible implementation, the construction module 902 is specifically used for: Each candidate protein sequence pair was scored using an initial reward model; The loss function of the initial reward model is trained and optimized based on the score and preference label of each candidate protein sequence pair. When the difference between the score and the preference label of the initial reward model for each candidate protein sequence pair is less than a preset threshold, training of the initial reward model is stopped, and the reward model is obtained.

[0124] In one possible implementation, the first acquisition module 901 is specifically used for: constructing an initial strategy model based on the diffusion model; The initial policy model is trained using natural protein sequences from a natural protein sequence database to obtain a pre-trained policy model.

[0125] In one possible implementation, the first acquisition module 901 is specifically used to: perform forward noise processing on the natural protein sequence according to a preset noise to obtain a noisy protein sequence; By using noisy protein sequences and natural protein sequences, the denoising network in the initial policy model is trained so that the pre-trained policy model has the ability to reverse generate protein sequences from random noise.

[0126] Based on the same inventive concept, this application also provides a protein sequence generation device corresponding to the protein sequence generation method. Since the principle of the device in this application is similar to the protein sequence generation method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0127] Figure 10 This is a schematic diagram of the structure of a protein sequence generation device provided in an embodiment of this application, as shown below. Figure 10 As shown, the device includes: a second acquisition module 1001 and a second generation module 1002; wherein, the second acquisition module 1001 is used to acquire random noise; The generation module 1000 is used to generate protein sequences based on random noise using a protein sequence generation model, wherein the protein sequence generation model is a protein sequence generation model trained using any of the above-mentioned reinforcement learning-based protein sequence generation model training methods.

[0128] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0129] This application also provides a computer device. Figure 11 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 11 As shown, the computer device includes a processor 1101 and a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the computer device is running, the processor 1101 and the memory 1102 communicate via the bus 1103. When the machine-readable instructions are executed by the processor 1101, the steps of the aforementioned reinforcement learning-based protein sequence generation model training method and protein sequence generation method are performed.

[0130] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described reinforcement learning-based protein sequence generation model training method and protein sequence generation method.

[0131] This application also provides a computer program product, which includes computer program code. When the computer program code is executed by a computing device, the computing device performs the steps of the above-described reinforcement learning-based protein sequence generation model training method and protein sequence generation method.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods S described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0134] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A protein sequence generation model training method based on reinforcement learning, comprising an offline computation stage and an online optimization stage, characterized in that, In the offline computing phase, the method includes: Obtain a pre-trained policy model, wherein the policy model is used to inversely generate protein sequences based on random noise; the pre-trained policy model is a policy model constructed based on a diffusion model, and the input of the pre-trained policy model is random noise, and the output is a protein sequence or a protein sequence pair; Multiple candidate protein sequences are generated using the aforementioned strategy model; According to a preset configuration method, the multiple candidate protein sequences are paired to obtain multiple candidate protein sequence pairs; For each of the multiple candidate protein sequence pairs, quantum chemical calculations are performed to obtain the quantum chemical calculation results for each candidate protein sequence pair. Based on the quantum chemical calculation results of each pair of candidate protein sequences, a preference label is determined for each pair of candidate protein sequences. The preference label is used to indicate the chemical performance of the two protein sequences in each pair of candidate protein sequences. Each pair of candidate protein sequence pairs and its corresponding preference label are grouped into a set of data to form the first dataset, which is used to train the reward model. Based on the first dataset, the initial reward model is trained to obtain the reward model; And in the online optimization phase, the method includes: The score output by the reward model is used as a reward signal. The parameters of the strategy model are optimized based on the reward signal to obtain a protein sequence generation model. The protein sequence generation model is a protein sequence generation model constructed based on a diffusion model. The input of the protein sequence generation model is random noise, and the output is a protein sequence.

2. The method according to claim 1, characterized in that, The step of using the score output by the reward model as a reward signal, and optimizing the parameters of the strategy model based on the reward signal to obtain a protein sequence generation model, includes: The reward model is used to score each protein sequence output by the strategy model to obtain the reward signal; Based on the policy gradient algorithm and reward signal, the parameters of the policy model are adjusted to maximize the expected reward of the protein sequence generated by the policy model. And when the preset conditions are met during the online optimization phase, the latest strategy model is used as the protein sequence generation model.

3. The method according to claim 2, characterized in that, The method further includes: A divergence constraint is added to the policy model, wherein the divergence constraint is used to balance the maximization of the expected reward with the rationality of the protein sequence generated by the policy model.

4. The method according to claim 2, characterized in that, The preset conditions include any one of the following: the number of optimization rounds reaches a preset number of rounds, the expected reward reaches a preset threshold, and the expected reward is in a convergent state.

5. The method according to claim 1, characterized in that, The step of training an initial reward model based on the first dataset to obtain the reward model includes: Each candidate protein sequence pair is scored using the initial reward model. The loss function of the initial reward model is trained and optimized based on the score and preference label of each candidate protein sequence pair. When the difference between the score and the preference label of the initial reward model for each candidate protein sequence pair is less than a preset threshold, the training of the initial reward model is stopped, and the reward model is obtained.

6. The method according to claim 1, characterized in that, The acquisition of the pre-trained policy model includes: Construct an initial strategy model based on a diffusion model; The initial strategy model is trained based on natural protein sequences from a natural protein sequence database to obtain the pre-trained strategy model.

7. The method according to claim 6, characterized in that, The process of training the initial strategy model based on natural protein sequences from a natural protein sequence database to obtain the pre-trained strategy model includes: The natural protein sequence is subjected to forward noise processing based on a preset noise to obtain a noisy protein sequence. The denoising network in the initial policy model is trained using the noisy protein sequence and the natural protein sequence, so that the pre-trained policy model has the ability to reverse generate protein sequences from random noise.

8. A method for generating a protein sequence, characterized in that, The method includes: Obtain random noise; Based on the random noise, a protein sequence is generated using a protein sequence generation model, wherein the protein sequence generation model is a protein sequence generation model trained using any of the reinforcement learning-based protein sequence generation model training methods in claims 1-7.

9. A computer device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the computer device is running, are executed by the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 8.

11. A computer program product, characterized in that, The computer program product includes computer program code, which, when executed by a computing device, performs the steps of the method according to any one of claims 1 to 8.

Images