Protein design method and device based on symmetric self-game online reinforcement learning
By employing a symmetric self-game online reinforcement learning approach and utilizing a dual-agent framework to collaboratively optimize protein design, this approach addresses the issues of insufficient exploration capabilities and multi-objective conflicts in existing technologies, achieving high efficiency, diversity, and stability in protein design.
Patent Information
- Application Number
- CN202610110243.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-19
AI Technical Summary
Existing protein backfolding techniques rely on static data, resulting in limited exploration capabilities. Single-agent online reinforcement learning suffers from problems such as pattern collapse, difficulty in coordinating multiple objectives, and training stagnation, making it difficult to achieve effective protein design without relying on massive amounts of expert data.
A symmetric self-game online reinforcement learning method is adopted to construct a dual-agent framework, in which one agent focuses on structural consistency and the other focuses on physical stability. They are trained by providing each other with dynamic adversarial examples to collaboratively optimize the quality and diversity of the design sequence.
It achieves a synergistic improvement in structural fidelity and physical foldability, avoids gradient vanishing and mode collapse, and significantly improves the diversity and success rate of design sequences.
Smart Images
Figure CN122067593A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of bioinformatics and artificial intelligence, specifically to a protein design method and apparatus based on symmetric self-game online reinforcement learning. Background Technology
[0002] Proteins are the primary agents of life processes, their functions determined by their three-dimensional structure, which in turn is formed by the folding of amino acid sequences. In synthetic biology and drug development, protein design is a core technology aimed at creating novel protein molecules with specific structures and functions. Structure-based protein sequence design, often referred to as "protein inverse folding," involves predicting the amino acid sequence that can fold into a given target protein backbone structure. This technology has significant applications in antibody design, enzyme engineering, and biosensor development.
[0003] Traditional protein design methods (such as Rosetta Design) rely on physical energy functions, resulting in high computational costs. With the development of deep learning, data-driven methods have become mainstream. For example, ProteinMPNN uses graph neural networks for supervised learning, training on known structure-sequence pairs. While this is fast and has a high recovery rate, the generated sequences are severely limited by the natural distribution of the training data (such as the PDB database), making it difficult to explore novel sequence spaces and directly optimize specific physicochemical properties (such as thermal stability). ESM-IF1 improves generalization ability through pre-training on large-scale predicted structures, but it still does not break away from the paradigm of static supervised learning.
[0004] To overcome the limitations of static data, researchers have begun to introduce reinforcement learning (RL). For example, methods like ProteinZERO employ an online "generate-evaluate-update" model, using scores provided by structure prediction tools (such as ESMFold) as reward signals to optimize the generative model. However, such single-agent online reinforcement learning methods have significant drawbacks: 1. Mode Collapse: The model is prone to generating a single, simple high-scoring pattern (such as a repetitive sequence rich in specific amino acids), losing diversity; 2. Conflicting objectives: It is difficult to balance the two mutually restrictive objectives of "structural fidelity" (such as sc-TM score) and "physical foldability" (such as pLDDT score), which often leads to "geometric overfitting"—the generated sequence is structurally matched but physically unstable; 3. Training stagnation: When using methods such as Direct Preference Optimization (DPO), as the model's capabilities improve, the negative samples it generates become too simple, leading to gradient vanishing and performance reaching a bottleneck.
[0005] Therefore, existing protein backfolding technologies, whether based on static supervised learning or single-agent online reinforcement learning, are unable to effectively overcome pattern collapse, reconcile multi-objective conflicts, and achieve continuous performance evolution without relying on massive amounts of expert data. Summary of the Invention
[0006] The purpose of this invention is to provide a protein design method and apparatus based on symmetric self-game online reinforcement learning, which solves the technical problems in existing protein backfolding technology, such as limited exploration capabilities due to reliance on static data, and pattern collapse, difficulty in multi-objective coordination, and performance stagnation caused by the failure of negative samples in the later stages of training, in single-agent online reinforcement learning. By constructing an asymmetric two-agent online game framework, the two agents focus on optimizing structural consistency and physical stability respectively, and are trained by providing each other with dynamic and differentiated adversarial samples, thereby synergistically improving the quality and diversity of design sequences and achieving Pareto optimization of structural fidelity and physical foldability.
[0007] To achieve the above objectives, the technical solution of the present invention is as follows: A protein design method based on symmetric self-game online reinforcement learning includes the following steps: S1. Data preparation and preprocessing: Input the main chain structure information of the target protein, construct a training set based on the protein structure database, and characterize the protein structure as graph data; S2. Model Architecture Initialization: Construct an isovariant graph neural network EGNN as the basic architecture of the sequence generator, and initialize the first policy model, the second policy model and the reference model. The parameters of the first policy model and the second policy model are independent. S3. Symmetric Self-Game Online Training: S31. For samples in the training set, candidate amino acid sequences are generated using the first strategy model, the second strategy model, and the reference model, respectively; S32. Evaluate each candidate sequence to obtain a first reward that characterizes its geometric fit with the target structure and a second reward that characterizes its predicted folding stability; S33. Construct training preference pairs based on the first reward and the second reward; S34. Based on the training preference pair, jointly optimize the first policy model and the second policy model, wherein the objective function of the joint optimization includes: making the first policy model tend to generate sequences with high first rewards, making the second policy model tend to generate sequences with high second rewards, and keeping the output distributions of the first policy model and the second policy model different. S35. Iteratively execute the generation, evaluation, and optimization steps to update the model parameters in an online self-game manner; S36. Using the trained first and second strategy models, generate a design sequence for the input target protein backbone structure.
[0008] Furthermore, step S1 includes the following specific steps: S11. Input a value of size Protein backbone for sequence design Three-dimensional coordinate information, denoted as P ,in For the number of residues, For each residue Cartesian three-dimensional coordinates of an atom; S12. Download all single-stranded proteins from the PDB database, denoted as the dataset. D And extract the protein chains from dataset D. Atomic coordinates and primary amino acid sequences are denoted as follows: and ,in For dataset The total number of single chains in the chain; S13. Use the Foldseek tool to extract the dataset from S12. Clustering M Each cluster is analyzed, and representative proteins from all clusters are extracted to form a non-redundant training set. , including input features and tags ; S14. For the training set in S13 The length of each line in the text is Coordinate features of proteins , with each residue Define a protein graph using atoms as nodes. ,in, The graph's edges represent the node feature set, which is represented by the eight secondary structural states of a protein. Indicates the current protein residue number and residue number Euclidean distance and , It is a coordinate set.
[0009] Furthermore, S2 specifically includes: initializing the equivariant graph neural network. The obtained protein map , No. layer The forward process is defined as follows:
[0010] in, and This represents a multilayer perceptron that updates nodes and edges. This indicates taking the average. Composed of multilayer perceptron and Function composition and Convert to a scalar, a complete The forward process is defined as Output , Indicates the first The residue was predicted to be the first The predicted probability of amino acid-like compounds was obtained by sampling amino acid sequences using a Top-K sampling algorithm with a temperature of 0.5 and K=5. As a design outcome.
[0011] Furthermore, the isomorphic graph neural network in S2 includes three isomorphic graph convolutional layers and one fully connected layer, used to extract geometric features from the protein backbone coordinates and predict the amino acid probability distribution.
[0012] Furthermore, S31 specifically includes: initializing two policy models. and a reference model For a protein sample , respectively Design three sizes Candidate amino acid sequence library , and The ESMFold algorithm was used to predict the 3D structures, obtaining the pTM and pLDDT indices for each design sequence in the library. The Us-align tool was then used to align the predicted structures with the actual structures. Structural comparison was performed to obtain the scRMSD and scTM indices.
[0013] Furthermore, S32 specifically includes: respectively targeting Define the following sequence-structure consistency reward function and prediction stability reward function : .
[0014] Furthermore, S33 specifically includes: […]. , and The three sequence libraries were merged, respectively according to and Sort and construct preference pairs using first-to-last pairing method { }and { }
[0015] Furthermore, S34 includes the following steps: S341. Construction preference pair , respectively targeting Define the following loss function and :
[0016] in, and They are respectively Regarding the predicted probability of the current forecast, For skeletal conditions, and These represent the preference pairs. and , for The cumulative predicted probability of the model for Output the entropy of the distribution; S342. Don't be... The AdamW optimizer is defined to perform backpropagation and optimization, with a learning rate of 1e-4, weight decay of 0.01, and gradient clipping of 1.0. , Every 50 steps, an EMA algorithm with a decay rate of 0.995 is used for following. Parameter average update 99.5% of the parameters.
[0017] This invention also provides a protein design device based on symmetric self-game online reinforcement learning, comprising: The structure input module is used to receive the coordinates of the protein backbone; The graph building module is used to convert structures into graph representations; The dual-model generation module includes two differentiated strategy models and one reference model; The evaluation module is used to call ESMFold to perform structural prediction and index calculation; Preference pair building blocks are used to sort and pair samples according to the reward function; The loss calculation and update module is used to calculate the loss and update the model parameters; The output module is used to generate the final protein sequence.
[0018] Furthermore, the two policy models in the dual-model generation module share the same isomorphic graph neural network architecture, but are trained differentially using different reward functions and loss terms.
[0019] By adopting the above technical solution, the present invention has the following advantages: 1. This invention provides a protein design method and apparatus based on symmetric self-game online reinforcement learning. Through two differentiated strategy models and a reference model, one model focuses on structural fidelity and the other focuses on physical stability. The two models constrain and correct each other in the game, forcing the generated sequence to simultaneously meet the dual stringent criteria. This breaks through the limitations of single-objective optimization and achieves a synergistic improvement in structural fidelity and physical stability, effectively solving the contradiction between "geometric overfitting" and "physical infeasibility".
[0020] 2. This invention provides a protein design method and apparatus based on symmetric self-game online reinforcement learning, which has the ability to continuously self-evolve: the online self-game mechanism can generate dynamic hard negative samples that match the current model's capabilities in real time, avoiding the gradient vanishing problem caused by overly simple negative samples in the later stages of offline reinforcement learning, enabling the model to continuously break through the performance ceiling and explore high-quality sequences outside the training set.
[0021] 3. This invention provides a protein design method and apparatus based on symmetric self-game online reinforcement learning, which effectively overcomes mode collapse and improves diversity: by introducing mutual exclusion loss and differentiated exploration strategies (such as different sampling temperatures), the two strategy models are forced to maintain policy differences, forming a "catfish effect". This not only prevents the model itself from falling into a single mode, but also greatly expands the overall search space, significantly improving the diversity of design and the success rate of complex skeletons. Attached Figure Description
[0022] Figure 1 A flowchart of a protein design method provided in an embodiment of the present invention. Detailed Implementation
[0023] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.
[0024] A protein design method based on symmetric self-game online reinforcement learning includes the following steps: Figure 1 As shown: S1. Data preparation and preprocessing: Input the main chain structure information of the target protein, construct a training set based on the protein structure database, and characterize the protein structure as graph data; S1 includes the following specific steps: S11. Input a value of size Protein backbone for sequence design Three-dimensional coordinate information, denoted as P ,in For the number of residues, For each residue Cartesian three-dimensional coordinates of an atom; S12. Download all single-stranded proteins from the PDB database, denoted as the dataset. D And extract the protein chains from dataset D. Atomic coordinates and primary amino acid sequences are denoted as follows: and ,in For dataset The total number of single chains in the chain; S13. Use the Foldseek tool with a clustering threshold of 0.4 to process the dataset obtained in S12. Clustering M Each cluster is analyzed, and representative proteins from all clusters are extracted to form a non-redundant training set. , including S12 and subset of and , where coordinates As input features, As a tag, it indicates the protein sequence to be designed; S14. For the training set in S13 The length of each line in the text is Coordinate features of proteins , with each residue Define a protein graph using atoms as nodes. ,in, The graph's edges represent the node feature set, which is represented by the eight secondary structural states of a protein. Indicates the current protein residue number and residue number Euclidean distance and , It is a coordinate set.
[0025] S2. Model Architecture Initialization: Construct an isovariant graph neural network EGNN as the basic architecture of the sequence generator, and initialize the first policy model, the second policy model and the reference model. The parameters of the first policy model and the second policy model are independent. Specifically, S2 includes: initializing a three-layer isomorphic graph convolutional layer. and a fully connected layer Equivariant graph neural network representation This is used to extract geometric features from the protein backbone coordinates and predict the amino acid probability distribution. The protein map obtained from S14... , No. layer The forward process is defined as follows:
[0026] in, and This represents a multilayer perceptron that updates nodes and edges. This indicates taking the average. Composed of multilayer perceptron and Function composition and Convert to a scalar, a complete The forward process is defined as Output , Indicates the first The residue was predicted to be the first The predicted probability of amino acid-like compounds was obtained by sampling amino acid sequences using a Top-K sampling algorithm with a temperature of 0.5 and K=5. As a design outcome.
[0027] S3. Symmetric Self-Game Online Training: S31. For the samples in the training set, candidate amino acid sequences are generated using the first strategy model, the second strategy model, and the reference model, respectively; Specifically, S31 includes: for S2 Neural network, initializing two policy models and a reference model For a protein sample , respectively Based on steps S12 and S2, three sizes are designed. Candidate amino acid sequence library , and The ESMFold algorithm was used to predict the 3D structures, obtaining the pTM and pLDDT indices for each design sequence in the library. The Us-align tool was then used to align the predicted structures with the actual structures. Structural comparison was performed to obtain the scRMSD and scTM indices; S32. Evaluate each candidate sequence to obtain a first reward that characterizes its geometric fit with the target structure and a second reward that characterizes its predicted folding stability; S32 specifically includes: for the obtained structural prediction indicators, respectively targeting... Define the following sequence-structure consistency reward function and prediction stability reward function :
[0028] S33. Construct training preference pairs based on the first and second rewards; S33 specifically includes: [The following is a list of components] , and The three sequence libraries were merged, respectively according to and Sort and construct preference pairs using first-to-last pairing method { }and { }
[0029] S34. Based on the training preference pairs, jointly optimize the first policy model and the second policy model, wherein the objective function of the joint optimization includes: making the first policy model tend to generate sequences with high first rewards, making the second policy model tend to generate sequences with high second rewards, and keeping the output distributions of the first policy model and the second policy model different. S34 includes the following specific steps: S341. Regarding the preference pairs constructed in the middle , respectively targeting Define the following loss function and :
[0030] in, and They are respectively Regarding the predicted probability of the current forecast, For skeletal conditions, and These represent the preference pairs. and , for The cumulative predicted probability of the model for Output the entropy of the distribution; S342. respectively Define the AdamW optimizer to perform backpropagation and optimization with a learning rate of 1e. -4 The weight decay is 0.01, and the gradient clipping is 1.0. , Every 50 steps, an EMA algorithm with a decay rate of 0.995 is used for following. Parameter average update 99.5% of the parameters.
[0031] S35. Iteratively execute the generation, evaluation, and optimization steps to update the model parameters in an online self-game manner; For the obtained protein training set Repeat steps S31-S34 to generate sample preference pairs online and update the model until training is complete.
[0032] S36. Using the trained first and second strategy models, generate a design sequence for the input target protein backbone structure.
[0033] The protein in S1 to be sequenced P main chain Inputting 3D coordinate information to construct and train In the process, amino acid sequences are sampled from the output probability distribution, and the average index calculated by the ESMFold tool is used to rank them. The Top-N sequences are selected as target candidate sequences.
[0034] Furthermore, the protein design device based on symmetric self-game online reinforcement learning of the present invention, corresponding to the protein design method based on symmetric self-game online reinforcement learning, specifically includes: The structure input module is used to receive the coordinates of the protein backbone; The graph building module is used to convert structures into graph representations; The dual-model generation module includes two differentiated strategy models. and a reference model The two policy models in the dual-model generation module share the same isomorphic graph neural network architecture, but are trained differently using different reward functions and loss terms.
[0035] The evaluation module is used to call ESMFold to perform structural prediction and index calculation; Preference pair building blocks are used to sort and pair samples according to the reward function; The loss calculation and update module is used to calculate the loss and update the model parameters; The output module is used to generate the final protein sequence.
[0036] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.
Claims
1. A protein design method based on symmetric self-game online reinforcement learning, characterized in that, Includes the following steps: S1. Data preparation and preprocessing: Input the main chain structure information of the target protein, construct a training set based on the protein structure database, and characterize the protein structure as graph data; S2. Model Architecture Initialization: Construct an isovariant graph neural network EGNN as the basic architecture of the sequence generator, and initialize the first policy model, the second policy model and the reference model. The parameters of the first policy model and the second policy model are independent. S3. Symmetric Self-Game Online Training: S31. For the samples in the training set, candidate amino acid sequences are generated using the first strategy model, the second strategy model, and the reference model, respectively; S32. Evaluate each candidate sequence to obtain a first reward that characterizes its geometric fit with the target structure and a second reward that characterizes its predicted folding stability; S33. Construct training preference pairs based on the first reward and the second reward; S34. Based on the training preference pair, jointly optimize the first policy model and the second policy model, wherein the objective function of the joint optimization includes: making the first policy model tend to generate sequences with high first rewards, making the second policy model tend to generate sequences with high second rewards, and keeping the output distributions of the first policy model and the second policy model different. S35. Iteratively execute the generation, evaluation, and optimization steps to update the model parameters in an online self-game manner; S36. Using the trained first and second strategy models, generate a design sequence for the input target protein backbone structure.
2. The protein design method based on symmetric self-game online reinforcement learning according to claim 1, characterized in that, S1 includes the following specific steps: S11. Input a value of... Protein backbone for sequence design Three-dimensional coordinate information, denoted as P ,in For the number of residues, For each residue Cartesian three-dimensional coordinates of an atom; S12. Download all single-stranded proteins from the PDB database, denoted as the dataset. D And extract the protein chains from dataset D. Atomic coordinates and primary amino acid sequences are denoted as follows: and ,in For dataset The total number of single chains in the chain; S13. Use the Foldseek tool to extract the dataset from S12. Clustering M Each cluster is analyzed, and representative proteins from all clusters are extracted to form a non-redundant training set. , including input features and tags ; S14. For the training set in S13 The length of each line in the text is Coordinate features of proteins , with each residue Define a protein graph using atoms as nodes. ,in, The graph's edges represent the node feature set, which is represented by the eight secondary structural states of a protein. Indicates the current protein residue number and residue number Euclidean distance and , It is a coordinate set.
3. The protein design method based on symmetric self-game online reinforcement learning according to claim 2, characterized in that, S2 specifically includes: initializing the equivariant graph neural network. The obtained protein map , No. layer The forward process is defined as follows: in, and This represents a multilayer perceptron that updates nodes and edges. This indicates taking the average. Composed of multilayer perceptron and Function composition and Convert to a scalar, a complete The forward process is defined as Output , Indicates the first The residue was predicted to be the first The predicted probability of amino acid-like compounds was obtained by sampling amino acid sequences using a Top-K sampling algorithm with a temperature of 0.5 and K=5. As a design outcome.
4. The protein design method based on symmetric self-game online reinforcement learning according to claim 3, characterized in that, The isomorphic graph neural network in S2 includes three isomorphic graph convolutional layers and one fully connected layer, which are used to extract geometric features from the protein backbone coordinates and predict the amino acid probability distribution.
5. The protein design method based on symmetric self-game online reinforcement learning according to claim 4, characterized in that, S31 specifically includes: initializing two policy models. and a reference model For a protein sample , respectively Design three sizes Candidate amino acid sequence library , and The ESMFold algorithm was used to predict the 3D structures, obtaining the pTM and pLDDT indices for each design sequence in the library. The Us-align tool was then used to align the predicted structures with the actual structures. Structural comparison was performed to obtain the scRMSD and scTM indices.
6. The protein design method based on symmetric self-game online reinforcement learning according to claim 5, characterized in that, S32 specifically includes: respectively targeting Define the following sequence-structure consistency reward function and prediction stability reward function : 。 7. The protein design method based on symmetric self-game online reinforcement learning according to claim 6, characterized in that, S33 specifically includes: […]. , and The three sequence libraries were merged, respectively according to and Sort and construct preference pairs using first-to-last pairing method { }and { } 8. The protein design method based on symmetric self-game online reinforcement learning according to claim 7, characterized in that, S34 includes the following steps: S341. Construction preference pair , respectively targeting Define the following loss function and : in, and They are respectively Regarding the predicted probability of the current forecast, For skeletal conditions, and These represent the preference pairs. and , for The cumulative predicted probability of the model for Output the entropy of the distribution; S342. Don't be... The AdamW optimizer is defined to perform backpropagation and optimization, with a learning rate of 1e-4, weight decay of 0.01, and gradient clipping of 1.
0. , Every 50 steps, an EMA algorithm with a decay rate of 0.995 is used for following. Parameter average update 99.5% of the parameters.
9. A protein design device based on symmetric self-game online reinforcement learning, characterized in that, include: The structure input module is used to receive the coordinates of the protein backbone; The graph building module is used to convert structures into graph representations; The dual-model generation module includes two differentiated strategy models and one reference model; The evaluation module is used to call ESMFold to perform structural prediction and index calculation; Preference pair building blocks are used to sort and pair samples according to the reward function; The loss calculation and update module is used to calculate the loss and update the model parameters; The output module is used to generate the final protein sequence.
10. The protein design device based on symmetric self-game online reinforcement learning according to claim 9, characterized in that, The two policy models in the dual-model generation module share the same isomorphic graph neural network architecture, but are trained differently using different reward functions and loss terms.